Charting and Navigating the Space of Solutions for Recurrent Neural Networks
Authors: Elia Turner
Kabir Vinay Dabholkar
Omri Barak
Venue: NeurIPS 2021 Poster (2021)
Keywords: RNN
underspecification
variability
space of solutions
neuroscience
reverse engineering
task optimized networks

Abstract: In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating under the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and machine learning challenge this assumption. Animals can approach a given task with a variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that every task is associated with a space of solutions. To date, the structure of this space is not understood, limiting the approach of comparing RNNs with neural data.
Here, we characterize the space of solutions associated with various tasks. We first study a simple two-neuron network on a task that leads to multiple solutions. We trace the nature of the final solution back to the network’s initial connectivity and identify discrete dynamical regimes that underlie this diversity. We then examine three neuroscience-inspired tasks: Delayed discrimination, Interval discrimination, and Time reproduction. For each task, we find a rich set of solutions. One layer of variability can be found directly in the neural activity of the networks. An additional layer is uncovered by testing the trained networks' ability to extrapolate, as a perturbation to a system often reveals hidden structure. Furthermore, we relate extrapolation patterns to specific dynamical objects and effective algorithms found by the networks. We introduce a tool to derive the reduced dynamics of networks by generating a compact directed graph describing the essence of the dynamics with regards to behavioral inputs and outputs. Using this representation, we can partition the solutions to each task into a handful of types and show that neural features can partially predict them.
Taken together, our results shed light on the concept of the space of solutions and its uses both in Machine learning and in Neuroscience.

CHARTING AND NAVIGATING THE SPACE OF SOLUTIONS FOR RECURRENT NEURAL NETWORKS
Elia Turner
Kabir Dabholkar
Department of Mathematics Technion, Israel Institute of Technology el iliaturnerocamuu.. technion onniiin.oee ac.
Department of Mathematics
Technion, Israel Institute of Technology kabir@campus technion. ac il
Omri Barak
Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion, Israel Institute of Technology omri. warribarrakgmaiil.. com
Abstract
In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating der the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and machine learning challenge this assumption. Animals can approach given task with variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that every task is associated with space of solutions. To date, the structure of this space not understood, limiting the approach of comparing RNNs with neural data. Here, we characterize the space of solutions associated with various tasks. We first study simple two-neuron network on a task that leads to multiple solutions. We trace the nature of the final solution back to the network's 's initial connectivity and identify discrete dynamical regimes that underlie this diversity. We then examine three neuroscience-inspinn terval discrimination, and Time reproduction. For each task, we find a rich set solutions. One layer of variability can be found directly in the neural activity of the networks. An additional layer is uncovered by testing the trained networks' ability extrapolate, Furthermore, we relate extrapolation patterns to specific dynamical objects and effective algorithms found by the networks. We introduce a tool to derive the reduced dynamics networks by generating a compact directed graph describing the essence of the dynamics with regards to behavioral inputs and outputs. Using this representation, we can partition the solutions to each task into handful types and show that neural features can partially predict them. Taken together, our results shed light on the concept of the space of solutions and its uses both in Machine learning and in Neuroscience.
tasks: Delayed discrimination, In-
perturbation to system often reveals hidden structure.
Introduction
Modern machine learning operates in over-parameterized regime, implying that many different parameter-sets can achieve low error on a given training set (I). This observation implies that for every task, there exists a space of solutions that implement it. What the properties of such a solution space? Are networks able to learn solutions that capture the intended underlying
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

phenomena or do they reach artificial shortcuts that do not generalize well? What biases networks to prefer one solution over the other? These questions remain largely unanswered. A parallel phenomenon occurs in Neuroscience. When animals are instructed to perform a task in a controlled environment, they exhibit both neural and behavioral variability, which stem from different task- strategies 2 4 5 6 7 8 9 10, 11. In Computational Neuroscience trained Recurrent Neural Networks (RNNs) are used as a tool to explain functions and mechanisms that are observed in brain dynamics (1).. In fact, various recent studies have matched the activity of trained RNNs to that of experimental recording 13 14 15 16 17 I8 19). In light of the variability that undoubtedly exists on both sides of the comparison, these results seem puzzling. In this work, we present multiple tasks for which trained RNNs produce a rich space of qualitatively different solutions. We argue that to properly use artificial networks, and RNNs in particular, models of neural circuits that support given task, it is necessary to chart the space of solutions that arises from training.
Here, we first apply this approach to simple two-neuron network and demonstrate how distinct solutions arise. We then study three tasks inspired by the neuroscience literature: interval reproduction (20), delayed discrimination (2I), and interval discrimination (22). We show that different networks with identical hyperparameters find qualitatively different solutions. We find one layer of variability within the neural activity in response to stimuli from the training set. Since by design the output of all networks is identical during training, this layer is akin to multiple realizability (23). Next, we expose additional layer of variability when we challenge the networks with inputs that outside the distribution of the training set and systematically characterize the responses. Furthermore, we manage to show that the diversity revealed with these challenging inputs corresponds to qualitatively different computations performed by the network. To chart the space of solutions, we introduce a tool that reduces the dynamics of a network into a graph that captures the essence of the computation performed. Applying it to all networks partitions the space into a handful of possible reduced dynamics. Additionally, these classes can be partially predicted using experimentally accessible neural activity obtained only in response to trained stimuli.
2 Related work
Recent work, 244 studied the effects of modeling choices over the dynamics of the trained solutions. in a few canonical tasks. They trained thousands of RNNs, while systematically controlling the hyperparameters, and analyzed the geometrical and topological aspects of each solution. They concluded that while the geometry of different solutions can vary significantly across different architectures, the underlying computation and dynamical objects are widely universal. Our results are superficially opposite to theirs. This is probably due both to the choice of tasks and to challenging networks with unexpected inputs.
The authors of (25) show that deep networks trained on vision tasks develop different internal representations, especially in the higher layers. While reaching similar conclusions on the need to use populations of networks for neuroscience comparisons, the analysis of feedforward networks naturally focuses on representations rather than on dynamics. While writing this manuscript, we were made aware of recent work by (26) that takes very similar approach to ours. Apart from the different tasks, architectures and learning rules studied there, our analysis of the two-neuron network also provides a tangible example of discrete basins of attraction in solution space. An underspecified two- dimensional epidemiological model was studied by (.. The variability solutions there, however, was continuous. Specifically, all solutions were connected on single manifold and the dynamics of the system did not undergo bifurcations along this manifold.
Underspecification was also highlighted in neuroscience models that are not based on artificial neural networks (27) and is observed more generally in complex systems (28).
3 What constitutes a solution?
Before comparing how the different networks solve the various tasks, it is worth dwelling on the concept of a solution. This is not a trivial concept and can be related to fundamental philosophical questions. Aristotle suggested (29) the four causes of understanding an object: its material form, its formal description, its efficient cause (creation process), and its final cause (purpose). We can draw an analogy to our understanding of RNN solutions, and ask: What is their architecture (material),
2

---

what is their underlying algorithm or dynamics (formal), which optimization process led to their final state (efficient), and what task do they solve (final). Because our motivation stems from comparing networks to biological data, we take an operative approach that relies on measures that could in principle be obtained experimentally. We thus consider two of these pillars: either the neural activity of the network while performing the task (formal description), or its predictions for unexpected stimuli (purpose).
Space of solutions in a 2D RNN task
We first study the space of solutions in an extremely simple scenario that nevertheless shows qualita- tively different solutions. We consider 2-neuron continuous-time network (Figure IA), in which the state the network æ E IR² evolves according to:
á + Wo(x), x(0)
00]
For the task, we require that (TT)
- 1 Figure 1 IB). We choose & = tanh and = 10.
Accordingly we define loss:
= x (T) 1
The system has four parameters, given by the elements W E R2x2, We train 10000 randomly initialized networks to minimize L and discover solutions with qualitatively different dynamics. We select three representative examples. (Figure DDD, The first example implements a stable fixed point near the target, the second implements limit cycle near the target, and the third exhibits transient amplification passing through the target before gradually decaying to zero.
To visualize the space of solutions, we consider 2D plane in the 4D parameter space containing three aforementioned solutions. We find that this plane explains 94 percent of variance of all solutions (compared to 95 percent by the first two principal components), and is thus representative description of solution space. The shading in Figure IC shows the loss along this plane, indicating that some solutions lie along single continuous manifold, while others inhabit discrete islands. Linearizing the dynamics at the origin allows us to obtain a bifurcation diagram on this plane (Figure IC, red and black lines), showing the existence of a bifurcation along the continuous manifold. In the simpler case of a linear network, we can also show how the discrete and continuous solution sets arise (see supplementary section 2.1)
thus conclude that the network converges to qualitatively different solutions. These can be attributed to discrete basins of attraction in parameter space and dynamical bifurcations of the system occurring within and across these basins.
Neuroscience Tasks
Guided by the results from the two-neuron network, we examine three more complex, neuroscience- inspired, tasks Figure A priori, it is not clear whether such tasks will exhibit more less variable solutions. On the one hand, a complex task might lend itself to multiple algorithmic solutions. On the other hand, complex task represents more constraints the network and hence might lead to convergence to a unique solution. More details can be found in the supplementary material subsection Section 2.1.2
In the Delayed discrimination task (21), two pulses of varying amplitudes (/1, /2 E 10) are presented, separated by a varying delay (l. E [5. 24)) F 2A. The lower panel shows the correct output in response various stimulus combinations, which is independent of la and partitions the (f1, f2) plane.

---

A
B
W.
x: )
Stable
Non-trivial
Origin
0
W 12
W
E
Stable
Origin
LC
LC
x
W2
Wi
E
F
2
loggoog)
0
-2
2
22
-2
Figure I: Two-neuron network. Network architecture, with the four trainable parameters. The task defined in phase space. The initial state fixed (green). The task is for the network to be at final state (red) at time = 10. C 2-D slice of the 4-D parameter space containing three selected solutions (black markers). Heatmap denotes task loss evaluated along the slice. Overlaid is bifurcation diagram along the slice; lines indicate dynamical bifurcations and text indicates regions with stable origin, non-trivial fixed point attractors and limit cycles (LC). D, E, Phase portraits of the three selected solutions marked in C, trajectories taken by each the networks during the task < T, thick purple) and subsequently T. thin purple). Heatmaps denote speed of dynamics q 2 || 2 30
In the Interval discrimination task (22). two pulses of equal amplitude presented at times and I + l2, where 4, l2 E [10, and I * l2. The network should produce an output pulse whos sign indicates whether (2 1 not Figure 2B. The lower panel once more shows the desired output.
Finally, in the Interval reproduction task (20) the network receives two input pulses - Ready and Set - separated by fin time steps. The task is to generate output pulse tout tir. time steps after the Set pulse. The training intervals were drawn from a uniform distribution 4.x E [20, 50). The lower panel shows the desired output, where having only parameter defining the trials (t.,,) allows to display the entire trial output on each line. Trials aligned to the Ready pulse (Figure 28, red). This results in the Set pulse (yellow) forming line with slope 1.0 and the Go pulse (green) line with slope 0.5. Figure 2C shows the output of trained network matching this pattern.
For each task we trained 400 Vanilla networks, with N (20, 30, 40, 40,50) hidden units. See the Supplementary material subsectior[Section 2.1.3 for more details regarding the training process, as well as results from GRU and LSTM networks trained on the interval reproduction task.

---

B
Delayed discrimination
Interval discrimination
Interval reproduction
f
Couc
D
E
F
delay
t2
Time in trial
f2
The network is presented with two pulses with amplitudes fi and f2 respectively. separated by a variable delay, and should respond with pulse with an amplitude equal to sign(fi f2). Bottom: The desired output for all training data, for each delay, fi, f2 the color of the 3D matrix is the expected response. B Interval discrimination task. Top: The network is presented with two pulses with unit amplitude, that arrive at times t and l2 4. In response, should produce a pulse with amplitude that is equal to sign(12 - -4). Bottom: Desired output for all 4 and l2 combinations. C Interval reproduction (Ready-Set-Go task. Top: The network is presented with two pulses (Ready and Set, provided through separate input channels) separated by tin stees The desired output is Go pulse, delayed by tout thn steps from the Set pulse. Bottom: Desired output for all trials, with Ready, Set and Go depicted red, yellow and green respectively.
Figure 2: The three tasks. A Delayed discrimination task. Top:
5.1 Multiple neural dynamics solve the same task
We demonstrate the diversity of neural dynamics in different solutions by showing couple of examples from each task. The left side Figure 3/B shows the activity of two networks solving the interval reproduction task projected onto the first principal components. It is possible to infer two distinct algorithms for solving the timing problem from these plots. Because the task requires counting time in distinct epochs from Ready Set and from Set Go to explain the algorithms. The network in panel B uses the same phase space trajectory for both epochs, effectively counting from the maximal delay (tin = 50) downwards upon the Ready pulse, and moving to an earlier point upon the Set pulse. This is in contrast to the network in panel A, in which the Set pulse leads to different area in phase space. Furthermore, the almost circular trajectory of the Ready-Set epoch hints at the oscillatory behavior that will be discussed below.
focus on these
The delayed discrimination task also admits multiple solutions. In this case, consider activity during the delay period. Figure 3CD show several delay trajectories, corresponding to different fi values. The network in panel D converges approximate fixed points for each such value, while the network in panel C converges to limit cycles. Note that the network was trained with variable delays BI). and hence the second stimulus arrives at random phases of these cycles. Nevertheless, both networks perform well on the training set.
Interval discrimination contains two timing-epochs from the onset of the trial to the first pulse, and between the two pulses. The network in panel E uses the same trajectory in phase space for both epochs, similar panel of the interval reproduction task. The second example exhibits both oscillatory behavior and distinct trajectories for the two phases.
Taken together, these examples show that networks trained with identical hyperparameters, and reaching similar performance nevertheless develop qualitatively different neural dynamics solve the same task.
5

---

5.2 Extrapolation
On the behavioral side, we challenge networks with stimuli that are outside of the trained distribution. Since each of our tasks is parametric, testing the networks on extrapolation while increasing each parameter is a natural choice. The middle column of Figure shows the results of this challenge for the various networks, which often shows traces of the neural diversity described above.
The almost circular Ready-Set neural trajectory of panel A apparently results in a limit cycle, as revealed by the extrapolation plot. Similarly, the fact that the Ready-Set trajectory of panel B leads directly to the Go pulse is reflected in the vertical Go line that precedes the Set pulse in the extrapolation plot. Other features, such as the fact that Set pulse delivered after the Go pulse results in second output, are only revealed through extrapolation and cannot be deduced from the PCA of neural activity during the training set.
In the delayed discrimination task, the oscillations shown in panel C were not reflected in the output on the training region. Extrapolating to larger amplitudes and delays, however, reveals output oscillations as depicted on the top face of the extrapolation cube.
In the interval discrimination task (panels E,F), the relation between neural trajectories and extrap- olation behavior is less clear. But in this case, as in all others, the striking differences between different solutions are manifested both in the training neural activity and in the response to behavioral challenges.
5.3 Reduced Dynamics
Part of the motivation for looking into extrapolation patterns uncovering the algorithm that networks use to solve the task. Within the framework of neural dynamics (18 this corresponds to mapping
the dynamical objects used by the network and their relation to behavioral inputs and outputs. Previous works focused on fixed point topology to achieve this goal (30) 24 14. The tasks considered in the present work are mostly dependent on transient dynamics and hence require a different approach. To this end, we developed tool to compress the state space while preserving the essential dynamics and the behavioral information. Briefly, the method generates trajectories by combining long stretches of autonomous dynamics with behavioral inputs. The trajectories are then merged and compressed into a graph. Each node represents areas in phase space that have a given output and share the same input-dependent past and future. The edges represent autonomous dynamics (black) or the various inputs (colors). The full details of the algorithm are in the supplementary material subsection 2.2. The left column Figuue 3] shows such reduced dynamics for all the example networks. Examining these graphs can reveal the dynamics driving extrapolation patterns.
The network of panel A has a limit cycle, which is depicted by the red node that has a black self-loop. Furthermore, the graph shows that after the Go pulse, the network can still respond to an additional Set pulse. The fact that the network of panel B utilizes the same trajectory for both epochs is reflected in the yellow self-loop in the corresponding graph. The graph also shows that a Go pulse can occur before the Set pulse.
The delayed discrimination task has more input types and therefore was less compressed by the algorithm. As result, the graphs can show both the logic of comparison during the training set and the extrapolation patterns with the limit cycles. A Similar scenario holds for the interval discrimination task.
5.4 A space of solutions
The examples described above suggest that similar to the two-neuron case, trained networks converge into distinct solutions which can be characterized both by neural signatures and by behavioral ones. In the case of the two-neuron network, the parameter space is only -ou--dimensional, and most of the solutions are in a two-dimensional subspace. Hence, we could span most of it and understand the solution types via the linearized dynamics around the origin. In the more complex tasks, the reduced dynamics graphs serve as method to characterize the space of solutions. We computed the reduced dynamics for all 400 networks of each task and found that they only contain few different graphs. Figure 4 shows the number of networks of each type for the interval reproduction task. The four major graphs and their corresponding representative extrapolation plots are shown in panels A-D.

---

5.5 Inferring reduced dynamics from neural activity
The reduced dynamics described above are obtained by simulating the networks for very long times, to obtain their asymptotic behavior. In neuroscience setting, such information is not readily available. Is it possible to infer this asymptotic behavior from neural activity during the training set? To answer this question, extracted set of features, as detailed in the supplementary material. The features mostly measure the relationship between neural activity in the different epochs of the task. Other features measure geometrical properties of single epoch, as in the curvature of the Ready-Set trajectory, which was inspired by networks like Figure 3/. While we tried to include features that are natural descriptors of the neural activity, there some arbitrariness in our choice. We chose to on the side of including more features, and later rely on cross-vale coos-vvaliationn to avoid overfitting in our predictions (32).
For each task, we trained and evaluated a classifier to predict the reduced dynamics of the pool of networks from the neural features. Figure 4F shows the resulting confusion matrix for the interval reproduction task (other tasks are in the supplementary material Figures 8 [and Cohen's kappa (33) was used to summarize the prediction quality, showing an above-chance performance.
maierialFiguues
5.6 Different architecture and tasks
The diversity of solutions reported here arise from identical hyperparameters and random initialization. To probe the biases introduced by changing hyperparameters, we tested the effect of varying network architecture for the interval reproduction task. The supplementary material shows results from training this task using LSTM and GRU networks. We find that some solution classes are shared between architectures, and some only appear in certain architectures (Figure 5A).. Furthermore, these choices bias the solutions, as reflected in the different histograms. For instance, the example of Figure 5B did not occur in any of the 400 Vanilla networks, but was the most common type in GRU networks. Despite these biases, there are solutions that appear under all architectures, as exemplified in Figure SC.D,E.
The chosen tasks so far emphasized variability. We also examined a context-dependent integration task, which was previously shown to exhibit universal solution (24). The supplementary material shows an analysis of the response of networks to various behavioral challenges. We find that there is considerable variability in the response of networks to challenges, but of quantitative rather than a qualitative nature. Furthermore, we also quantified various aspects of neural variability in these networks. We did not find significant correlations between the various behavioral and neural measures, suggesting that there are many axes of individuality for these networks.
6 Discussion
In recent years, trained RNNs were successfully used to model biological circuits. Specifically, these networks converged to solutions that were similar to those of their biological counterparts. This observation is puzzling for any complex system and particularly regarding the brain, complex biological system which variability the rule rather than the exception. Inspired by this puzzle, we study the space of solutions of RNNs for both a simple two-neuron network and three neuroscience- inspired tasks. Through an interplay between observation, analysis, and perturbation in the form of extrapolation, we discovered for each task qualitatively different solutions. By analyzing the neural activity of these networks, we observed that the variety behavioral phenotypes originate from only a handful of distinct dynamical mechanisms. To affirm these observations and describe the essence of each solution, we introduced tool that summarizes the network computation in a reduced- reduced-dynamics graph that contains the essence of the dynamics they relate to behavioral inputs and outputs. By extracting neural features from the training activity, we showed that the solution type can be partially predicted from experimentally accessible measurements.
We show using a 2D continuous- time RNN that qualitatively different dynamical topologies can arise in the context of a single task and that learning can find different solutions depending on initialization. In a related study, the authors analyze a low-D continuous-time GRU and find large diversity of dynamical topologies (34). They find that the 2D GRU finds topologically different solutions to variants of a working memory task depending on the input statisties. Such modification to the task, although subtle, modifies the loss landscape and learning dynamics. In our work emphasize
7

---

that such topologiaal diversity of solutions can arise for the same task, that is to say within the same loss landscape. Both our work and theirs identify the implications of topological diversity for generalization.
Asking seemingly trivial question "what is solution?", highlights a complex and vital topic. Even though artificial and biological networks are being compared regularly, and there is a common intuitive understanding of what properties are relevant for this comparison, the precise definitions of "solution" or "mechanism" are rarely discussed (23). Is dynamical similarity within the training regime sufficient for asserting two solutions are alike? What the proper way to test subjects' performance on learned task? Can we answer these questions while considering the specifics of each task? Under what conditions can we safely assume that there exists a universal solution? We believe that every study that models neural circuits should be explicit about such meta-concern, In this work, we charted solutions with an operational approach in mind and considered solutions as different as long as they qualitatively differ either in their neural activity within the training set or in their behavioral prediction on extrapolation trials. The reduced dynamies tool is our attempt to describe and differentiate solutions according to the criteria that we consider important.
Our work has several limitations that should be noted. We focused on the variability arising with identical hyperparameters and therefore did not systematically explore the effect of changing them. Specifically, how network size, learning algorithm, and choice of architecture bias the solutions remains to be explored.
The space of solutions for given task is, above all else, property of the task itself. We showed examples of three tasks that have substantial variability, and of one task (context dependent integration) that has much less variability. Yet, we do not know why some tasks admit multiple solutions and others do not. We conjecture that the tasks presented in (24) require an informative output at all points in time, whereas the tasks we considered require an informative output only once at the end of the trial.
The seduced-dynanee
tool relies on set of trajectories regions in phase space that are reachable with task cons-consittent inputs (similar to (30). While exploring choice may limit the description obtained by tool. Finally,
starting point. We opted include
the full phase space is not feasible, the space of solutions was described for networks that have already learned to perform a task. remains be seen whether it can be used a map in which understand how the process of learning itself takes place.
To conclude, we found that RNNs can produce diverse set of solutions to the same computational tasks. These solutions represent distinct algorithms and are supported by corresponding dynamical objects. The solutions are isolated in parameter space, causing the initial conditions to largely determine the outcome. Furthermore, we showed that experimentally accessible tools can be used to indirectly characterize the asymptotic properties of the solutions. We believe that exploring the space of solutions can advance neuroscience, machine learning, and their intersection - making more rigorous comparisons of models to data.
Acknowledgments and Disclosure of Funding
This work supported in part by the Israeli Science Foundation (grant number 346/16, OB) and by a Rappaport institute thematic grant (OB).
References
[1] A. D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen,
J. Deaton, J. Eisenstein, M. D. Hoffman, F. Hormozdiari, N. Houlsby, S. Hou, G. Jerfel, A. Karthikesalingam, M. Lucic, Y. Ma, McLean, D. Mincu, A. Mitani, A. Montanari, Z. Nado,
V. Natarajan, C. Nielson, F. Osborne, R. Raman, K. Ramasamy, R. Sayres, J. Schrouff,
M. Seneviratne, S. Sequeira, H. Suresh, V. Veitch, M. Vladymyrov, X. Wang, K. Webster, Yadlowsky, Yun, X. Zhai, and D. Sculley. Underspecificatioon Presents Challenges for Credibility in Modern Machine Learning. [Online]. Available: http://arri..rrggaa//011...3..
[2] C. K. McIntyre, I.. K. Marriott, and P. F. Gold, "Patterns of brain acetylcholine release predict individual differences in preferred learning strategies in rats," vol. 79, no. 2. pp. 177-183.
8

---

[Online]. Available: http:///ww...aa.i....oo.oooooooa
[3] G. Gholamrezaei and I. Q. Whishaw, "Individual differences in skilled reaching for food related to increased number of gestures: Evidence for goal and habit learning of skilled reaching." vol. 123, no. 4. pp. 863-874. [Online]. Available:
[4] E. Patitucei, A. J. D. Nelson, D. M. Dwyer, and R. C. Honey, "The origins of individual general-process differences in how learning is expressed in rats: A general process perspective." vol. 42, no. 4,
pp- 313-324. [Online]. Available: http:///w....oooooooooooootmtmtd
[5] G. Kempermann, "Environmental enrichment, new neurons and the neurobiology of individuality," vol. 20, no. pp. 235-245. [Online]. Available: http://www.naturr.commaartt s41583-019-0120-
[6] S. Musall, Urai, D. Sussillo, and A. Churchland, "Harnessing behavioral diversity to understand neural computations for cognition," " vol. 58, 229-2362 238.. [Online]. Available:
13300028
[7] T. Kurikawa, T. Haga, T. Handa, R. Harukuni, and T. Fukai, "Neuronal stability in medial
frontal cortex sets individual variability in decision-making." vol. 21, no. 12, pp. 1764-1773. [Online]. Available: http::/www..ntt
www.nna.ttmmmmanca http://www.nauur..oom/aaa
[8] Morita, H. Kang, Wolfe, S. Jadhav, and D. E. Feldman, "Psychometric Curve and Behavioral Strategies for Whisker-Basse Based Texture Discrimination in Rats," vol. 6, no. 6, e20437. [Online]. Available:
I/j.ri3alll p0ne.00204377
[9] L. Castro and E. A. Wasserman, "Humans deploy diverse strategies in learning same- different discrimination tasks," vol. 93, pp. 125-139. [Online]. Available: hhttps
/linkinghub.elseiierccoom httpa/vwww..oooeeeeeeee///////ss 1202602
[10] L. Lebovich, Darshan, Y. Lavi, D. Hansel, and Loewenstein, "Idiosyneratic choice bias naturally emerges from intrinsic stochasticity in neuronal dynamics," vol. 3, no. 11, pp. 1190-1202. [Online]. Available:
[11] A. Gilad, Y. Gallero-Salas, D. Groos, and F. Helmchen, "Behavioral Strategy Determines Frontal or Posterior Location of Short- Term Memory Neocortex." vol. 99, no. pp. 814-8284-0 [Online]. Available:
[12] O. Barak, "Recurrent neural networks versatile tools of neuroscience reauroscienneerssacchh... vol. 46, pp.
1-6. [Online]. Available:
[13] O. Barak, D. Sussillo, R. Romo, M. Tsodyks, and L. Abbott, "From fixed points to chaos: Three models of delayed discrimination." vol. 103, pp. 214-222 222. [Online]. Available: 30000129
[14] V. Mante, D. Sussillo, K. V. Shenoy, and W. T. Newsome, "Context-dependee" computation by recurrent dynamics in prefrontal cortex," " vol. 503, no. 7474, pp. 78 84. [Online]. Available:
12742
[15] D. Sussillo, M. M. Churchland, M. Kaufman, and K. V. Shenoy, "A neural network that finds naturalistic solution for the production of muscle activity." vol. 18, no. 7, pp. 1025-1033. [Online]. Available: http://wwwnnauure hwtp://wwauee.oom//rrcc/e//emnae 4042
[16] G. R. Yang. M. R. Joglekar, H. F. Song, W. T. Newsome, and X.-J Wang, "Task representations in neural networks trained to perform many cognitive tasks." " vol. 22, no. 2, pp. 297-306.
[Online]. Available: http::/wwwwnattrr.ccomm http://wwntuur..oooaaaa hat.aree..oo//atee/////11331111.
[17] M. Schrimpf, Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher,
J. Tenenbaum, and E. Fedorenko, "The neural architecture of language: Integrative reverse-engineerring converges on a model for predictive processing." [Online]. Available:

---

[18] S. Vyas, M. D. Golub, D. Sussillo, and K. V. Shenoy. "Computation Through Neural Population Dynamics." vol. 43, no. 1, pp. 249-275. [Online]. Available:
http://wwwaammalrreeew.....//// http://rrrddo/o11///////eeeeeeee 1 (9-44155
[19] E. D. Remington, D. Narain, E. A. Hosseini, and M. Jazayeri, "Flexible Sensorimotor
Computations through Rapid Reconfiguration of Cortical Dynamics," " vol. 98, no. 5, pp. 1005-
1019.e5. [Online]. Available: https://lnnkinghub... elweeeesssooeeieeee.ee pttee//ccoo//eereee/////ooooooo0
[20] M. Jazayeri and M. N. Shadlen, "Temporal context calibrates interval timing," vol. 13, no. 8, pp.
1026. [Online]. Available: http:://www.atthee http://wwwwtwtt.c.mmm www.:naau...o//artlleeee..o wwtp///acoc/om/m..222..
[21] R. Romo, C. D. Brody, A. Hernández, and L. Lemus, "Neuronal correlates of parametric working memory in the prefrontal cortex," " vol. 399, no. 6735, pp. 470-473. [Online]. Available:
http://wwww.aauurr.co////tcc////
[22] Z. Bi and C. Zhou, "Understanding the computation of time using neural network models," vol.
117. no. 19, pp. 1003--10540 [Online]. Available: http://www.ppasssro/oooooo/oo/// pnas. 1921699111
[23] O. Guest and A. E. Martin, "On logical inference over brains, behaviour, and artificial neural networks." [Online]. Available: https://ost..o//bmegg
[24] N. Maheswaranathan, A. Williams, M. Golub, S. Ganguli, and D. Sussillo, "Universality and individuality in neural dynamics across large populations of recurrent networks," in Advances in Neurul Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, http://cceeeeeeeiiiiircccccp//pp
Inc.. pp. 15 629-15 641. [Online]. Available: 515547706777655388111111... Papp
[25] J. Mehrer, C. J. Spoerer, N. Kriegeskorte, and T. C. Kietzmann, "Individual differences among deep neural network models," vol. 11, no. 1, p. 5725. [Online]. Available:
19632-
[26] E. Ghazizadeh and S. Ching, "Slow manifolds within network dynamics encode working memory efficiently and robustly," vol. 17, no. 9, e1009366. [Online]. Available:
wwtp://plllo......11/oooooooooe 10099093366
[27] E. Marder and A. L. Taylor. "Multiple models to capture the variability in biological neurons and networks," vol. 14, no. 2, pp. 133-138. [Online]. Available: http
www..nature.co//arrillcsssm
[28] B. B. Machta, R. Chachra, M. Transtrum, and J. P. Sethna, "Parameter Space Compression Underlies Emergent Theories and Predictive Models," vol. 342, no. 6158, pp. 604-607. [Online]. Available:
htc.:////.....1/2//cccceeeeeee.e 1238723
[29] F., Andrea, "Aristotle Causality." [Online]. Available: Spp2010/ennticss/aiitttlle---s causallltllya
[30] D. Sussillo and O. Barak, "Opening the Black Box: Low-Dimensional Dynamics in High-Dimensione Recurrent Neural Networks," vol. 25, no. 3. pp. 626 649. [Online].
Available: http://wwwwwmippessosomo/o////// 1wtp://011112///...... CaCCCCC0___009000 00409
[31] A. E. Orhan and W. J. Ma, "A diverse range of factors affect the nature of neural representations underlying short-term memory," vol. 22, no. pp. 275-283, [Online]. Available:
[32] R. M. Khan, C.-H. Luk, A. Flinker, A. Aggarwal, H. Lapid, R. Haddad, and N. Sobel, "Predicting Odor Pleasantness from Odorant Structure: Pleasantness as Reflection of the Physical World," vol. 27, no. 37, pp. 10015-10023. [Online]. Available:
[33] M. I.. McHugh "Interrater reliability: The kappa statistic," vol. 22, no. 3. pp. 276-282.
[34] I. D. Jordan, P. A. Sokół, and I. M. Park, "Gated Recurrent Units Viewed Through the Lens of Continuous Time Dynamical Systems." " vol. 15, p. 678158. [Online]. Available:
10

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Discussion
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work basic science and does not develop any algorithm that might be used in such manner.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] github in supplement
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] In supplement
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] The effect of the random seed is part of the paper
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

---

A
Time trial
B
Time in trial
to
C
delay
D
delay
E
F
PURZUTTE SEEGRRAdS
Figure 3: Diverse solutions for the same tasks. Two example networks shown for each task (rows). Left column: two-dimensional PCA of the network activity. Middle column: Network output in response extrapolation across task parameters. Right column: the reduced dynamics of the networks. (A,B) Solutions to the time production task. The PCA plots show the activity from the Ready pulse (red dot and black line), as well from the Set pulse (yellow) up to the beginning of the output production, for three different task parameters (colorbar). Note that in A the Ready-Set epoch separated from the Set-Go, whereas in these epochs converge. (C,D) Solutions the delayed discrimination task. The PCA plots show the activity between the two input pulses, for various fi amplitudes. The memory of the different amplitudes is kept either via limit cycles (C) or slow points (D). (E,F) Solutions to the interval discrimination task. The PCA plots show activity from trial onset (black dot and line), well as from the first pulse (yellow) until the maximal delay, for three different 4 values (colorbar). Note the convergence of the two epochs to the same trajectory in E, similar to B.
12

---

A
8
B
Reduced-Dynameen type
5
true
D
S
-10
Time
predicted
Figure 4: The space of solutions for the interval reproduction task. A-D Representative extrapolation plots (left) and reduced dynamics graphs (right) for the four most common solution types. E Distribution solution types for the 400 networks trained. The four solutions shown account for 93% of the networks. J' Neural features obtained during the training can partially predict the solution type that includes extrapolation dynamics. The confusion matrix shows the result of this prediction. Note that the first solution types are mixed this prediction, but their dynamics during the training intervals is similar and they only differ in the dynamics after the Go pulse.
13

---

A
500
Vands CRU LETI
250
COurt
C
100
00
D
E
Figure 5: Architecture biases, but does determine the solution. A A histogram of the six most common reduced dynamics across all three architectures for the time-reproduction task, shown by stacking the urchitecture-specife histograms on top of one another. B, C The reduced dynamics (left) and the extrapolation patterns (right) of the second and third most common solutions across all architectures, but rarely occur within the Vanilla networks. D, E, F The 2D PCA of the dynamies of three networks from all three architectures (Vanilla, GRU, LSTM), that reach the solution shown in panel
14


Reviews:
[{"review_id": "wLNkmxB4jhN", "reviewer": "Reviewer_teJ3", "summary": "The authors study the solutions  RNNs find and characterize them based on extrapolation and recurrent dynamics. They primarily consider two cases -- a simple two neuron network performing a memory task, and larger networks performing a Ready-Set-Go task. They show that the final solution depends on the initial connectivity, but not so much on the hyperparameters of learning. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Update:** Based on the authors response, and new results mentioned, I have updated my score to reflect my more positive evaluation of the paper.\n\nOverall, this is very interesting and nicely done work but the paper has some flaws that resulted in my recommendation of rejection.\n\n## Originality: \nThere are other works characterizing RNN dynamics, e.g. (Maheswaranathan et al. 2019) which is cited in the related work. All the aspects where this work differs is not very clear (might be useful to add), but one original contribution of this paper seems to be the use of extrapolation for characterizing the solution, which is an interesting idea. Another contribution is to show that initial connectivity and not hyperparameters influence the final solution (although it remains to be seen how generally applicable this is).\n\n## Quality: \nThe experiments performed in the paper are well defined, and the conclusions reached are reasonable. But the take-home message of the paper is not clear. Concrete comparisons with biological data would have strengthened the paper quite a lot.\n\nOne specific comment: It's strange that the parameters are trained in the PC space for the 2-neuron network as described in the caption for Fig. 1. Training in the original weight space and projecting onto the PC space just for plotting would have been more meaningful.\n\n## Clarity: \nThe writing is reasonably clear, but there are some gaps in the explanations:\n* How are the classes of networks defined? Is there a quantitative measure for this? Are the labels done manually? This key detail is not very clearly spelled out.\n* Sec. 4.3: Details about the networks being tested are not given. How large are these networks, and of what type?\n* Fig. 5 has no axis labels and explanation of what's plotted\n* l.219: Not sure reference [18] makes sense in the context of that sentence.\n\nMinor:\n* Fig. 2A: $t_p$ should be $t_s$?\n* Lines 128-135: Networks are referred using A,B,C,D while the figures are labelled D-G causing some confusion.\n* l.152: Figure 2A should be Figure 2D.\n\n## Significance: \nUnderstanding the space of solutions of RNNs is a very important problem, and this paper adds a couple of insights related to this.\nAnother key contribution of the paper is to show the variety of solutions that emerge for the considered tasks -- highlighting the selection bias present in papers that show activity similar to biology emerging in networks trained to do specific tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have a fairly good discussion of the limitations of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "OTr2qiMHqY", "reviewer": "Reviewer_H1DK", "summary": "This paper analyzes the topology of the solutions of RNNs trained on time counting tasks, revealing a rich set of possible dynamical solutions and architectural biases that affect the generalization properties of the solution. It uses numerical experiments to illustrate the diversity of solutions that can arise for tasks where the solution relies on slow transients rather than fixed points for performing the task and highlights some of the difficulties in performing a systematic analysis in such contexts. This is potentially relevant for understanding across subject variability in animals performing such tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall evaluation: An interesting idea but the lack of systematicity in the approach diminishes the value and interpretability of the results. More work needed before publication. \n\nMajor concerns: While liking the general idea, there's not a sense that we have really understood the phenomenology by the end of the paper.  The analysis feels incomplete, and the described topological structure somewhat anecdotal. I would have thought that the 2-neuron network toy example would have provided more analytical backing for the arguments but is seems to also primarily rely on simulations to make the points. Are the topological features in fig3 hand designed by the scientist as a human interpretation of continual trajectories or could this topological structure be identified algorithmically? For the last part when identifying dynamic signatures of the solution that can be used to differentiate various algorithmic solutions (which are defined 'manually' by the scientist based on the same trajectories), I am unsure what have we actually learned there. Is there a way to define classes of solutions straight from data and to which degree are these methods applicable to neural data (i.e. are these quantities that one could reasonably estimate from limited neural recordings?). Also not sure what the take home message is for the behavioral predictions in fig 6. \n\nSignificance: highlights the computational distinctions across 'working memory'-like tasks and furthers the arguments for a topological/algorithmic understanding of RNNs computations. Missed opportunities in relating to across animal variability in similar neuroscience experiments.\n\nOriginality: Since the number of degrees of freedom in trained RNNs is often much larger than the task constraints, there is an increasing interest in characterizing the nature of the solutions trained by RNNs and their generalization properties, with relevance for both neuroscience and machine learning. In particular, Susillo et al have shown that certain memory task show a certain form of universality, where the qualitative nature of the solution is more or less the same regardless choices of architecture and hyperparameters. This paper uses similar ideas to show that other tasks that count time are less constrained in the nature of their solution with several distinctive algorithms.\n\nClarity: The abstract and introduction are very well written, but the related literature lacks substance and the discussion is poorly structured and needs a fair bit of work.  The quality of the figures is poor, with systematically missing axis labels, inconsistent colors across panels, etc; it would also greatly help to have in figure legends for different colors. the captions help somewhat to make sense of the lot but makes the reader's life unnecessarily difficult.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No obvious negative social impact. The limitations of the approach are touched upon in the discussion but not very satisfactorily.", "review_text": "", "extra_scores": {"ethical_concerns": "No  ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "5mSEFMmOUtL", "reviewer": "Reviewer_Q7WG", "summary": "This study proposes an approach for characterising the space of solutions obtained by fitting neural networks to a couple of tasks (one simple but general task and one neuroscience-related task). The analyses suggest that network initialisation and architecture are two key factors for the degeneracy of solutions obtained when fitting neural networks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe novelty of the study resides in the development of a set of analyses to characterise the space of neural network solutions. Previous work is appropriately cited and it is clear how this work differs from previous contributions.\n\n### Quality\n\nThe paper is technically sound, and claims are appropriately backed by empirical evaluation.\n\n\n### Clarity\n\nThe paper is clearly written, and provides enough information for an expert reader to understand all the steps to reproduce the results. There is just a minor typo I would like to point out: in lines 128-134, the authors reference the networks as A to D, and I believe these references to be wrong.\n\n\n### Significance\n\nI believe the developed analyses would be of interest to the neuroscience community, in particular applied to real neuroscience datasets. However, the conclusions drawn from the analyses are somewhat unsurprising from the Machine Learning point of view, as it is well known that neural network fitting is a problem with multiple local minima, and that this depends on the initialisation and architecture. In fact, it is common and good practice in Machine Learning to train networks several times, with multiple seeds, initialisations and hyperparameters, in order to assess the robustness of the drawn conclusions. In order to improve the significance and impact of the manuscript, I would recommend the authors to apply their set of analyses to a Machine Learning task, with the hope of getting new insights e.g. in terms of optimisation (note that the manuscript is one page shorter than the limit, and thus would have the space for extra analyses).\n\n\n### After rebuttal\n\nThank you for the careful answer to all reviews. After reading all the reviews and authors' rebuttal, I believe the authors made an effort to address all the reviews, and in my opinion, to some satisfaction. I increased my score to better reflect the quality and significance of the work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2XMfqkudqVn", "reviewer": "Reviewer_vfv5", "summary": "This paper makes two main contributions. First, the authors study a couple of examples to show that the learned structure and dynamics of an RNN are not completely determined by the task on which it was trained -- legit solutions form a vast space, and where an RNN lands depends on where it starts. Second, on the same examples, the authors set out to dissect how the different solutions differ, both in the algorithmic strategies they employ, and in the realisation of these strategies in the network's internal dynamics. The most novel result is that aspects of the former can be predicted by features of the latter. The first point about degeneracy of solutions is not exactly news, but it is very nicely illustrated here on examples, and it is an important reminder to the field that one should be careful in interpreting the emerging dynamics of trained RNNs as brain-like solutions. The second is potentially of greater practical importance to the field, and I commend the authors for even trying to attack such a difficult problem of relating dynamical motifs at the neural level to \"algorithmic strategies\"! However, I have many issues with the clarity of the paper, and struggle to understand if the methodology developed in this paper will be of any use beyond the particular Ready-set-go task considered.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have found the paper difficult to understand:\n\n- the abstract is cryptic, with intriguing sentences such as \"[...] by testing the network's ability to extrapolate, as a perturbation to a system often reveals [...]\" → not sure a priori what the link might be between extrapolating and responding to a perturbation; similarly, \"we relate extrapolation patterns to specific dynamical objects\" → too vague to be useful; similarly, \"Using a set of features, we define a space of solutions.\" → again, too vague to be a useful summary.\n\n- In section 4.3, the reader has to work out for themselves that panels D, E, F and G of Figure 2 refer to networks A, B, C, and D in the main text.\n\n- Figure 3: what are we looking at?? What is the meaning of these circles and squared, open vs filled, coloured vs white? The corresponding Section 4.4 is very opaque -- there are lengthy descriptions of the various possibilities, but no sense of how they were arrived at. Is there a general approach to categorising strategies? \"shows the salient locations in phase space\" - what does that mean? \"enter a slow area\" -- where do we see this? \"some networks emit a go pulse\" -- isn't it the case that all successfully trained networks will emit a go pulse? At this point I am rather lost.\n\n- Figure 5: what's on the x and y axes?\n \n------\n\nMinor:\n\n- Figure 2 : t_s corresponding to each shade of blue in Fig2 D-G should be labelled\n- Figure 2H seems a bit arbitrary; why not compute the participation ratio or something?\n- Section 4.4: \"there are no guarantees\" -- for what?\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is not addressed at all, but I think we can live with that given the paper's main topic.\nThe limitations are discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present in this work an analysis of the multiple solutions that can be identified by a recurrent neural network trained on a task. in particular, the authors discuss the implications for computational neuroscience studies that leverage such tools. The reviewers all agree that the work was well presented, and in particular strengthened by a second analysis performed in response to the reviewer comments that remove a potential confound of timing. While the scores themselves seem to be borderline, the topic is very timely and the results appear to be highly relevant to the use of ML in Neuroscience. I therefore recommend this work (with the addition of the extra analysis) be accepted to NeurIPS. 
Decision: Accept (Poster)