Charting and Navigating the Space of Solutions for Recurrent Neural Networks
Authors: Elia Turner
Kabir Vinay Dabholkar
Omri Barak
Venue: NeurIPS 2021 Poster (2021)
Keywords: RNN
underspecification
variability
space of solutions
neuroscience
reverse engineering
task optimized networks

Abstract: In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating under the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and machine learning challenge this assumption. Animals can approach a given task with a variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that every task is associated with a space of solutions. To date, the structure of this space is not understood, limiting the approach of comparing RNNs with neural data.
Here, we characterize the space of solutions associated with various tasks. We first study a simple two-neuron network on a task that leads to multiple solutions. We trace the nature of the final solution back to the network’s initial connectivity and identify discrete dynamical regimes that underlie this diversity. We then examine three neuroscience-inspired tasks: Delayed discrimination, Interval discrimination, and Time reproduction. For each task, we find a rich set of solutions. One layer of variability can be found directly in the neural activity of the networks. An additional layer is uncovered by testing the trained networks' ability to extrapolate, as a perturbation to a system often reveals hidden structure. Furthermore, we relate extrapolation patterns to specific dynamical objects and effective algorithms found by the networks. We introduce a tool to derive the reduced dynamics of networks by generating a compact directed graph describing the essence of the dynamics with regards to behavioral inputs and outputs. Using this representation, we can partition the solutions to each task into a handful of types and show that neural features can partially predict them.
Taken together, our results shed light on the concept of the space of solutions and its uses both in Machine learning and in Neuroscience.

CHARTING AND NAVIGATING THE SPACE OF SOLUTIONS FOR RECURRENT NEURAL NETWORKS
Elia Turner
Kabir Dabholkar
Department of Mathematics Technion, Israel Institute of Technology el iliaturnerocamus.. technion onniiin.oee ac.
Department of Mathematics
Technion, Israel Institute of Technology kabir@campus technion. ac il
Omri Barak
Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion, Israel Institute of Technology omri. warribarrakgmaiil.. com
Abstract
In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating der the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and machine learning challenge this assumption. Animals can approach given task with variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that every task is associated with space of solutions. To date, the structure of this space not understood, limiting the approach of comparing RNNs with neural data. Here, we characterize the space of solutions associated with various tasks. We first study simple two-neuron network on a task that leads to multiple solutions. We trace the nature of the final solution back to the network's 's initial connectivity and identify discrete dynamical regimes that underlie this diversity. We then examine three neuroscience-inspinn terval discrimination, and Time reproduction. For each task, we find a rich set solutions. One layer of variability can be found directly in the neural activity of the networks. An additional layer is uncovered by testing the trained networks' ability extrapolate, Furthermore, we relate extrapolation patterns to specific dynamical objects and effective algorithms found by the networks. We introduce a tool to derive the reduced dynamics networks by generating a compact directed graph describing the essence of the dynamics with regards to behavioral inputs and outputs. Using this representation, we can partition the solutions to each task into handful types and show that neural features can partially predict them. Taken together, our results shed light on the concept of the space of solutions and its uses both in Machine learning and in Neuroscience.
tasks: Delayed discrimination, In-
perturbation to system often reveals hidden structure.
Introduction
Modern machine learning operates in over-parameterized regime, implying that many different parameter-sets can achieve low error on a given training set (I). This observation implies that for every task, there exists a space of solutions that implement it. What the properties of such a solution space? Are networks able to learn solutions that capture the intended underlying
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

phenomena or do they reach artificial shortcuts that do not generalize well? What biases networks to prefer one solution over the other? These questions remain largely unanswered. A parallel phenomenon occurs in Neuroscience. When animals are instructed to perform a task in a controlled environment, they exhibit both neural and behavioral variability, which stem from different task- strategies 2 4 5 6 7 8 9 10, 11. In Computational Neuroscience trained Recurrent Neural Networks (RNNs) are used as a tool to explain functions and mechanisms that are observed in brain dynamics (1).. In fact, various recent studies have matched the activity of trained RNNs to that of experimental recording 13 14 15 16 17 I8 19). In light of the variability that undoubtedly exists on both sides of the comparison, these results seem puzzling. In this work, we present multiple tasks for which trained RNNs produce a rich space of qualitatively different solutions. We argue that to properly use artificial networks, and RNNs in particular, models of neural circuits that support given task, it is necessary to chart the space of solutions that arises from training.
Here, we first apply this approach to simple two-neuron network and demonstrate how distinct solutions arise. We then study three tasks inspired by the neuroscience literature: interval reproduction (20), delayed discrimination (2I), and interval discrimination (22). We show that different networks with identical hyperparameters find qualitatively different solutions. We find one layer of variability within the neural activity in response to stimuli from the training set. Since by design the output of all networks is identical during training, this layer is akin to multiple realizability (23). Next, we expose additional layer of variability when we challenge the networks with inputs that outside the distribution of the training set and systematically characterize the responses. Furthermore, we manage to show that the diversity revealed with these challenging inputs corresponds to qualitatively different computations performed by the network. To chart the space of solutions, we introduce a tool that reduces the dynamics of a network into a graph that captures the essence of the computation performed. Applying it to all networks partitions the space into a handful of possible reduced dynamics. Additionally, these classes can be partially predicted using experimentally accessible neural activity obtained only in response to trained stimuli.
2 Related work
Recent work, 244 studied the effects of modeling choices over the dynamics of the trained solutions. in a few canonical tasks. They trained thousands of RNNs, while systematically controlling the hyperparameters, and analyzed the geometrical and topological aspects of each solution. They concluded that while the geometry of different solutions can vary significantly across different architectures, the underlying computation and dynamical objects are widely universal. Our results are superficially opposite to theirs. This is probably due both to the choice of tasks and to challenging networks with unexpected inputs.
The authors of (25) show that deep networks trained on vision tasks develop different internal representations, especially in the higher layers. While reaching similar conclusions on the need to use populations of networks for neuroscience comparisons, the analysis of feedforward networks naturally focuses on representations rather than on dynamics. While writing this manuscript, we were made aware of recent work by (26) that takes very similar approach to ours. Apart from the different tasks, architectures and learning rules studied there, our analysis of the two-neuron network also provides a tangible example of discrete basins of attraction in solution space. An underspecified two- dimensional epidemiological model was studied by (.. The variability solutions there, however, was continuous. Specifically, all solutions were connected on single manifold and the dynamics of the system did not undergo bifurcations along this manifold.
Underspecification was also highlighted in neuroscience models that are not based on artificial neural networks (27) and is observed more generally in complex systems (28).
3 What constitutes a solution?
Before comparing how the different networks solve the various tasks, it is worth dwelling on the concept of a solution. This is not a trivial concept and can be related to fundamental philosophical questions. Aristotle suggested (29) the four causes of understanding an object: its material form, its formal description, its efficient cause (creation process), and its final cause (purpose). We can draw an analogy to our understanding of RNN solutions, and ask: What is their architecture (material),
2

---

what is their underlying algorithm or dynamics (formal), which optimization process led to their final state (efficient), and what task do they solve (final). Because our motivation stems from comparing networks to biological data, we take an operative approach that relies on measures that could in principle be obtained experimentally. We thus consider two of these pillars: either the neural activity of the network while performing the task (formal description), or its predictions for unexpected stimuli (purpose).
Space of solutions in a 2D RNN task
We first study the space of solutions in an extremely simple scenario that nevertheless shows qualita- tively different solutions. We consider 2-neuron continuous-time network (Figure IA), in which the state the network æ E IR² evolves according to:
á + Wo(x), x(0)
00]
For the task, we require that (TT)
- 1 Figure 1 IB). We choose & = tanh and = 10.
Accordingly we define loss:
= x (T) 1
The system has four parameters, given by the elements W E R2x2, We train 10000 randomly initialized networks to minimize L and discover solutions with qualitatively different dynamics. We select three representative examples. (Figure DDD, The first example implements a stable fixed point near the target, the second implements limit cycle near the target, and the third exhibits transient amplification passing through the target before gradually decaying to zero.
To visualize the space of solutions, we consider 2D plane in the 4D parameter space containing three aforementioned solutions. We find that this plane explains 94 percent of variance of all solutions (compared to 95 percent by the first two principal components), and is thus representative description of solution space. The shading in Figure IC shows the loss along this plane, indicating that some solutions lie along single continuous manifold, while others inhabit discrete islands. Linearizing the dynamics at the origin allows us to obtain a bifurcation diagram on this plane (Figure IC, red and black lines), showing the existence of a bifurcation along the continuous manifold. In the simpler case of a linear network, we can also show how the discrete and continuous solution sets arise (see supplementary section 2.1)
thus conclude that the network converges to qualitatively different solutions. These can be attributed to discrete basins of attraction in parameter space and dynamical bifurcations of the system occurring within and across these basins.
Neuroscience Tasks
Guided by the results from the two-neuron network, we examine three more complex, neuroscience- inspired, tasks Figure A priori, it is not clear whether such tasks will exhibit more less variable solutions. On the one hand, a complex task might lend itself to multiple algorithmic solutions. On the other hand, complex task represents more constraints the network and hence might lead to convergence to a unique solution. More details can be found in the supplementary material subsection Section 2.1.2
In the Delayed discrimination task (21), two pulses of varying amplitudes (/1, f2 E 10) are presented, separated by a varying delay (l. E [5. 24)) F 2A. The lower panel shows the correct output in response various stimulus combinations, which is independent of la and partitions the (f1, f2) plane.

---

A
B
W.
x: )
Stable
Non-trivial
Origin
0
W 12
W
E
Stable
Origin
LC
LC
x
W2
Wi
E
F
2
loggoog)
0
-2
2
22
-2
Figure I: Two-neuron network. Network architecture, with the four trainable parameters. The task defined in phase space. The initial state fixed (green). The task is for the network to be at final state (red) at time = 10. C 2-D slice of the 4-D parameter space containing three selected solutions (black markers). Heatmap denotes task loss evaluated along the slice. Overlaid is bifurcation diagram along the slice; lines indicate dynamical bifurcations and text indicates regions with stable origin, non-trivial fixed point attractors and limit cycles (LC). D, E, Phase portraits of the three selected solutions marked in C, trajectories taken by each the networks during the task < T, thick purple) and subsequently T. thin purple). Heatmaps denote speed of dynamics q 2 || 2 30
In the Interval discrimination task (22). two pulses of equal amplitude presented at times and I + l2, where 4, l2 E [10, and I * l2. The network should produce an output pulse whos sign indicates whether (2 1 not Figure 2B. The lower panel once more shows the desired output.
Finally, in the Interval reproduction task (20) the network receives two input pulses - Ready and Set - separated by fin time steps. The task is to generate output pulse tout tir. time steps after the Set pulse. The training intervals were drawn from a uniform distribution 4.x E [20, 50). The lower panel shows the desired output, where having only parameter defining the trials (t.,,) allows to display the entire trial output on each line. Trials aligned to the Ready pulse (Figure 28, red). This results in the Set pulse (yellow) forming line with slope 1.0 and the Go pulse (green) line with slope 0.5. Figure 2C shows the output of trained network matching this pattern.
For each task we trained 400 Vanilla networks, with N (20, 30, 40, 40,50) hidden units. See the Supplementary material subsectior[Section 2.1.3 for more details regarding the training process, as well as results from GRU and LSTM networks trained on the interval reproduction task.

---

B
Delayed discrimination
Interval discrimination
Interval reproduction
f
Couc
D
E
F
delay
t2
Time in trial
f2
The network is presented with two pulses with amplitudes fi and f2 respectively. separated by a variable delay, and should respond with pulse with an amplitude equal to sign(fi f2). Bottom: The desired output for all training data, for each delay, fi, f2 the color of the 3D matrix is the expected response. B Interval discrimination task. Top: The network is presented with two pulses with unit amplitude, that arrive at times t and l2 4. In response, should produce a pulse with amplitude that is equal to sign(12 - -4). Bottom: Desired output for all 4 and l2 combinations. C Interval reproduction (Ready-Set-Go task. Top: The network is presented with two pulses (Ready and Set, provided through separate input channels) separated by tin stees The desired output is Go pulse, delayed by tout thn steps from the Set pulse. Bottom: Desired output for all trials, with Ready, Set and Go depicted red, yellow and green respectively.
Figure 2: The three tasks. A Delayed discrimination task. Top:
5.1 Multiple neural dynamics solve the same task
We demonstrate the diversity of neural dynamics in different solutions by showing couple of examples from each task. The left side Figure 3/B shows the activity of two networks solving the interval reproduction task projected onto the first principal components. It is possible to infer two distinct algorithms for solving the timing problem from these plots. Because the task requires counting time in distinct epochs from Ready Set and from Set Go to explain the algorithms. The network in panel B uses the same phase space trajectory for both epochs, effectively counting from the maximal delay (tin = 50) downwards upon the Ready pulse, and moving to an earlier point upon the Set pulse. This is in contrast to the network in panel A, in which the Set pulse leads to different area in phase space. Furthermore, the almost circular trajectory of the Ready-Set epoch hints at the oscillatory behavior that will be discussed below.
focus on these
The delayed discrimination task also admits multiple solutions. In this case, consider activity during the delay period. Figure 3CD show several delay trajectories, corresponding to different fi values. The network in panel D converges approximate fixed points for each such value, while the network in panel C converges to limit cycles. Note that the network was trained with variable delays BI). and hence the second stimulus arrives at random phases of these cycles. Nevertheless, both networks perform well on the training set.
Interval discrimination contains two timing-epochs from the onset of the trial to the first pulse, and between the two pulses. The network in panel E uses the same trajectory in phase space for both epochs, similar panel of the interval reproduction task. The second example exhibits both oscillatory behavior and distinct trajectories for the two phases.
Taken together, these examples show that networks trained with identical hyperparameters, and reaching similar performance nevertheless develop qualitatively different neural dynamics solve the same task.
5

---

5.2 Extrapolation
On the behavioral side, we challenge networks with stimuli that are outside of the trained distribution. Since each of our tasks is parametric, testing the networks on extrapolation while increasing each parameter is a natural choice. The middle column of Figure shows the results of this challenge for the various networks, which often shows traces of the neural diversity described above.
The almost circular Ready-Set neural trajectory of panel A apparently results in a limit cycle, as revealed by the extrapolation plot. Similarly, the fact that the Ready-Set trajectory of panel B leads directly to the Go pulse is reflected in the vertical Go line that precedes the Set pulse in the extrapolation plot. Other features, such as the fact that Set pulse delivered after the Go pulse results in second output, are only revealed through extrapolation and cannot be deduced from the PCA of neural activity during the training set.
In the delayed discrimination task, the oscillations shown in panel C were not reflected in the output on the training region. Extrapolating to larger amplitudes and delays, however, reveals output oscillations as depicted on the top face of the extrapolation cube.
In the interval discrimination task (panels E,F), the relation between neural trajectories and extrap- olation behavior is less clear. But in this case, as in all others, the striking differences between different solutions are manifested both in the training neural activity and in the response to behavioral challenges.
5.3 Reduced Dynamics
Part of the motivation for looking into extrapolation patterns uncovering the algorithm that networks use to solve the task. Within the framework of neural dynamics (18 this corresponds to mapping
the dynamical objects used by the network and their relation to behavioral inputs and outputs. Previous works focused on fixed point topology to achieve this goal (30) 24 14. The tasks considered in the present work are mostly dependent on transient dynamics and hence require a different approach. To this end, we developed tool to compress the state space while preserving the essential dynamics and the behavioral information. Briefly, the method generates trajectories by combining long stretches of autonomous dynamics with behavioral inputs. The trajectories are then merged and compressed into a graph. Each node represents areas in phase space that have a given output and share the same input-dependent past and future. The edges represent autonomous dynamics (black) or the various inputs (colors). The full details of the algorithm are in the supplementary material subsection 2.2. The left column Figuue 3] shows such reduced dynamics for all the example networks. Examining these graphs can reveal the dynamics driving extrapolation patterns.
The network of panel A has a limit cycle, which is depicted by the red node that has a black self-loop. Furthermore, the graph shows that after the Go pulse, the network can still respond to an additional Set pulse. The fact that the network of panel B utilizes the same trajectory for both epochs is reflected in the yellow self-loop in the corresponding graph. The graph also shows that a Go pulse can occur before the Set pulse.
The delayed discrimination task has more input types and therefore was less compressed by the algorithm. As result, the graphs can show both the logic of comparison during the training set and the extrapolation patterns with the limit cycles. A Similar scenario holds for the interval discrimination task.
5.4 A space of solutions
The examples described above suggest that similar to the two-neuron case, trained networks converge into distinct solutions which can be characterized both by neural signatures and by behavioral ones. In the case of the two-neuron network, the parameter space is only -ou--dimensional, and most of the solutions are in a two-dimensional subspace. Hence, we could span most of it and understand the solution types via the linearized dynamics around the origin. In the more complex tasks, the reduced dynamics graphs serve as method to characterize the space of solutions. We computed the reduced dynamics for all 400 networks of each task and found that they only contain few different graphs. Figure 4 shows the number of networks of each type for the interval reproduction task. The four major graphs and their corresponding representative extrapolation plots are shown in panels A-D.

---

5.5 Inferring reduced dynamics from neural activity
The reduced dynamics described above are obtained by simulating the networks for very long times, to obtain their asymptotic behavior. In neuroscience setting, such information is not readily available. Is it possible to infer this asymptotic behavior from neural activity during the training set? To answer this question, extracted set of features, as detailed in the supplementary material. The features mostly measure the relationship between neural activity in the different epochs of the task. Other features measure geometrical properties of single epoch, as in the curvature of the Ready-Set trajectory, which was inspired by networks like Figure 3/. While we tried to include features that are natural descriptors of the neural activity, there some arbitrariness in our choice. We chose to on the side of including more features, and later rely on cross-vale coos-vvaliationn to avoid overfitting in our predictions (32).
For each task, we trained and evaluated a classifier to predict the reduced dynamics of the pool of networks from the neural features. Figure 4F shows the resulting confusion matrix for the interval reproduction task (other tasks are in the supplementary material Figures 8 [and Cohen's kappa (33) was used to summarize the prediction quality, showing an above-chance performance.
maierialFiguues
5.6 Different architecture and tasks
The diversity of solutions reported here arise from identical hyperparameters and random initialization. To probe the biases introduced by changing hyperparameters, we tested the effect of varying network architecture for the interval reproduction task. The supplementary material shows results from training this task using LSTM and GRU networks. We find that some solution classes are shared between architectures, and some only appear in certain architectures (Figure 5A).. Furthermore, these choices bias the solutions, as reflected in the different histograms. For instance, the example of Figure 5B did not occur in any of the 400 Vanilla networks, but was the most common type in GRU networks. Despite these biases, there are solutions that appear under all architectures, as exemplified in Figure SC.D,E.
The chosen tasks so far emphasized variability. We also examined a context-dependent integration task, which was previously shown to exhibit universal solution (24). The supplementary material shows an analysis of the response of networks to various behavioral challenges. We find that there is considerable variability in the response of networks to challenges, but of quantitative rather than a qualitative nature. Furthermore, we also quantified various aspects of neural variability in these networks. We did not find significant correlations between the various behavioral and neural measures, suggesting that there are many axes of individuality for these networks.
6 Discussion
In recent years, trained RNNs were successfully used to model biological circuits. Specifically, these networks converged to solutions that were similar to those of their biological counterparts. This observation is puzzling for any complex system and particularly regarding the brain, complex biological system which variability the rule rather than the exception. Inspired by this puzzle, we study the space of solutions of RNNs for both a simple two-neuron network and three neuroscience- inspired tasks. Through an interplay between observation, analysis, and perturbation in the form of extrapolation, we discovered for each task qualitatively different solutions. By analyzing the neural activity of these networks, we observed that the variety behavioral phenotypes originate from only a handful of distinct dynamical mechanisms. To affirm these observations and describe the essence of each solution, we introduced tool that summarizes the network computation in a reduced- reduced-dynamics graph that contains the essence of the dynamics they relate to behavioral inputs and outputs. By extracting neural features from the training activity, we showed that the solution type can be partially predicted from experimentally accessible measurements.
We show using a 2D continuous- time RNN that qualitatively different dynamical topologies can arise in the context of a single task and that learning can find different solutions depending on initialization. In a related study, the authors analyze a low-D continuous-time GRU and find large diversity of dynamical topologies (34). They find that the 2D GRU finds topologically different solutions to variants of a working memory task depending on the input statisties. Such modification to the task, although subtle, modifies the loss landscape and learning dynamics. In our work emphasize
7

---

that such topologiaal diversity of solutions can arise for the same task, that is to say within the same loss landscape. Both our work and theirs identify the implications of topological diversity for generalization.
Asking seemingly trivial question "what is solution?", highlights a complex and vital topic. Even though artificial and biological networks are being compared regularly, and there is a common intuitive understanding of what properties are relevant for this comparison, the precise definitions of "solution" or "mechanism" are rarely discussed (23). Is dynamical similarity within the training regime sufficient for asserting two solutions are alike? What the proper way to test subjects' performance on learned task? Can we answer these questions while considering the specifics of each task? Under what conditions can we safely assume that there exists a universal solution? We believe that every study that models neural circuits should be explicit about such meta-concern, In this work, we charted solutions with an operational approach in mind and considered solutions as different as long as they qualitatively differ either in their neural activity within the training set or in their behavioral prediction on extrapolation trials. The reduced dynamies tool is our attempt to describe and differentiate solutions according to the criteria that we consider important.
Our work has several limitations that should be noted. We focused on the variability arising with identical hyperparameters and therefore did not systematically explore the effect of changing them. Specifically, how network size, learning algorithm, and choice of architecture bias the solutions remains to be explored.
The space of solutions for given task is, above all else, property of the task itself. We showed examples of three tasks that have substantial variability, and of one task (context dependent integration) that has much less variability. Yet, we do not know why some tasks admit multiple solutions and others do not. We conjecture that the tasks presented in (24) require an informative output at all points in time, whereas the tasks we considered require an informative output only once at the end of the trial.
The seduced-dynanee
tool relies on set of trajectories regions in phase space that are reachable with task cons-consittent inputs (similar to (30). While exploring choice may limit the description obtained by tool. Finally,
starting point. We opted include
the full phase space is not feasible, the space of solutions was described for networks that have already learned to perform a task. remains be seen whether it can be used a map in which understand how the process of learning itself takes place.
To conclude, we found that RNNs can produce diverse set of solutions to the same computational tasks. These solutions represent distinct algorithms and are supported by corresponding dynamical objects. The solutions are isolated in parameter space, causing the initial conditions to largely determine the outcome. Furthermore, we showed that experimentally accessible tools can be used to indirectly characterize the asymptotic properties of the solutions. We believe that exploring the space of solutions can advance neuroscience, machine learning, and their intersection - making more rigorous comparisons of models to data.
Acknowledgments and Disclosure of Funding
This work supported in part by the Israeli Science Foundation (grant number 346/16, OB) and by a Rappaport institute thematic grant (OB).
References
[1] A. D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen,
J. Deaton, J. Eisenstein, M. D. Hoffman, F. Hormozdiari, N. Houlsby, S. Hou, G. Jerfel, A. Karthikesalingam, M. Lucic, Y. Ma, McLean, D. Mincu, A. Mitani, A. Montanari, Z. Nado,
V. Natarajan, C. Nielson, F. Osborne, R. Raman, K. Ramasamy, R. Sayres, J. Schrouff,
M. Seneviratne, S. Sequeira, H. Suresh, V. Veitch, M. Vladymyrov, X. Wang, K. Webster, Yadlowsky, Yun, X. Zhai, and D. Sculley. Underspecificatioon Presents Challenges for Credibility in Modern Machine Learning. [Online]. Available: http://arri..rrggaa//011...3..
[2] C. K. McIntyre, I.. K. Marriott, and P. F. Gold, "Patterns of brain acetylcholine release predict individual differences in preferred learning strategies in rats," vol. 79, no. 2. pp. 177-183.
8

---

[Online]. Available: http:///ww...aa.i....oo.oooooooa
[3] G. Gholamrezaei and I. Q. Whishaw, "Individual differences in skilled reaching for food related to increased number of gestures: Evidence for goal and habit learning of skilled reaching." vol. 123, no. 4. pp. 863-874. [Online]. Available:
[4] E. Patitucei, A. J. D. Nelson, D. M. Dwyer, and R. C. Honey, "The origins of individual general-process differences in how learning is expressed in rats: A general process perspective." vol. 42, no. 4,
pp- 313-324. [Online]. Available: http:///w....oooooooooooootmtmtd
[5] G. Kempermann, "Environmental enrichment, new neurons and the neurobiology of individuality," vol. 20, no. pp. 235-245. [Online]. Available: http://www.naturr.commaartt s41583-019-0120-
[6] S. Musall, Urai, D. Sussillo, and A. Churchland, "Harnessing behavioral diversity to understand neural computations for cognition," " vol. 58, 229-2362 238.. [Online]. Available:
13300028
[7] T. Kurikawa, T. Haga, T. Handa, R. Harukuni, and T. Fukai, "Neuronal stability in medial
frontal cortex sets individual variability in decision-making." vol. 21, no. 12, pp. 1764-1773. [Online]. Available: http::/www..ntt
www.nna.ttmmmmanca http://www.nauur..oom/aaa
[8] Morita, H. Kang, Wolfe, S. Jadhav, and D. E. Feldman, "Psychometric Curve and Behavioral Strategies for Whisker-Basse Based Texture Discrimination in Rats," vol. 6, no. 6, e20437. [Online]. Available:
I/j.ri3alll p0ne.00204377
[9] L. Castro and E. A. Wasserman, "Humans deploy diverse strategies in learning same- different discrimination tasks," vol. 93, pp. 125-139. [Online]. Available: hhttps
/linkinghub.elseiierccoom httpa/vwww..oooeeeeeeee///////ss 1202602
[10] L. Lebovich, Darshan, Y. Lavi, D. Hansel, and Loewenstein, "Idiosyneratic choice bias naturally emerges from intrinsic stochasticity in neuronal dynamics," vol. 3, no. 11, pp. 1190-1202. [Online]. Available:
[11] A. Gilad, Y. Gallero-Salas, D. Groos, and F. Helmchen, "Behavioral Strategy Determines Frontal or Posterior Location of Short- Term Memory Neocortex." vol. 99, no. pp. 814-8284-0 [Online]. Available:
[12] O. Barak, "Recurrent neural networks versatile tools of neuroscience reauroscienneerssacchh... vol. 46, pp.
1-6. [Online]. Available:
[13] O. Barak, D. Sussillo, R. Romo, M. Tsodyks, and L. Abbott, "From fixed points to chaos: Three models of delayed discrimination." vol. 103, pp. 214-222 222. [Online]. Available: 30000129
[14] V. Mante, D. Sussillo, K. V. Shenoy, and W. T. Newsome, "Context-dependee" computation by recurrent dynamics in prefrontal cortex," " vol. 503, no. 7474, pp. 78 84. [Online]. Available:
12742
[15] D. Sussillo, M. M. Churchland, M. Kaufman, and K. V. Shenoy, "A neural network that finds naturalistic solution for the production of muscle activity." vol. 18, no. 7, pp. 1025-1033. [Online]. Available: http://wwwnnauure hwtp://wwauee.oom//rrcc/e//emnae 4042
[16] G. R. Yang. M. R. Joglekar, H. F. Song, W. T. Newsome, and X.-J Wang, "Task representations in neural networks trained to perform many cognitive tasks." " vol. 22, no. 2, pp. 297-306.
[Online]. Available: http::/wwwwnattrr.ccomm http://wwntuur..oooaaaa hat.aree..oo//atee/////11330111.
[17] M. Schrimpf, Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher,
J. Tenenbaum, and E. Fedorenko, "The neural architecture of language: Integrative reverse-engineerring converges on a model for predictive processing." [Online]. Available:

---

[18] S. Vyas, M. D. Golub, D. Sussillo, and K. V. Shenoy. "Computation Through Neural Population Dynamics." vol. 43, no. 1, pp. 249-275. [Online]. Available:
http://wwwaammalrreeew.....//// http://rrrwdo/o11///////eeeeeeee 1 (9-44155
[19] E. D. Remington, D. Narain, E. A. Hosseini, and M. Jazayeri, "Flexible Sensorimotor
Computations through Rapid Reconfiguration of Cortical Dynamics," " vol. 98, no. 5, pp. 1005-
1019.e5. [Online]. Available: https://lnnkinghub... elweeeesssooeeieeee.ee pttee//ccoo//eereee/////ooooooo0
[20] M. Jazayeri and M. N. Shadlen, "Temporal context calibrates interval timing," vol. 13, no. 8, pp.
1026. [Online]. Available: http:://www.atthee http://wwwwtwtt.c.mmm www.:naau...o//artlleeee..o wwtp///acoc/om/m..222..
[21] R. Romo, C. D. Brody, A. Hernández, and L. Lemus, "Neuronal correlates of parametric working memory in the prefrontal cortex," " vol. 399, no. 6735, pp. 470-473. [Online]. Available:
http://wwww.aauurr.co////tcc////
[22] Z. Bi and C. Zhou, "Understanding the computation of time using neural network models," vol.
117. no. 19, pp. 1003--10540 [Online]. Available: http://www.ppasssro/oooooo/oo/// pnas. 1921699110
[23] O. Guest and A. E. Martin, "On logical inference over brains, behaviour, and artificial neural networks." [Online]. Available: https://ost..o//bmegg
[24] N. Maheswaranathan, A. Williams, M. Golub, S. Ganguli, and D. Sussillo, "Universality and individuality in neural dynamics across large populations of recurrent networks," in Advances in Neurul Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, http://cceeeeeeeiiiiircccccp//pp
Inc.. pp. 15 629-15 641. [Online]. Available: 515547706777655388111111... Papp
[25] J. Mehrer, C. J. Spoerer, N. Kriegeskorte, and T. C. Kietzmann, "Individual differences among deep neural network models," vol. 11, no. 1, p. 5725. [Online]. Available:
19632-w
[26] E. Ghazizadeh and S. Ching, "Slow manifolds within network dynamics encode working memory efficiently and robustly," vol. 17, no. 9, e1009366. [Online]. Available:
wwtp://plllo......11/oooooooooo 10099093366
[27] E. Marder and A. L. Taylor. "Multiple models to capture the variability in biological neurons and networks," vol. 14, no. 2, pp. 133-138. [Online]. Available: http
www..nature.co//arrillcsssm
[28] B. B. Machta, R. Chachra, M. Transtrum, and J. P. Sethna, "Parameter Space Compression Underlies Emergent Theories and Predictive Models," vol. 342, no. 6158, pp. 604-607. [Online]. Available:
htc.:////.....1/2//cccceeeeeee.e 1238723
[29] F., Andrea, "Aristotle Causality." [Online]. Available: Spp2010/entticss/aiitttlle---s causallltllya
[30] D. Sussillo and O. Barak, "Opening the Black Box: Low-Dimensional Dynamics in High-Dimensione Recurrent Neural Networks," vol. 25, no. 3. pp. 626 649. [Online].
Available: http://wwwwwmippessosomo/o////// 1wtp://011112///...... CaCCCCC0___009000 00409
[31] A. E. Orhan and W. J. Ma, "A diverse range of factors affect the nature of neural representations underlying short-term memory," vol. 22, no. pp. 275-283, [Online]. Available:
[32] R. M. Khan, C.-H. Luk, A. Flinker, A. Aggarwal, H. Lapid, R. Haddad, and N. Sobel, "Predicting Odor Pleasantness from Odorant Structure: Pleasantness as Reflection of the Physical World," vol. 27, no. 37, pp. 10015-10023. [Online]. Available:
[33] M. I.. McHugh "Interrater reliability: The kappa statistic," vol. 22, no. 3. pp. 276-282.
[34] I. D. Jordan, P. A. Sokół, and I. M. Park, "Gated Recurrent Units Viewed Through the Lens of Continuous Time Dynamical Systems." " vol. 15, p. 678158. [Online]. Available:
10

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Discussion
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work basic science and does not develop any algorithm that might be used in such manner.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] github in supplement
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] In supplement
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] The effect of the random seed is part of the paper
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

---

A
Time trial
B
Time in trial
to
C
delay
D
delay
E
F
PURZUTTE SEEGRRAdS
Figure 3: Diverse solutions for the same tasks. Two example networks shown for each task (rows). Left column: two-dimensional PCA of the network activity. Middle column: Network output in response extrapolation across task parameters. Right column: the reduced dynamics of the networks. (A,B) Solutions to the time production task. The PCA plots show the activity from the Ready pulse (red dot and black line), as well from the Set pulse (yellow) up to the beginning of the output production, for three different task parameters (colorbar). Note that in A the Ready-Set epoch separated from the Set-Go, whereas in these epochs converge. (C,D) Solutions the delayed discrimination task. The PCA plots show the activity between the two input pulses, for various fi amplitudes. The memory of the different amplitudes is kept either via limit cycles (C) or slow points (D). (E,F) Solutions to the interval discrimination task. The PCA plots show activity from trial onset (black dot and line), well as from the first pulse (yellow) until the maximal delay, for three different 4 values (colorbar). Note the convergence of the two epochs to the same trajectory in E, similar to B.
12

---

A
8
B
Reduced-Dynameen type
5
true
D
S
-10
Time
predicted
Figure 4: The space of solutions for the interval reproduction task. A-D Representative extrapolation plots (left) and reduced dynamics graphs (right) for the four most common solution types. E Distribution solution types for the 400 networks trained. The four solutions shown account for 93% of the networks. J' Neural features obtained during the training can partially predict the solution type that includes extrapolation dynamics. The confusion matrix shows the result of this prediction. Note that the first solution types are mixed this prediction, but their dynamics during the training intervals is similar and they only differ in the dynamics after the Go pulse.
13

---

A
500
Vands CRU LETI
250
COurt
C
100
00
D
E
Figure 5: Architecture biases, but does determine the solution. A A histogram of the six most common reduced dynamics across all three architectures for the time-reproduction task, shown by stacking the urchitecture-specife histograms on top of one another. B, C The reduced dynamics (left) and the extrapolation patterns (right) of the second and third most common solutions across all architectures, but rarely occur within the Vanilla networks. D, E, F The 2D PCA of the dynamies of three networks from all three architectures (Vanilla, GRU, LSTM), that reach the solution shown in panel
14


Reviews:
[{"review_id": "l0zSp9JapjR", "reviewer": "Reviewer_PRAa", "summary": "This paper aims to characterize the space of solutions for a recurrent neural network and show that the specific solution reached after training is determined by weight initialization. It first presents a toy problem, a 2-by-2 RNN characterized by the eigenvalues of the connectivity matrix of the initialization. Then, it presents a task that uses a larger network and is inspired by a timing task in neuroscience experiments, for which the characterization is a set of six network topologies, predicted for new networks by a decision tree. The paper shows examples of qualitatively distinct sets of solutions emerging in each case, then discusses the limitations of this approach, a key one being that in the case of the larger RNN, the characterization is task-dependent. \n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality:* This paper positions itself against existing work extensively and I generally agree with its analysis. My only concern is with reference 8 - line 53 seems to be saying that ref 8 deals with representations and doesn’t separate mechanisms, though it is about dynamical mechanisms and finds four that it calls out. The related work comes across as somewhat defensive because it is entirely devoted to distinguishing from similar works, with no space devoted to having lots of references set up the field and this paper. This is mitigated by the presence of similar text in the intro, though. Regarding novelty, my interpretation is that the primary contribution of this paper is a parametrization for the space of solutions to an RNN. In the 2-by-2 case, where this characterization seems to boil down to eigenvalues, I am on the fence about whether it’s a creative enough application of a pretty basic mathematical tool to count as novel. The expansion to larger RNNs for Ready-Set-Go seems unequivocally novel.\n\n*Quality:* I think more experimentation or theoretical analysis is needed to warrant this paper’s claims, though it is on the right track. In the 2-by-2 case, the idea of applying the eigenvalues of the network’s connectivity is interesting, but the idea that the solutions follow from it seem to be argued by inspection of a couple examples presented in Fig. 1. To me, this seems insufficient for a toy problem in a paper that suggests theory. I think this would benefit from either a large set of examples or proof of equivalence/set membership. \n\nWe then move to the timing task. I’m unable to gather from Section 4.4 and 4.5 whether the experiments are limited to the six topologies given in Fig. 3 or not. Either way, there is room for explanation of why these are canonical and not just six examples that emerged in testing, and/or some explanation of why they constitute a sufficient characterization of the space. The authors mention task structure-dependence as a limitation of this paper, and that’s a good future work - if that were the only limitation, then I would consider this a good contribution in pursuit of that end goal due to the task-agnosticity of the method of getting a characterization. However, the six-topology characterization feels insufficient even given task-dependencee. Overall, I think there is lots of potential here. Showing solution dependence on initial weights is compelling, and this paper presents examples. However, I think it makes bigger claims than it warrants. \n\n*Clarity:* This paper starts out clear. The research question is presented well and sounds intriguing. Starting with section 3, the language becomes more opaque - the paper seems to be asking for some intervention from the reader by making conclusions from visual inspection of figures or presenting precise ideas in prose. In section 4, the organization becomes difficult to follow. Section 4.2, “what constitutes a solution”, could be one or two sentences at the beginning of section 4.3. Section 4.4 relies on section 4.5 to actually get to “a space of solutions”, but the headers don’t suggest this. I would suggest anchoring the paper more on one main contribution, or a primary and secondary contribution, then collapsing subsections and making sure they are written with the compass of those contributions in mind.\n\n*Significance:* If this paper warranted its original claims, I would consider it a significant result with implications for interpretability and prediction, and neuroscience insofar as it opened new directions to explore. However, given that the warrants in this paper are limited and the idea proposed is compelling but simple, the significance is diminished in my opinion. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of limitations is honest and extensive, and I think is a great example of communicating the scope and importance of one’s work without overstating it. Not every limitation is easily surmountable, but the section generally sets up interesting future work. I don’t see major limitations that are unaddressed. The paper is indeed theoretical and core to the point that discussion of societal impact seems unnecessary. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "JQxWltM0YG", "reviewer": "Reviewer_VAVV", "summary": "The authors demonstrate situations where RNNs converge to fundamentally different algorithmic solutions to under-constrained computational tasks. The major focus of the work is on the \"Ready-Set-Go\" task which has been used before in the neuroscience literature. The fact that fundamentally different solutions naturally arise in RNNs adds complexity to the suggested link between artificial and biological neural networks.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I really like the fundamental ideas and results presented in this paper, I found many specific details hard to follow.\n\nIn Figure 2, panels D-G are difficult to interpret. Please add x-axis and y-axis labels and a legend in the figure to explain the red / blue / yellow / green coloring in the phase space plots. The phase space plots might be easier to understand if arrows were added showing the evolution of time. Please increase the font size on the x and y axes all the panels in this figure (and elsewhere throughout the paper). Also, the text mentions networks A, B, C, D but these are not labeled in panels D, E, F, G (though I can surmise they are matched in this respective order).\n\nDespite the comments above, I think I understand the main results of figure 3 and I think they are interesting! However, given only the PC dynamics, it is only clear how the network in panel G (network D?) solves the task. How are the other three networks solving the task? More explanation is needed. I think adding fixed points to these phase plots using the methods of Sussillo & Barak (2013) is really necessary to understand what is going on.\n\nFigure 3 is a bit more confusing, because it looks to be a schematic of possible algorithms that networks could implement, rather than a demonstration that RNNs actually do cover this entire space of possible algorithms. Could this figure be modified to include examples of each topology (i.e. include PC trajectories from an example network next to each diagram)?\n\nOverall, there is a glimmer of something interesting in this paper, but I will give a borderline score since I am having a hard time understanding the fine details. I am certainly open to raising my score if additional clarity can be provided in the author response.\n\nSome additional comments\n-------------------------------\n\n- I think the title should be modified to make it clear that the paper is about RNNs solving a particular \"Ready-Set-Go\" task. Right now the title is too broad / vague, compared to the content of the paper.\n\n- Fig 3, panel E is hard for me to understand. If the nonlinearities are saturating, then at some point the network must either reach a fixed point, enter a limit cycle, or exhibit chaotic dynamics -- the activity can't just go out to infinity? I suppose this blowing up of dynamics could happen if ReLU was used as the nonlinearity?\n\n- \"We trained a decision tree to predict the topological class from the neural features\" -- How was this done? Was the decision tree trained on hand-labeled data? The supplement provides some more details, but I think they should be included in the main text.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are adequately described and I foresee no negative societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4w-rJ2djcH-", "reviewer": "Reviewer_M2vD", "summary": "This manuscript investigates the solution space of  RNNs optimized to perform certain tasks. The manuscript starts from a simple two-neuron network and shows that multiple solutions exist in the trained RNNs. It then moves on to a ready-set-go task, inspired by neurophysiology experiments. \nUnderstanding the solution space of trained RNNs is an interesting and potentially important problem. However, the scope of the paper in it current form is quite limited. Although some of the elements are interesting, overall the results are not fully developed, and do not provide substantial insights compared to previous work. Having said that, I think the paper has very good potential, and upon further investigation, it could be extended into a useful contribution. \n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This work has some novelty, although the underspecification of deep network models is well known among practitioners in machine learning and the neuroscience community. There are some clever ideas in this paper, such as using the network’s ability to extrapolate as a way to probe the RNN solution.\n\nQuality: \nSome elements of this work are thoughtful. However, the results are preliminary, and a bit far from being fully developed. The result on how the initialization may bias the optimization solution is not really surprising, and furthermore it is conceivable that the amount of dependence may also depend on the methods used to train the network. \nThe paper only studies a single neuroscience task. It is unclear how these results generalize to other tasks. They may well not, as the RNN solution space is likely highly task dependent (as the authors acknowledged as well). Therefore, it is uncertain whether any general insight is learned through this exercise.  \nOverall, I feel the quality of the manuscript at this stage does not quite cross the bar for a good NeurIPS paper. The analysis described lays a good foundation, but substantial amount of additional work is needed.\n\nClarity: The writing is generally clear. However, the part related to the topologies of the network dynamics is confusing. The analysis there is not well motivated, and the concept is not clearly defined.\n\nSignificance: The results and scope of the work as described are limited, and may not generalize to other tasks. These factors substantially reduced the significance of the work, despite that the study is attacking an important problem. I would highly encourage the authors to consider testing their analysis framework on a variety of commonly used neurophysiological tasks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the ideas behind this work but agreed that it needs another conference cycle's worth of work before it is ready for publication. Though the authors have some promising new results, the changes are too substantial to be vetted for this year's conference. 
Decision: Accept (Poster)