Charting and Navigating the Space of Solutions for Recurrent Neural Networks

Abstract: In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating under the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and machine learning challenge this assumption. Animals can approach a given task with a variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that every task is associated with a space of solutions. To date, the structure of this space is not understood, limiting the approach of comparing RNNs with neural data.Here, we characterize the space of solutions associated with various tasks. We first study a simple two-neuron network on a task that leads to multiple solutions. We trace the nature of the final solution back to the network’s initial connectivity and identify discrete dynamical regimes that underlie this diversity. We then examine three neuroscience-inspired tasks: Delayed discrimination, Interval discrimination, and Time reproduction. For each task, we find a rich set of solutions. One layer of variability can be found directly in the neural activity of the networks. An additional layer is uncovered by testing the trained networks' ability to extrapolate, as a perturbation to a system often reveals hidden structure. Furthermore, we relate extrapolation patterns to specific dynamical objects and effective algorithms found by the networks. We introduce a tool to derive the reduced dynamics of networks by generating a compact directed graph describing the essence of the dynamics with regards to behavioral inputs and outputs. Using this representation, we can partition the solutions to each task into a handful of types and show that neural features can partially predict them.Taken together, our results shed light on the concept of the space of solutions and its uses both in Machine learning and in Neuroscience.


Reviews and author responses:

 Thanks for the detailed response, and I think you've done a pretty good job of considering the feedback. I appreciate your attention to my comments on related work and clarity. 

I appreciate the motivation for the 2-by-2 case, but my main concern was with originality, not quality. Like the other reviewers have said, the other changes are significant - the full space of topologies is a good, but in my opinion major, change. It's hard to imagine the paper with such a large change - even if the resulting topologies are similar to the ones in the current version, the method is very different (and good!). I would definitely encourage submission in an upcoming conference, but at this time I can't justify changing the score of what's been submitted. 

 Thank you for your response and the update on new analysis. I think some of the ideas presented in the paper are interesting, and the new analysis the authors responded also sounds promising. However, it is difficult for me to imagine what a revised paper would look like given the original paper submission was too premature. I agree with reviewer VAVV that another conference cycle  would likely yield better results. 

 Dear Reviewers,

We are aware that the discussion period is near its end, but would like to bring to your attention the results of an additional task we analyzed.

The main tasks analyzed thus far involved timing, and we wanted to verify that our observations also hold for tasks in which timing plays no role, and the natural solution involves a line attractor.

To this end, **we trained networks on a delayed discrimination task** (Romo et al. Nature, 1999). The task consists of two input pulses of varying amplitude separated by a delay, and the desired output is an indicator of whether the first stimulus is larger than the second one. We used **variable delays** to discourage networks from reaching an oscillatory solution in which the desired memory is only available at a specific time interval (Orhan and Ma, Nature Neuroscience, 2019). We found that networks converge to **at least two qualitatively different solutions**. These differences were evident in the network dynamics during the original training stimuli, without extrapolation. One solution involved a **line attractor**, where each slow point corresponds to a different value of the first stimulus. Another solution that emerged was a **line of limit cycles**, where each limit cycle corresponds to a different stimulus value, and the oscillation is roughly orthogonal to the direction of the line. Similar to the examples presented in our paper, **we could classify these solutions according to dynamical signatures in the training set**. Furthermore, **we could design test-stimuli that reveal the nature of the solution**. Specifically, injecting noise during the delay had an oscillatory effect on classification performance in one type of solution.

We believe this example strengthens our main message - canonical neuroscience tasks admit multiple, qualitatively different solutions. These solutions can be classified by examining network dynamics, and can also be revealed behaviorally by designing perturbative input trials.

We hope these new results strengthen the generality of the paper, and in any case thank you for your time and effort in reviewing it.

 Thank you for your replies. I think that this paper shows great promise, but it doesn't seem ready for publication in its current form. (I wish that we could review re-submissions with the edits you suggest.) I think another conference cycle of work on the project will bring it to fruition.

 We thank the reviewer for the time and comments.
First, the main changes since the original submission:
* The 2D example was extended to show conditions without qualitatively different solutions, and analysis showing sources for variable solutions.
* Two more tasks were studied: An interval comparison task and a sentiment analysis task. Our tools and main findings transferred to these tasks, and we discovered more variability in the solutions of sentiment analysis than was previously reported.
* The manual topology classification method was replaced by an automatic algorithm operating on well-defined principles.
Prediction of topologies from dynamical features was done in a more principled and task-agnostic manner.

Below we relate to the specific points raised by the reviewer:
1. **Reference 8 (Mehrer et al) and related work in general.** We realize this came out defensive, and also much relevant literature was omitted. This will be amended in the final version.
2. **2D example.** The 2D example has several motivations. First, it shows that qualitatively different solutions can arise in a very simple setting. We now also added an analysis of this degeneracy for both feed-forward and recurrent networks. Namely, the task is
$\begin{bmatrix}0 \\\\ 1\end{bmatrix} = A \begin{bmatrix}1 \\\\ 0\end{bmatrix}$, where the matrix $A\in \mathbb{R}^{2\times2}$ is either the parameter matrix (feed-forward), or $A=W^n$ for a recurrent network that propagates n steps in time. The former case has two unconstrained parameters ($A_{12}$, $A_{22}$), and therefore a 2D space of solutions. The latter case has two n^th order equations and therefore can generate an additional level of redundancy in solutions. This is the underlying reason for the limit cycle solutions, where the different frequencies correspond to different roots of these equations.
A second motivation is to highlight the contribution of timing tasks to this variability. To this end, we now add a 2D task that requires the output to remain constant for a prolonged time. This results in parametric variability (many connectivity matrices), but no qualitative variability (only fixed points, no limit cycles).
We also show the discrete nature of solutions arises because under certain conditions of $A$ the solutions of $A=W^2$ are isolated like the roots of a polynomial.
3. **Full space of topologies.** The six classes of topologies in the current version were semi-manually defined by classifying the 1200 trained networks. We now use a novel principled algorithm to define topologies  (see response #1 to reviewer VAVV) on a new set of 1200 networks. The resulting topologies are very similar to those reported in the current version. This is not to prove that the six topologies are an exhaustive list. It is possible, in this specific task, to define a tree that would characterize all possible topologies. This is because there is a small and finite set of stimuli given to the network. Specifically, following the ready pulse, the network can reach a limit cycle, fixed point, or infinity (for LSTM). For each of these options, a set pulse can result in the same three options. A Go pulse can occur in each such epoch, leading to more splits in the tree. Ignoring a few details, we can describe the full space of possible topologies using roughly 30 possible topologies, as the number of leaves in the tree. In practice, we find that not all topologies are obtained via training.
4. **Clarity.** We apologize for the lack of clarity and thank the reviewer for the suggestions on improving it.
5. **The claims of the paper.** We hope that our modifications help back the claims made in the paper. Two new tasks were studied. The definitions of topologies and neural features are not hand-tuned anymore and transfer to a novel task. We show that existing tasks (sentiment analysis) contain more variability than was previously recognized. We extended the 2D analysis to more clearly chart the different solutions and how they arise. 


 We thank the reviewer for the time and comments.
First, the main changes since the original submission:
* The 2D example was extended to show conditions without qualitatively different solutions, and analysis showing sources for variable solutions.
* Two more tasks were studied: An interval comparison task and a sentiment analysis task. Our tools and main findings transferred to these tasks, and we discovered more variability in the solutions of sentiment analysis than was previously reported.
* The manual topology classification method was replaced by an automatic algorithm operating on well-defined principles.
Prediction of topologies from dynamical features was done in a more principled and task-agnostic manner.


Below we relate to the specific points raised by the reviewer:
1. **Generality.** We now train two additional tasks - a sentiment analysis task and an interval discrimination task. In the sentiment analysis task, similar to that of Maheswaranathan and Sussillo  2020 (arXiv), we show that probing it with custom stimuli can reveal hysteresis, indicating that the line attractor has some thickness, and different histories can lead to different locations in this other dimension. Crucially, the thickness profile varies between networks. This shows that custom stimuli (such as extrapolation) can reveal underlying variabilities, even in cases that were thought to be universal. We believe that timing tasks lend themselves to greater qualitative differences, and hence also trained an interval discrimination task. We used our new algorithm for defining topologies on this task (see response #1 to reviewer VAVV). We also used the task-agnostic neural features (see response #6 to reviewer VAVV) and were able to predict topological classes without fine-tuning either the dynamic observables or the topological algorithm.
2. **The topology analysis.** We discuss the motivation and method in response #1 to reviewer VAVV.


 We thank the reviewer for the time and comments.
First, the main changes since the original submission:
* The 2D example was extended to show conditions without qualitatively different solutions, and analysis showing sources for variable solutions.
* Two more tasks were studied: An interval comparison task and a sentiment analysis task. Our tools and main findings transferred to these tasks, and we discovered more variability in the solutions of sentiment analysis than was previously reported.
* The manual topology classification method was replaced by an automatic algorithm operating on well-defined principles.
Prediction of topologies from dynamical features was done in a more principled and task-agnostic manner.

Below we relate to the specific points raised by the reviewer:
1. **Fixed points to describe dynamics.** This is an important point, as it highlights the challenge in analyzing tasks that have a strong temporal component. Because the task revolves around counting time, it is anchored on transients of the dynamics rather than on fixed/slow points. To cope with this issue, we designed a principled algorithm that operates in two stages. First, converging trajectories from different trials are identified, and a graph of the phase space of the system is constructed, in which nodes are states and edges define transitions due to time or stimulus. At the second stage, the graph is systematically reduced by collapsing parallel paths along the graph. In this manner, only qualitative differences remain. The result is very similar to the manual classification appearing in the current version but does not require any hand-tuning of features or classes.
2. **Clarity of figures and detailed explanations.** We apologize for the lack of clarity and will amend it. Both the addition of arrows in Figure 2 and the accompanying PCA plots for Figure 3 are great suggestions, and we will implement them. 
3. **Figure 3 algorithms.** We will add detailed explanations for all topologies (perhaps some will have to be in supplementary). The topologies are more informative of how the networks behave during extrapolation than the algorithm for the task itself, though they are certainly linked. For instance, Fig 3B and Figure 3D both similarly solve the original task but differ with respect to extrapolation. Figure 3B will reach a fixed point in the absence of a Set pulse, whereas Figure 3D will enter a limit cycle - leading to a periodic extrapolation profile.
4. **Title of paper.** We now train two additional tasks - a sentiment analysis task, and an interval discrimination task. We, therefore, feel that the broader title is (now) appropriate.
5. **Infinite activity.** This is a feature of LSTM, in which activity is unbounded. This is also what allows LSTM networks to be perfect counters. We were unsure whether to include this, as it is a very “unbiological” feature, but decided to keep it to show another class of possible solutions.
6. **Decision tree.** The aim here was to show that experimentally accessible measurements, taken during the normal training regime, can predict the topological class that is defined via dynamics outside this regime (in the extrapolation regime). We now use more task-agnostic measures that relate to geometrical features of the activity in the two task epochs and relations between them. As further proof of the generality of these measures, we also used them on a new task (interval discrimination) and were able to predict topological classes without fine-tuning either the dynamic observables or the topological algorithm. We will explain this more clearly and completely. 

The authors demonstrate situations where RNNs converge to fundamentally different algorithmic solutions to under-constrained computational tasks. The major focus of the work is on the "Ready-Set-Go" task which has been used before in the neuroscience literature. The fact that fundamentally different solutions naturally arise in RNNs adds complexity to the suggested link between artificial and biological neural networks.  While I really like the fundamental ideas and results presented in this paper, I found many specific details hard to follow.

In Figure 2, panels D-G are difficult to interpret. Please add x-axis and y-axis labels and a legend in the figure to explain the red / blue / yellow / green coloring in the phase space plots. The phase space plots might be easier to understand if arrows were added showing the evolution of time. Please increase the font size on the x and y axes all the panels in this figure (and elsewhere throughout the paper). Also, the text mentions networks A, B, C, D but these are not labeled in panels D, E, F, G (though I can surmise they are matched in this respective order).

Despite the comments above, I think I understand the main results of figure 3 and I think they are interesting! However, given only the PC dynamics, it is only clear how the network in panel G (network D?) solves the task. How are the other three networks solving the task? More explanation is needed. I think adding fixed points to these phase plots using the methods of Sussillo & Barak (2013) is really necessary to understand what is going on.

Figure 3 is a bit more confusing, because it looks to be a schematic of possible algorithms that networks could implement, rather than a demonstration that RNNs actually do cover this entire space of possible algorithms. Could this figure be modified to include examples of each topology (i.e. include PC trajectories from an example network next to each diagram)?

Overall, there is a glimmer of something interesting in this paper, but I will give a borderline score since I am having a hard time understanding the fine details. I am certainly open to raising my score if additional clarity can be provided in the author response.

Some additional comments
-------------------------------

- I think the title should be modified to make it clear that the paper is about RNNs solving a particular "Ready-Set-Go" task. Right now the title is too broad / vague, compared to the content of the paper.

- Fig 3, panel E is hard for me to understand. If the nonlinearities are saturating, then at some point the network must either reach a fixed point, enter a limit cycle, or exhibit chaotic dynamics -- the activity can't just go out to infinity? I suppose this blowing up of dynamics could happen if ReLU was used as the nonlinearity?

- "We trained a decision tree to predict the topological class from the neural features" -- How was this done? Was the decision tree trained on hand-labeled data? The supplement provides some more details, but I think they should be included in the main text. Limitations are adequately described and I foresee no negative societal impacts.

This manuscript investigates the solution space of  RNNs optimized to perform certain tasks. The manuscript starts from a simple two-neuron network and shows that multiple solutions exist in the trained RNNs. It then moves on to a ready-set-go task, inspired by neurophysiology experiments. 
Understanding the solution space of trained RNNs is an interesting and potentially important problem. However, the scope of the paper in it current form is quite limited. Although some of the elements are interesting, overall the results are not fully developed, and do not provide substantial insights compared to previous work. Having said that, I think the paper has very good potential, and upon further investigation, it could be extended into a useful contribution. 
  Originality: This work has some novelty, although the underspecification of deep network models is well known among practitioners in machine learning and the neuroscience community. There are some clever ideas in this paper, such as using the network’s ability to extrapolate as a way to probe the RNN solution.

Quality: 
Some elements of this work are thoughtful. However, the results are preliminary, and a bit far from being fully developed. The result on how the initialization may bias the optimization solution is not really surprising, and furthermore it is conceivable that the amount of dependence may also depend on the methods used to train the network. 
The paper only studies a single neuroscience task. It is unclear how these results generalize to other tasks. They may well not, as the RNN solution space is likely highly task dependent (as the authors acknowledged as well). Therefore, it is uncertain whether any general insight is learned through this exercise.  
Overall, I feel the quality of the manuscript at this stage does not quite cross the bar for a good NeurIPS paper. The analysis described lays a good foundation, but substantial amount of additional work is needed.

Clarity: The writing is generally clear. However, the part related to the topologies of the network dynamics is confusing. The analysis there is not well motivated, and the concept is not clearly defined.

Significance: The results and scope of the work as described are limited, and may not generalize to other tasks. These factors substantially reduced the significance of the work, despite that the study is attacking an important problem. I would highly encourage the authors to consider testing their analysis framework on a variety of commonly used neurophysiological tasks.
 yes.

This paper aims to characterize the space of solutions for a recurrent neural network and show that the specific solution reached after training is determined by weight initialization. It first presents a toy problem, a 2-by-2 RNN characterized by the eigenvalues of the connectivity matrix of the initialization. Then, it presents a task that uses a larger network and is inspired by a timing task in neuroscience experiments, for which the characterization is a set of six network topologies, predicted for new networks by a decision tree. The paper shows examples of qualitatively distinct sets of solutions emerging in each case, then discusses the limitations of this approach, a key one being that in the case of the larger RNN, the characterization is task-dependent. 
  *Originality:* This paper positions itself against existing work extensively and I generally agree with its analysis. My only concern is with reference 8 - line 53 seems to be saying that ref 8 deals with representations and doesn’t separate mechanisms, though it is about dynamical mechanisms and finds four that it calls out. The related work comes across as somewhat defensive because it is entirely devoted to distinguishing from similar works, with no space devoted to having lots of references set up the field and this paper. This is mitigated by the presence of similar text in the intro, though. Regarding novelty, my interpretation is that the primary contribution of this paper is a parametrization for the space of solutions to an RNN. In the 2-by-2 case, where this characterization seems to boil down to eigenvalues, I am on the fence about whether it’s a creative enough application of a pretty basic mathematical tool to count as novel. The expansion to larger RNNs for Ready-Set-Go seems unequivocally novel.

*Quality:* I think more experimentation or theoretical analysis is needed to warrant this paper’s claims, though it is on the right track. In the 2-by-2 case, the idea of applying the eigenvalues of the network’s connectivity is interesting, but the idea that the solutions follow from it seem to be argued by inspection of a couple examples presented in Fig. 1. To me, this seems insufficient for a toy problem in a paper that suggests theory. I think this would benefit from either a large set of examples or proof of equivalence/set membership. 

We then move to the timing task. I’m unable to gather from Section 4.4 and 4.5 whether the experiments are limited to the six topologies given in Fig. 3 or not. Either way, there is room for explanation of why these are canonical and not just six examples that emerged in testing, and/or some explanation of why they constitute a sufficient characterization of the space. The authors mention task structure-dependence as a limitation of this paper, and that’s a good future work - if that were the only limitation, then I would consider this a good contribution in pursuit of that end goal due to the task-agnosticity of the method of getting a characterization. However, the six-topology characterization feels insufficient even given task-dependencee. Overall, I think there is lots of potential here. Showing solution dependence on initial weights is compelling, and this paper presents examples. However, I think it makes bigger claims than it warrants. 

*Clarity:* This paper starts out clear. The research question is presented well and sounds intriguing. Starting with section 3, the language becomes more opaque - the paper seems to be asking for some intervention from the reader by making conclusions from visual inspection of figures or presenting precise ideas in prose. In section 4, the organization becomes difficult to follow. Section 4.2, “what constitutes a solution”, could be one or two sentences at the beginning of section 4.3. Section 4.4 relies on section 4.5 to actually get to “a space of solutions”, but the headers don’t suggest this. I would suggest anchoring the paper more on one main contribution, or a primary and secondary contribution, then collapsing subsections and making sure they are written with the compass of those contributions in mind.

*Significance:* If this paper warranted its original claims, I would consider it a significant result with implications for interpretability and prediction, and neuroscience insofar as it opened new directions to explore. However, given that the warrants in this paper are limited and the idea proposed is compelling but simple, the significance is diminished in my opinion.  The discussion of limitations is honest and extensive, and I think is a great example of communicating the scope and importance of one’s work without overstating it. Not every limitation is easily surmountable, but the section generally sets up interesting future work. I don’t see major limitations that are unaddressed. The paper is indeed theoretical and core to the point that discussion of societal impact seems unnecessary. 


Meta-review: The reviewers appreciated the ideas behind this work but agreed that it needs another conference cycle's worth of work before it is ready for publication. Though the authors have some promising new results, the changes are too substantial to be vetted for this year's conference. 

Decision: accept