Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper analyzes the topology of the solutions of RNNs trained on time counting tasks, revealing a rich set of possible dynamical solutions and architectural biases that affect the generalization properties of the solution. It uses numerical experiments to illustrate the diversity of solutions that can arise for tasks where the solution relies on slow transients rather than fixed points for performing the task and highlights some of the difficulties in performing a systematic analysis in such contexts. This is potentially relevant for understanding across subject variability in animals performing such tasks.
main_review: Overall evaluation: An interesting idea but the lack of systematicity in the approach diminishes the value and interpretability of the results. More work needed before publication. 

Major concerns: While liking the general idea, there's not a sense that we have really understood the phenomenology by the end of the paper.  The analysis feels incomplete, and the described topological structure somewhat anecdotal. I would have thought that the 2-neuron network toy example would have provided more analytical backing for the arguments but is seems to also primarily rely on simulations to make the points. Are the topological features in fig3 hand designed by the scientist as a human interpretation of continual trajectories or could this topological structure be identified algorithmically? For the last part when identifying dynamic signatures of the solution that can be used to differentiate various algorithmic solutions (which are defined 'manually' by the scientist based on the same trajectories), I am unsure what have we actually learned there. Is there a way to define classes of solutions straight from data and to which degree are these methods applicable to neural data (i.e. are these quantities that one could reasonably estimate from limited neural recordings?). Also not sure what the take home message is for the behavioral predictions in fig 6. 

Significance: highlights the computational distinctions across 'working memory'-like tasks and furthers the arguments for a topological/algorithmic understanding of RNNs computations. Missed opportunities in relating to across animal variability in similar neuroscience experiments.

Originality: Since the number of degrees of freedom in trained RNNs is often much larger than the task constraints, there is an increasing interest in characterizing the nature of the solutions trained by RNNs and their generalization properties, with relevance for both neuroscience and machine learning. In particular, Susillo et al have shown that certain memory task show a certain form of universality, where the qualitative nature of the solution is more or less the same regardless choices of architecture and hyperparameters. This paper uses similar ideas to show that other tasks that count time are less constrained in the nature of their solution with several distinctive algorithms.

Clarity: The abstract and introduction are very well written, but the related literature lacks substance and the discussion is poorly structured and needs a fair bit of work.  The quality of the figures is poor, with systematically missing axis labels, inconsistent colors across panels, etc; it would also greatly help to have in figure legends for different colors. the captions help somewhat to make sense of the lot but makes the reader's life unnecessarily difficult.  
limitations_and_societal_impact: No obvious negative social impact. The limitations of the approach are touched upon in the discussion but not very satisfactorily.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the solutions  RNNs find and characterize them based on extrapolation and recurrent dynamics. They primarily consider two cases -- a simple two neuron network performing a memory task, and larger networks performing a Ready-Set-Go task. They show that the final solution depends on the initial connectivity, but not so much on the hyperparameters of learning. 

main_review: **Update:** Based on the authors response, and new results mentioned, I have updated my score to reflect my more positive evaluation of the paper.

Overall, this is very interesting and nicely done work but the paper has some flaws that resulted in my recommendation of rejection.

## Originality: 
There are other works characterizing RNN dynamics, e.g. (Maheswaranathan et al. 2019) which is cited in the related work. All the aspects where this work differs is not very clear (might be useful to add), but one original contribution of this paper seems to be the use of extrapolation for characterizing the solution, which is an interesting idea. Another contribution is to show that initial connectivity and not hyperparameters influence the final solution (although it remains to be seen how generally applicable this is).

## Quality: 
The experiments performed in the paper are well defined, and the conclusions reached are reasonable. But the take-home message of the paper is not clear. Concrete comparisons with biological data would have strengthened the paper quite a lot.

One specific comment: It's strange that the parameters are trained in the PC space for the 2-neuron network as described in the caption for Fig. 1. Training in the original weight space and projecting onto the PC space just for plotting would have been more meaningful.

## Clarity: 
The writing is reasonably clear, but there are some gaps in the explanations:
* How are the classes of networks defined? Is there a quantitative measure for this? Are the labels done manually? This key detail is not very clearly spelled out.
* Sec. 4.3: Details about the networks being tested are not given. How large are these networks, and of what type?
* Fig. 5 has no axis labels and explanation of what's plotted
* l.219: Not sure reference [18] makes sense in the context of that sentence.

Minor:
* Fig. 2A: $t_p$ should be $t_s$?
* Lines 128-135: Networks are referred using A,B,C,D while the figures are labelled D-G causing some confusion.
* l.152: Figure 2A should be Figure 2D.

## Significance: 
Understanding the space of solutions of RNNs is a very important problem, and this paper adds a couple of insights related to this.
Another key contribution of the paper is to show the variety of solutions that emerge for the considered tasks -- highlighting the selection bias present in papers that show activity similar to biology emerging in networks trained to do specific tasks.
limitations_and_societal_impact: The authors have a fairly good discussion of the limitations of the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper makes two main contributions. First, the authors study a couple of examples to show that the learned structure and dynamics of an RNN are not completely determined by the task on which it was trained -- legit solutions form a vast space, and where an RNN lands depends on where it starts. Second, on the same examples, the authors set out to dissect how the different solutions differ, both in the algorithmic strategies they employ, and in the realisation of these strategies in the network's internal dynamics. The most novel result is that aspects of the former can be predicted by features of the latter. The first point about degeneracy of solutions is not exactly news, but it is very nicely illustrated here on examples, and it is an important reminder to the field that one should be careful in interpreting the emerging dynamics of trained RNNs as brain-like solutions. The second is potentially of greater practical importance to the field, and I commend the authors for even trying to attack such a difficult problem of relating dynamical motifs at the neural level to "algorithmic strategies"! However, I have many issues with the clarity of the paper, and struggle to understand if the methodology developed in this paper will be of any use beyond the particular Ready-set-go task considered.
main_review: I have found the paper difficult to understand:

- the abstract is cryptic, with intriguing sentences such as "[...] by testing the network's ability to extrapolate, as a perturbation to a system often reveals [...]" → not sure a priori what the link might be between extrapolating and responding to a perturbation; similarly, "we relate extrapolation patterns to specific dynamical objects" → too vague to be useful; similarly, "Using a set of features, we define a space of solutions." → again, too vague to be a useful summary.

- In section 4.3, the reader has to work out for themselves that panels D, E, F and G of Figure 2 refer to networks A, B, C, and D in the main text.

- Figure 3: what are we looking at?? What is the meaning of these circles and squared, open vs filled, coloured vs white? The corresponding Section 4.4 is very opaque -- there are lengthy descriptions of the various possibilities, but no sense of how they were arrived at. Is there a general approach to categorising strategies? "shows the salient locations in phase space" - what does that mean? "enter a slow area" -- where do we see this? "some networks emit a go pulse" -- isn't it the case that all successfully trained networks will emit a go pulse? At this point I am rather lost.

- Figure 5: what's on the x and y axes?
 
------

Minor:

- Figure 2 : t_s corresponding to each shade of blue in Fig2 D-G should be labelled
- Figure 2H seems a bit arbitrary; why not compute the participation ratio or something?
- Section 4.4: "there are no guarantees" -- for what?
 
limitations_and_societal_impact: Societal impact is not addressed at all, but I think we can live with that given the paper's main topic.
The limitations are discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study proposes an approach for characterising the space of solutions obtained by fitting neural networks to a couple of tasks (one simple but general task and one neuroscience-related task). The analyses suggest that network initialisation and architecture are two key factors for the degeneracy of solutions obtained when fitting neural networks.
main_review: ### Originality

The novelty of the study resides in the development of a set of analyses to characterise the space of neural network solutions. Previous work is appropriately cited and it is clear how this work differs from previous contributions.

### Quality

The paper is technically sound, and claims are appropriately backed by empirical evaluation.


### Clarity

The paper is clearly written, and provides enough information for an expert reader to understand all the steps to reproduce the results. There is just a minor typo I would like to point out: in lines 128-134, the authors reference the networks as A to D, and I believe these references to be wrong.


### Significance

I believe the developed analyses would be of interest to the neuroscience community, in particular applied to real neuroscience datasets. However, the conclusions drawn from the analyses are somewhat unsurprising from the Machine Learning point of view, as it is well known that neural network fitting is a problem with multiple local minima, and that this depends on the initialisation and architecture. In fact, it is common and good practice in Machine Learning to train networks several times, with multiple seeds, initialisations and hyperparameters, in order to assess the robustness of the drawn conclusions. In order to improve the significance and impact of the manuscript, I would recommend the authors to apply their set of analyses to a Machine Learning task, with the hope of getting new insights e.g. in terms of optimisation (note that the manuscript is one page shorter than the limit, and thus would have the space for extra analyses).


### After rebuttal

Thank you for the careful answer to all reviews. After reading all the reviews and authors' rebuttal, I believe the authors made an effort to address all the reviews, and in my opinion, to some satisfaction. I increased my score to better reflect the quality and significance of the work.
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work.
review_title: null

Meta-review: The authors present in this work an analysis of the multiple solutions that can be identified by a recurrent neural network trained on a task. in particular, the authors discuss the implications for computational neuroscience studies that leverage such tools. The reviewers all agree that the work was well presented, and in particular strengthened by a second analysis performed in response to the reviewer comments that remove a potential confound of timing. While the scores themselves seem to be borderline, the topic is very timely and the results appear to be highly relevant to the use of ML in Neuroscience. I therefore recommend this work (with the addition of the extra analysis) be accepted to NeurIPS. 
Decision: Accept (Poster)