Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the effect of learning rules on the algorithms learned by recurrent networks on context-dependent integration tasks. A number of continuous time vanilla RNNs are trained on integration tasks with different learning rules: backprop, genetic algorithms, hebbian learning, and full-force. The paper finds that across these learning rules, all networks learn consistent attractor topologies in order to solve the task. As the complexity of the task is increased (increasing number of contexts), the representations start to differ, but the attractor structure is preserved. The paper finds that using line attractors to solve integration seems to be a universal property of these networks, independent of learning rule. Finally, the paper shows how properties of the learned attractors (both the topology and whether the attractors are continuous manifolds or discrete points) vary with task input noise.
main_review: Overall, I enjoyed reading this paper. It is well written and addresses an important problem: what aspects of the algorithms learned by networks trained to solve particular tasks are sensitive to the choice of learning rule used to train the network?

### Questions / concerns
- The tasks are presented as being a part of three separate types (RDM, CDI, and N-CDI), but I feel like it makes more sense to present the tasks as all being part of the same family of tasks. They are all context-dependent integration tasks, but what varies is the number of contexts: ranging from one (RDM) to two (CDI) to multiple (N-CDI). Is my understanding correct?
- L184/185: "This is possibly due to the fact that Hebbian RNNs do not compute a linear readout ... but instead use an arbitrary recurrent neuron as the output". This seems like a significant difference, and should be controlled for. Why not train the other networks (BPTT / GA) in this fashion, by picking an arbitrary neuron to be the readout (as opposed to having a linear readout). That would help control for this difference.
- It would be nice to see results for other tasks (besides integration). Are there other simple tasks that might be good candidates?
- I thought the results showing a transition from transient to stable integration with increasing task noise particularly interesting. Are there other aspects of the task that might induce similar transitions? This point is particularly relevant for neuroscientists that might observe continuous or discrete integration in the brain.
- In Figure 2a, it looks like the BPTT representations are all similar, all the GA networks are different (not only from BPTT, but different amongst themselves as well). Any intuition for this? It looks like some GA networks are similar to the BPTT networks, especially for smaller numbers of contexts. Is that variability due just to the random seed? Can you identify what it is about certain random seeds that makes some GA networks look like BPTT networks?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: A popular paradigm for modeling the brain using task-optimized deep learning models identifies three key features governing the solutions learned by neural networks: architecture, task, and learning rule. Building on work by Maheswaranathan et al. [1], which showed that architecture influences the representations learned by trained recurrent neural networks (RNNs), but not their attractor structure, the authors of this work extend the analysis to study the effect of learning rule and task on the representations and attractor structure of trained RNNs — with broadly consistent results. The authors show that the attractor structure is largely invariant to the choice of learning rule (at least on two simple tasks: random dot motion and context-dependent integration, with large input noise), but the learned representations are not invariant, especially as the complexity of the task (the number of individual contexts to integrate) increases. However, the authors also exhibit a setting where the attractor structure is *not* invariant to the choice of learning rule: when the input noise is low in either task. They further show that the learned attractor structure depends sensitively on the amount of noise injected into the network, with a transition between one solution in the low-noise regime and another in the high-noise regime. While these results may seem contradictory, they may be consistent under the hypothesis that the attractor structure of trained RNNs exhibits universality across architectures and learning rules *when the task is sufficiently difficult*.
main_review: If this is indeed a point the authors wish to make, it would have been nice to see the models trained on a greater diversity of tasks, with more interesting attractor topologies. [1] studied three tasks with fairly unique attractor topologies, which were further investigated and linearized. Of these the present work studies only the context dependent integration task, whose attractor topology has been studied exhaustively in previous works ([2], although the authors here identify a new solution as a function of input noise), and additionally studies a random dot motion task. But the only analysis of attractor structure reveals either one or two fixed points (for zero input) in each case. It is perhaps not so surprising that different learning rules converge on these same simple solutions (although it would be interesting to investigate how the attractor topologies differ in the case of low input noise! The authors could consider including this analysis). When the task is made more difficult by including N contexts, only two learning rules lead to successful training (backprop and a genetic algorithm). The two learning rules lead to consistent attractor structure as evaluated by the MDS analysis, but the attractor structure itself is not investigated.

This work includes several interesting conceptual insights beyond previous works, including the sensitive dependence of learned solutions on input noise. And the demonstration of settings where trained networks can exhibit universality to learning rules is an important step towards better understanding and interpreting neural networks, and using them as models of the brain. Overall the analyses are well performed and the presentation is clear. That said, the extensions this work presents beyond [1] are relatively modest, and no new methods or analysis techniques are introduced which could be of independent interest (aside from the silhouette score, the methods in the present work appear to be identical to those introduced in [1]).

Finally, a “cautionary tale” discussed in [1] is that CCA may actually be a poor metric of representational similarity, given that in some settings the canonical correlation between trained and untrained RNNs can be greater than the canonical correlation between two trained RNNs with different design choices (e.g. nonlinearities). It would be interesting for the authors here to discuss this concern, and/or to try out a different metric for representational similarity.



[1] N. Maheswaranathan, A. H. Williams, M. D. Golub, S. Ganguli, and D. Sussillo, “Universality and individuality in neural dynamics across large populations of recurrent networks,” 2019.

[2] H. F. Song, G. R. Yang, and X. J. Wang, “Training excitatory-inhibitory recurrent neural networks for cognitive tasks: a simple and flexible framework,” PLoS Comput. Biol., vol. 12, no. 2, pp. 1–30, 2016.
limitations_and_societal_impact: The results are fairly presented and the authors are clear throughout the paper about the limitations of their work as well as potential future directions and impact on neuroscience.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper aims to characterize the impact of design choices in RNN-based experiments on learning dynamics and representation geometries, both of which are important data sources when using RNNs to model biological intelligence. It begins by setting up the problem of investigating RNNs’ fitness to model cognitive and neural systems, and drawing a distinction from the most similar work (Maheshwaranathan et al). It then describes a standard continuous-time RNN architecture, three tasks, one of which is a complexity-modular version of another, and four learning rules of which I believe one is local. Along with the model setup, it details evaluation and visualization methods. The bulk of the paper appropriately focuses on results. This goes through the following conclusions: dynamics do not vary with learning rule (implicitly, for more complex tasks), representations vary with task complexity, dynamics do not vary with task complexity, dynamics do vary with learning rule under low input noise with simpler tasks, and finally dynamics are confirmed to not vary with learning rule under low input noise with more complex tasks (suggested explanation being that more complex tasks have a more constrained solution space). The paper ends with a summary of these conclusions and their implications for the use of RNNs in computational neuroscience.
main_review: Great paper! Impressively clear, well-scoped, doesn’t overpromise, delivers something interesting. 

*Originality:* The work seems sufficiently original. It is a contribution to the community and is very clear and honest in comparing itself even to the most similar work, distinguishing itself specifically by focusing on learning rules. It includes a good range of related work and positions itself as a helpful addition.

*Quality:* This work seems technically sound. It scopes itself well and lists out certain claims, then goes on to provide experimental backing for each claim. The experiments are well-designed and clear, and answer the questions asked. Particular compliments to Fig 1, which illustrates the primary claim of the paper almost independently. Though some of this and other figures ask for visual inspection, they are backed up by descriptive statistics in the text. I have no qualms about experimentation quality. My only concern would be that I would have liked to see more tasks even of these complexities, so that there were more examples for various experiments (especially the ones using only N-CDI). Without these, the paper does seem to be putting all its eggs in one basket for certain claims, but the thoroughness of the experiments that are present mitigates this a lot. 

The methods are intuitive and thorough. One concern is that some of the selected learning rules become incompatible with certain experiments, so it could be that a larger set of learning rules would have been valuable so that certain experiments weren’t limited to one or two rules, but I could see this being left for future work (and I realize that options are limited). 

I don’t know that the paper discusses limitations sufficiently to satisfy the current NeurIPS guidelines (see below), but the paper also doesn’t overstate its contributions. I think it’s relatively intellectually honest. I do see it as a work in progress in that it leaves a lot of space for expansion, but as a paper in computational neuroscience I think it’s sufficient. 

*Clarity:* This paper is really well-written. Many papers start off strong and proceed to fall apart writing-wise in the results, but this one has great organization, signposting, and writing throughout. It also follows the rule of “say what you’re going to say, say it, say what you said”, which was very useful for my understanding. My only concern is that I interpret the findings to be the following in order: “dynamics do not vary with learning rule (implicitly, for more complex tasks), representations vary with task complexity, dynamics do not vary with task complexity, dynamics do vary with learning rule under low input noise with simpler tasks, dynamics are confirmed to not vary with learning rule under low input noise with more complex tasks (suggested explanation being that more complex tasks have a more constrained solution space)” (quote from my summary). It’s a little confusing to understand how the beginning and end of this flow fit together. These aren’t exactly organized by either design choice (LR, input noise, etc.) or output (dynamics, representations) and might benefit from such an organization. I may also not fully understand the flow, so I am open to the existing ordering. 

Minor: the paper starts off referring to task complexity as “task hyperparameters” exclusively; use of “task complexity” starts around halfway through the paper. Terms should be consistent regardless, but this is especially confusing because the paper is about learning rules. So, I recommend only using “task complexity” so no one has the opportunity to get confused even if they are skimming, even if the N in N-CDI could be considered a hyperparameter. 

*Significance:* I think this paper is a useful contribution to understanding of how RNNs should be used when modeling biological intelligence. To my knowledge, the results are unique and set up both lots of ground for expansion and framework for experimentation meant to model the brain. To me, this seems like a worthwhile addition to the field. 

limitations_and_societal_impact: The checklist claims that this work discusses its limitations, but the referenced limitation is “mechanisms in the RNNs should be interpreted as a hypothesis for biological mechanisms”. I wouldn’t consider this a discussion of limitations of this work. I’m not even sure how it’s a limitation at all, but if the idea is that RNNs aren’t a perfect model for biological intelligence and are primarily theorized as one, then that’s a critique of the whole field (in fact, one that this paper is trying to rectify). The one limitation that this work does discuss in the main text is that experiments should extend to more complex tasks, but this is given half a sentence and is posed as a future work. I totally agree and think the paper is complete enough to leave that for future work, but that’s not a sufficient discussion of limitations as per the NeurIPS checklist guidelines. I think the paper should discuss the limited space of tasks (even in the range of complexity of RDM and (N-)CDI) and learning rules, or biological implausibility of some of the learning rules. It’s certainly possible that I misinterpreted the note in the checklist, but I don’t think this paper really discusses the limitations. I also don’t think it would be hurt by a discussion - it’s a good paper that does enough to survive an honest discussion of limitations. 

I agree that this paper is theoretical enough to not need a discussion of societal impacts. Since the work “involves understanding deep neural networks”, I’m not sure that it “inherits the broader positive and negative societal impacts of deep learning”. It does very indirectly (which is what I think the authors are saying), but (also indirectly, but less so) it inherits more of the impacts of neural network interpretability than neural network application. One thing that isn’t touched is the societal impact of having good models of biological intelligence, which are huge and rarely discussed. I don’t think this warrants a flag for ethical review, since 1) there is so much ground between the state of the field and a societally impactful model and 2) I rarely see such discussion in comp neuro papers, but I do believe it warrants consideration if it is the ultimate goal.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors extend the recent work of Maheswaranathan et al to explore how learning rule and task complexity affect the solutions obtained by recurrent neural networks trained on neuroscience-inspired tasks.
Specifically, four different learning rules are used to train networks on a suite of similar tasks with systematically increasing complexity. The authors show that learning rule has some effect on representation geometry, and this effect grows with task complexity. Attractor topology, however, remains universal for almost all tasks and learning rules.
The authors also show some instances in which network dynamics are influenced by input noise. This sensitivity, however, decreased as task complexity increased.

main_review: The question of individuality and universality of solutions obtained by trained recurrent neural networks is an important and timely one. These networks are used to compare with experimental data, and it is important to understand how the nature of solution is affected by various design choices.
The specific extensions studied here – learning rule and task complexity – are widely used to optimize networks, so it is critical to understand how each criterion biases the obtained solutions, especially when claiming similarity between RNNs and brains. Of particular note is the use of genetic algorithms, that are not often used in this context, and could hence provide novel viewpoints on the space of solutions. Furthermore, the systematic approach to task complexity is a promising approach to understand its effect.
Despite these strengths, there are several major issues with the current submission: The novel insights beyond the work of Maheswaranathan et al are not large; The effects uncovered by the authors are not explored; The dynamical system analysis is non-standard and not justified appropriately.
1.	As the authors state, this work is very much in line with that of Maheswaranathan et al.  As such, the added value should be much more clearly stated. In particular, the reasons for discrepancy should be studied. The authors speculate that “increasing task complexity… constraining possible solutions”. And yet the empirical finding is that solutions of complex tasks are more individualistic. This seems like a promising avenue for exploration – but it is not pursued.
2.	Hebbian learning is distinct from all other rules. This is interesting. Why is it so? The topic is not pursued.
3.	The analysis used to arrive at the conclusion is composed of many processing steps. At the end of all these steps, we are left with a single number (Silhouette score) that describes variability. Compressing a complex phenomenon like the solution found by a network to a single number is desirable in order to produce figures like 2C and 2E. But it also precludes an understanding of what the underlying reasons for these observations are. It would be very helpful to accompany such analyses with several intermediates like comparing PCA (or other dimensionality reduction methods) of various networks. 
4.	The authors define fixed points as having a speed of 1e-8 and below, and slow points of a speed of 1 and below. This is a very high threshold for the slow points (compare to Sussillo & Barak 2013). Furthermore, the analysis used inputs that are close to zero to define some ordering between fixed points. If I understand correctly, the authors actually find slow points that have a speed considerably lower than 1. By slightly perturbing the input from zero, these slow points cross the threshold of being considered fixed points. First – this is my guess. Second – this could be an interesting heuristic for reverse engineering trained networks. Third – none of this is explained or justified in the appendix. Standard analysis requires using zero input. Other choices might be useful, but should be justified and clearly stated.

Specific points:
5.	The authors analyze one task with varying complexity and find that the dynamics are almost invariant. It could be that other tasks have more variability, and hence could be better suited to answer the main question posed here – does learning rule affect solution?
6.	Silhouette score is used extensively. It could be useful to provide some intuition on the meaning of the various values.
7.	Figure 1 caption “networks are highly structured in attractor structure”. This sentence is somewhat confusing when combined with a low silhouette score
8.	Figure 2B 5-CDI panel shows two clusters for the GA. Why is that? Such observations are not mentioned or analyzed in the paper.
9.	Figure 3A the attractor changes shape from a line to a sigmoid. Why is that?
10.	Figure 3a: It is unclear where the slow areas appear in relationship to the attractors, which differences arise when the Silhouette score is high (e.g. figure 3B for low variance) and what is the contribution of the learning rule to this variability. This is especially needed because the Hebbian learning rule appears to dominate the variability. 
11.	L271: “memory of the pulsed input”. Should this be memory of the sign of the input?
12.	Figure 4. Perhaps looking at the actual speed values of points along the attractor can be informative here.
13.	Figure 5 caption is hard to understand
14.	L302: “two distinct dynamical mechanisms used for integrating the inputs”. I’m not sure whether it was demonstrated how these mechanisms are used for the task.
15.	Figure S6 in appendix. The panel D, which represents stable integration looks quite similar to the  transient panel A. This is not commented upon.
16.	In general, interpretation and explanation of the numerical results would greatly strengthen the paper.

limitations_and_societal_impact: Societal impact – not relevant.
Limitations: mentioned above.
review_title: null

Meta-review: This paper is an empirical study of the impact of learning rule and input noise on the RNN solutions in a small family of tasks. The reviewers agree that writing is clear, and showed excitement in seeing the results, but concerned with the narrow range of tasks. Reviewers made excellent constructive suggestions  that I strongly encourage the authors to incorporate in the final, stronger manuscript.
Decision: Accept (Poster)