Venue: NeurIPS 2019 Workshop Neuro_AI

Reviews:

final_score: rating: null
review_content: importance_comment: The question of how networks maintain memory over long timescales is a longstanding and important one, and to my knowledge this question hasn't been thoroughly explored in spiking, trained recurrent neural networks (RNN). The importance is tempered by the findings only covering what is to be expected, and not pushing beyond this or describing a path to push beyond this.
rigor_comment: The work would benefit from more detailed discussion of the training algorithm that provides some indication that the results aren't unduly sensitive to these details. In particular, the setting of synaptic decay constants is an important detail in a paper about working memory. A short discussion of other training algorithms (such as surrogate gradient or surrogate loss methods) and why the given one was chosen instead would have been helpful. A comparison with Bellec et al. 2018, which looks at working memory tasks in spiking networks, would also have been appropriate.

The statistical tools are fairly well described and appear to be well-suited for illustrating the phenomena of interest. I feel that more tools should have been used to further support or push the results. For instance, while the heatmaps in Figure 3 provide visual evidence for their claims (except see my comments below), the work could have benefitted from a quantification of this evidence. For instance, it is hard to see differences between the cue periods in the bottom two heatmaps, but differences may appear in some numerical measure of the average discriminability over these regions.
clarity_comment: The technical details are presented clearly on the whole. However, I feel that the work lacked clarity when it came to interpretation of the results. For instance, the claim of "stronger cue-specific differences across the cue stimulus window" between fast and slow intrinsic timescale neurons in the RNN model isn't clearly supported by the heatmap in Figure 3 -- the cue-specific differences for the short instrinsic timescale group to me appears to be at least as great as that of the long intrinsic timescale group within the cue stimulus window. I would be curious to know if making the input weaker or only giving it to a random subset of neurons makes this phenomenon more apparent. 

It seems that one of the main points of the work is that "longer intrinsic timescales correspond to more stable coding", but I didn't find that this point was made very convincingly. The work would have benefited from a discussion of the implications of longer intrinsic timescale neurons retaining task-relevant information for longer -- in particular, this finding feels a bit "trivial" without the case being made for why this should push understanding in the field. I think the interesting part may be in quantifying just how much of a difference there is between short and long timescale neurons -- for instance, does task-relevant information in both neuron groups fall off in a way that can be well predicted by their intrinsic time constants? How does this relate to their synaptic time constants? Does limiting the synaptic time constants limit the intrinsic time constants, and if so by how much?

The same type of comments apply to the second part of the results, which demonstrates that a task that doesn't require working memory results in neurons with shorter intrinsic timescales compared to the working memory task.
intersection_comment: The authors use an artificial network model to shed light on the biological mechanisms enabling and shaping working memory in the brain. The paper in the process reveals some (expected) results about how spiking RNNs behave on a working memory task. The proof-of-concept work (among others) that this can be done with spiking RNN may inspire more work in this area.
comment: The work is a basic proof-of-concept of results that may not do much to advance understanding since they are what one would expect to see (i.e. the antithesis of their thesis seems very unlikely). Looking into the nuances of the explored phenomena may provide new information for the field. The paper should also seek to connect with more of the recent work being done in spiking recurrent neural networks.
review_title: Clear and straightforward modeling exploration, but it is less clear how the work pushes understanding

final_score: rating: null
review_content: importance_comment: In this study, the author(s) trained spiking RNNs with BPTT in delayed/instant tasks, then compared the autocorrelation of the trained units with data from macaque. Though the presented work have some technical and biological issues, I think this work is potentially interesting. 
rigor_comment: The author(s) used the word “intrinsic timescale” and “autocorrelation” interchangeably, but they are different. In particular, even if the intrinsic timescales of the neurons are the same, depending on the connectivity structure, neurons develop different autocorrelation (see eg. R Chaudhuri et al., Neuron, 2015). Moreover, because the author used autocorrelation sigma, instead of the intrinsic synaptic decay constant tau_s, for the analysis, it is impossible to tell if the functional segregation shown in Fig. 3 is originated from optimization of w or tau_s. Thus, I believe further clarification is needed.
clarity_comment: “experimental model” is a bit misleading; “experimental data” is probably better.
intersection_comment: In this study, the author(s) trained RNN with a ML method (BPTT), then compared that with the experimental data. That, I think, is a nice intersection.
comment: As mentioned on the rigor section, the author(s) should disentangle the effect of w and tau_s, for instance, by comparing the performance of RNN with or without optimization of tau_s, or analysing the effect of tau_s more directly.
Another potential issue is the biological meaning of the optimization of tau_s. For a given pair of neurotransmitter and receptor, the variability of the synaptic time constant is unlikely to be large. I think it is biologically more plausible to optimize AMPA/NMDA ratio, while fixing tau_AMPA and tau_NMDA at their typical values.
review_title: Potentially interesting work on neuronal timescales, but a disentanglement is needed

final_score: rating: null
review_content: importance_comment: Interesting and important initial characterization and comparison of neural response dynamics  between an RNN and neural data. Was not clear whether similarities were surprising or inevitable: e.g. would an untrained net have diversity of autocorrelation decay. No novel predictions or insights into neural data were provided or at least made clear.
rigor_comment: Results were convincing and carefully performed. The task DMS task seems simple enough (multiplication). It would have been nice to motivate training by gradient descent vs what could presumably be designed by hand. Perhaps it is not so simple? Quantification of variability across networks for different training regimes and architecture would have been nice and provided some indication of how general the results are.
clarity_comment: Very clear exposition, excellent figures. In areas was brief on motivation and model set up but two pages isn’t a lot of space and they cite relevant literature. RNN Model vs experimental model (line 73)? Experimental model is the neural data? Calling it a model is confusing if it is neural data.

intersection_comment: Used RNN model which is also used in AI but authors never explicitly make connection to AI applications or otherwise. It would have been nice if they gave some indication or speculation of what in the networks changes to achieve different decays in autocorrelation which could then be matched to the natural time scale of AI tasks if prior knowledge was available.
comment: The clarity and simplicity of the measurements made in the study are very nice. It would have been nice if the study had gone a step past characterization and comparison to novel prediction and insight. Whats the function of short time scale units? What in the weights of networks creates different time scales and diversity of time scales? Over all the work is preliminary but of high quality and interest. Good work!
review_title: Interesting characterization and comparison of RNN and neural time scales

Decision: Accept (Poster)