Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors show how the nervous system of C. elegans can be modelled and simulated with data-driven models using different neural network architectures. Specifically, they target the use of state of the art recurrent neural networks architectures such as LSTMs and GRUs and compare these architectures in terms of their properties and their RMSE, as well as the complexity of the resulting models. Authors show that GRU models with a hidden layer size of 4 units are able to accurately reproduce the system’s response to very different stimuli.
main_review: In this paper authors create models for the C. elegans nervous system with three different recurrent neural networks architectures: simple RNNs, LSTMs and GRUs. The objective is to further generate a low-order description to replace the original, detailed model in the NEURON simulator. 

Authors should improve the automation in choosing appropriate stimuli for the training, validation and test sets as well as optimal parameter selection and perform a systematic analysis of compression possibilities of the learning-based models with error control.
summary_of_the_review: Overall its a weak accept. Authors should improve the automation in choosing appropriate stimuli for the training, validation and test sets as well as optimal parameter selection and perform a systematic analysis of compression possibilities of the learning-based models with error control.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper shows that a small recurrent neural network can be used to predict the activity of 4 neurons in a C. elegans simulation with good accuracy as measured by RMSE.

main_review: 
The paper uses a connectome-based in-house model of the worm, utilizing compartmental neuron models and chemical and electrical synapses to generate the ground truth data. In general, I found that insufficient details are provided about how this model was built & validated (how were the biophysical properties chosen, what synapse models were used, etc). The authors state that the simulator is "high-fidelity" because they "assume it reproduces with fidelity the real output of C. elegans neurons.". The model might be high-complexity, but as far as I could see, fidelity was not demonstrated anywhere. In fact, the only test case, also used for all subsequent modeling work with neural networks, is "forward locomotion". In it, 2 sensory neurons and 2 interneurons are stimulated to generate activity in 4 neurons known to be active in locomotion. It's possible that additional information is available in the supplementary materials which I was not able to access ("ERRORS.The value of "offset" is out of range. It must be >= 0 && <= 17825792. Received 17825795"). Even if that's the case, more of this information should be in the main text.

I found section 3.1 with detailed discussion of GRU, LSTM, and problems related to training RNNs a bit too verbose. These are standard and well-known architectures at this point, and could just be briefly introduced with references to the original papers. The reader would benefit much more from learning the details of the C. elegans simulation.

The paper does a good job of systematically testing various variants of recurrent neural networks (LSTM, GRU, vanilla RNN) in a simple setup involving a single recurrent layer followed by a dense layer with 4 outputs. The authors focus on GRU, and conclude that a tiny neural network with just 4 GRU units is sufficient.

That a GRU network works for modeling time-series data is not particularly surprising. I found the experimental work presented in the paper far too limited to yield useful insights into using RNNs as a black-box model of real brain activity. Ideally, the experiments should cover more neurons and behaviors, and variations in synapse strengths (see doi:10.1038/s41586-021-03778-8) and the biophysical parameters. What is missing is also a comparison of the computational complexity of the simulation and the proposed reduced order model. One could also ask whether the model used in the simulation is actually necessary to produce this "forward motion" scenario -- could some neurons be excluded, or simpler neuron models used instead?

Specific comments:
* The text mentions that the separation of the simulation results into train/test/validation was done manually. Please provide more details on the criteria you used to ensure diversity within each group.
* In experiment 3, you state that "[..] data sampled with different time steps as this leads to longer sequences". This should probably be rephrased to highlight that the same physical time is simulated.
* What exactly is the input to the network? Only the 2 currents on the sensory neurons or also on the interneurons?
* Why is it necessary to provide stimulus on interneurons instead of just sensory neurons?



summary_of_the_review: The paper effectively shows that a tiny GRU network can reproduce a few time-series generated from a larger simulation. The general research direction of building reduced-order models for such simulations is interesting, but the very limited empirical results presented here and lack of details about the simulation make it impossible for me to recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the use of recurrent neural networks as a
model reduction tool in computational neuroscience. More specifically,
the authors consider the problem of predicting the activity of a set
of four neurons in the C Elegans nematode worm, resulting from the
(simulated) electrical stimulation of other neurons in the animal's
connectome. Three experiments are carried out, testing different
network architectures, network sizes, and the effect of increasing the
temporal resolution of the data. The results show that a small
GRU-based network is sufficient for achieving excellent agreement with
the starting data, which was generated using computationally demanding
simulations of a network of multi-compartimental neurons. The paper
also includes an introduction on popular RNN architectures, and on the
problem of model reduction in computational neuroscience.

main_review: ## Strenghts
- The paper is very well written and is an enjoyable read. It explains
  well the problem it is trying to address and why it's important, and
  it introduces all concepts and techniques it uses in an accessible
  way. I think this is particularly important for this paper, as its
  main audience would presumably be computational neuroscientists, so
  it seems a good idea not to take for granted a deep knowledge of
  (say) recurrent network architectures.
- Addressing the problem of model reduction using off-the-shelf
  machine learning approaches is a timely choice, given the very rapid
  increase of experimental data available for modeling in this field.
- The main thesis of the paper is well supported by the evidence,
  which is laid out in a logical and clear progression.

## Weaknesses
- Mirroring what I wrote above in one of the "strenghts" points: the
  fact that paper is a rather simple application of off-the-shelf RNN
  architectures to a well-defined modeling problem means that the
  paper is not hugely ambitious, and that it doesn't bring much in the
  way of theoretical or conceptual insight; however, in my opinion
  this does not detract from the quality of the work.

## Minor suggestions:
- When discussing related literature, I invite the authors to consider
  the inclusion of a broader selection of papers that in recent years
  applied deep network approaches to modeling neural activity data, as
  the works currently cited in the paper investigate only fMRI data
  (which is only distantly related to actual neural activity), or are
  specific to C Elegans. Two examples that come to mind are
  Molano-Mazon et al, ICLR 2018, *Synthesizing realistic neural
  population activity patterns using generative adversarial networks*,
  and Bellec et al, NeurIPS 2021, *Fitting summary statistics of neural
  data with a differentiable spiking network simulator*.
- If possible, it would be great to have an actual comparison of the
  compute time/cost incurred in running the trained RNN (even just for
  the final selected version of the architecture, the 4- or 8-units
  GRU), versus running the ground-truth simulation with NEURON. I
  apologise if this is given somewhere and I missed it.
- Please make all figure labels bigger. Axes labels, tick labels and
  legend contents are currently almost impossible to read!
- on page 7, last line of section 5.1: "*[...] we consider it the main
  option and keepING the LSTM as an alternative architecture*" → should
  be "keep".


summary_of_the_review: This is a solid paper that addresses a question which will be of
interest to part of the ICLR community. The research is somewhat
limited in scope, but very well executed and written up, and overall a
worthy contribution. It is also a good demonstration of the usefulness
of open databases of neuroscience models and data.

My recommendation is to **accept**.
review_title: null

Meta-review: This paper explores the use of recurrent neural networks to model neural activity time-series data. The hope is that computationally demanding biophysical models of neural circuits could be replaced by RNNs when the goal is simply to capture the right input-output functions. The authors show that they can fit RNNs to the behaviour of a complex, biophysical model of the C elegans nervous system, and they explore the space of hyperparameter and network choices that lead to the best fits.

The reviews for this paper were borderline, with scores of 3, 6, and 8. On the positive side, the reviewers agreed that the paper is very effective in demonstrating that the input-output behaviour of the biophysical model of C elegans can be replicated by RNNs. But, on the negative side there were concerns about the limited nature of the empirical results, lack of details about the simulation, too much emphasis in describing well-known RNN architectures, and lack of systematic strategy for applying this technique in other systems. The rebuttals did not change the borderline scores.

Thus, this is an instance where the AC must be a bit more involved in the decision. After reading the paper and reviews, the AC felt that this work was not sufficiently general in its application. Ultimately, using artificial neural networks to fit neural data is common practice nowadays, so really, this paper serves as a proof-of-concept for replacing a complex biophysical model with a simpler RNN. But, given that RNNs are quite good at modelling sequence data, it's not terribly surprising that this works. Moreover, though the authors do a very careful search over network design decisions, they don't provide a systematic strategy for others to employ if they so wished. Also, the authors do not provide much insight into what the RNNs learn that might help us to better understand the modelled neural circuits. And most importantly, this only demonstrates the effectiveness for systems where we have biophysical models with well-established accuracy, which is not the case for most neural circuits. Given these considerations, a reject decision was reached.
Decision: Reject