Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors construct a GAN network for generating artificial calcium imaging activity from large neural populations. This is an interesting but relatively straightforward application of GANs in a neuroscience context. Unfortunately, I don't see the pressing need for such methods, and the analysis in the paper leaves me unconvinced that the author's method works really convincingly.

* Need: The authors motivate their method with experimental hurdles to recording large ensembles of neurons, but recent years have seen a dramatic improvement in this respect (see e.g. work of Steinmetz, Pachitariu, Stringer and colleagues) who routinely record from 1000s of neurons. The authors should carefully think about what their method allows one to do that is not possible with real data.

* Evaluation: I don't understand why the authors chose to evaluate their method on the deconvolved spikes. Surely this is interesting as well, but I would have liked to see actually more of the generated data and more evaluation on the generated data directly. From Fig. 2 it seems e.g. that the noise characteristics of the generated data are quite different than for the true data. Is this the case? The DG generated data is very hard to judge, since neither the artificial or generated data are shown, just summary statistics in Fig. 1.

* Clarity: I would have liked to see a clearer explanation of what goes into the network and what comes out. Somewhere, they mention a sliding window approach - a figure would go a long way.

* What was the role of the Wasserstein objective function? Where others tried? What was their outcome?

* Table 1: How was the DG model for the experimental data estimated? Combining a normal DG with the forward model of Friedrich? 


typo: p8: ref to Donahue is having () in wrong place  
review_title: Interesting, but straightforward applications of GANs

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The paper proposes to use a GAN framework to generate the realistic neuronal calcium signals, enabling to scale-up the neuronal population activity data. The solution is based on WAVEGAN architecture with Wasserstein distance to train on calcium fluorescent signals. The experiments are performed in comparison to artificial calcium signals with known ground-truth closely resembles the underlying data distribution. The accuracy of the approach, robustness of generated signals from the model are evaluated.

Strengths:
- The application is very relevant as an increase in calcium imaging samples helps analyze large populations of neurons.

Weaknesses:
- This paper lacks contribution, and the novelty of the work is limited in terms of methods.
- Fig.2 does not actually show significant activation locality in the corresponding neurons; the peaks are not present in the generated signal for Neuron 27, similarly for Neuron 75 (completely mismatched with recorded data).
- This is further confirmed by Fig.3, where the generated markers are scattered across all the neurons in time, the neuronal activity irrelevant to recorded activity.
- Figures 3, 4, and 5 are not very informative as no legend is included. 
- Here, the author aims to increase more realistic calcium signals for analysis of neuronal population, but how it will be useful for the healthcare/research community in terms of applications is not clear.
- In Fig.2, the generated recording does not have an inspired spike for the Neuron 27. Please clarify this.
Details for reproducing the experiments of dichotomized gaussian data are left unclear.

Major Comments:
- Why are the Pearson correlation values low (i.e., between -0.25 and 0.25)?.
- Please clarify whether the Pearson correlation is calculated between the recorded signal and the generated signal?.
- What is the performance when the model trained on real calcium signals recorded from the primary visual cortex of behaving mice and tested on artificial data?
- Suppose, if authors perturb the real calcium signals data, is it affecting the model performance?.
- Typos:
Abstract: Here we propose (Here, we propose)
Introduction: paragraph-1, high quality (high-quality)

Addressed Concerns:
- corrected the typos
- Some figures are updated
 
Not Addressed:
- Figure 1b is not uniform as 1a (legend information is missing)
- As mentioned in weaknesses: the novelty aspect of this work is missing.
- Further, the authors have not answered question 1 in major comments.
- Based on Fig 2 & 3, Calcium GAN generated signals are mismatching with recorded data. How one can deploy this model in real-time when model results are not identical at all?.
 
review_title: Official Blind Review #3

final_score: rating: 3: Clear rejection
review_content: This manuscript proposes a GAN-based approach to generate calcium traces of neurons by using adapting methods from Wasserstein GANs and WaveGAN to produce the traces.  Using OASIS to extract spikes gives similar patterns to real data on a variety of metrics. This is confirmed over a relatively large number of neurons.

The big pitfall of this manuscript is that the authors have not demonstrated the scientific utility of their method.  The purported rationale, "Generative models of neuronal activity hold the promise of alleviating the above problem by enabling the synthesis of an unlimited number of realistic samples for assessing advanced analysis methods," is not shown.  The motivation about how generative approaches can provide additional data to find higher-order relationships is fraught: perhaps the advanced analysis techniques would find more patterns by augmenting fake data, but it doesn't really increase the scientific confidence nor address the fundamental uncertainty due to limited data.  While there are some cases where research groups have used GANs to help advanced analysis techniques, such as in domain adaptation and semi-supervised learning, those cases are not shown here, nor is it clear how to build them in.  This paper needs to be revised with a clear use case to motivate it.

Pros:

The manuscript is clearly written, and shows an effective use of GANs to mimic scientific data.

Significant exploration of the results are shown.

Cons:

The metrics shown compare derived properties of the traces, not the traces themselves.  This is limiting.  In Figure 2, the real data and synthetic data have clearly different noise levels and background patterns.  If the results are visually different in a relatively small figure, it points that more exploration of the traces themselves and the generative quality needs to be explored.

The title suggests that "calcium imaging data" is being generated, but that's not really true.  Instead, the calcium traces on individual neurons are being generated, which is a derived product from calcium imaging data.  Perhaps this is just semantics, but it's misleading and should be clarified up front.

As in the major point, no clearly elucidate use case.

The lack of conditioning (on trials, behavior, unlying spike train) is a major limitation, since for most scientific applications we care about recovering these properties.
review_title: Effective calcium imaging trace generation but lacks demonstrated utility

Meta-review: The approach proposed here have raised major concerns from multiple reviewers especially concerning the novelty and the experimental validation procedure. Authors did not succeed in convincing reviewers of the value of their work for ML or calcium imaging processing.
Decision: Reject