Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper applies the GAN framework to learn a generative model of spike trains.  The generated spike trains are compared to traditional model fitting methods, showing comparable or superior ability to capture statistical properties of real population  activity.

This seems like an interesting exercise, but it’s unclear what it contributes to our understanding of neural circuits in the brain.  The advantage of structured models is that they potentially correspond to underlying mechanisms and can provide insight.  The authors point to the superior ability to capture temporal structure, but this does not seem like a fundamental limitation of traditional approaches.

The potential applicability of this approach is alluded to in this statement toward the end of the paper:

“...be used to describe and interpret experimental results and discover the key units of neural information used for functions such as sensation and behavior.”

It is left for the reader to connect the dots here and figure out how this might be done.  It would be helpful if the authors could at least sketch out a path by which this could be done with this approach.

Perhaps the most compelling application is to perturbing neural activity, or intervening to inject specific activity patterns into the brain.

review_title: Interesting idea, but applicability unclear 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The paper proposes to use GANs for synthesizing realistic neural activity patterns. Learning generative models of neural population activity is an important problem in computational neuroscience. The authors apply an established method (WGAN) to a new context (neuroscience). The work does not present a technical advance in itself, but it could be a valuable contribution if it created novel insights in neuroscience. Unfortunately, I do not see any such insights and am therefore not sure about the value of the paper.



Pros:

- Using GANs to synthesize neural activity patterns is novel (to my knowledge).

- Using GANs allows learning the crucial statistical patterns from data, which is more flexible than MaxEnt modeling, where one needs to define the statistics to be matched in advance.

- Using the critic to learn something about what are the crucial population activity patterns is an interesting idea and could be a valuable contribution.



Cons:

- Important architecture details missing: filter sizes, strides.

- The only demonstrated advantage of the proposed approach over MaxEnt models is that it models temporal correlations. However, this difference has nothing to do with MaxEnt vs. GAN. A 64d MaxEnt model does not care whether you’re modeling 64 neurons in a single time bin or 8 neurons in 8 subsequent time bins. Thus, the comparison in Fig. 3E,F is not apples to apples. An apples-to-apples comparison would be to use a MaxEnt model that includes multiple time lags (if that’s infeasible with 16 neurons x 32 bins, use a smaller model). Given that the MaxEnt model does a better job at modeling neuron-to-neuron correlations, I would expect it to also outperform the GAN at modeling temporal correlations. There may well be a scalability issue of the MaxEnt model to large populations and long time windows, but that doesn’t seem to be the current line of argument.

- GANs have well-known problems like mode collapse and low entropy of samples. Given the small amount of training examples (<10k) and large number of model parameters (3.5M), this issue is particularly worrisome. The authors do not address this issue, neither qualitatively nor quantitatively, although both would be possible:

  a) A quantitative approach would be to look at the entropy of the data, the MaxEnt model and the GAN samples. Given the binary and relatively low-dimensional nature of the observations, this may be feasible (in contrast to image data). One would potentially have to look at shorter segments and smaller subpopulations of neurons, where entropy estimation is feasible given the available amount of data, but it’s definitely doable

  b) Qualitative approaches include the typical one of showing the closest training example for each sample. 

- The novel idea of using the critic to learn something about the crucial population activity patterns is not fleshed out at all. I think this aspect of the paper could be a valuable contribution if the authors focused on it, studied it in detail and provided convincing evidence that it can be useful in practice (or, even better, actually produced some novel insight).

  a) Visualizing the filters learned by the critic isn’t really useful in practice, since the authors used their ground truth knowledge to sort the neurons. In practice, the (unsorted) filters will look just as uninterpretable as the (unsorted) population activity they show.

  b) Detection of the ‘packets’ via importance maps is an interesting idea to find potential temporal codes without explicitly prescribing their hypothesized structure. Unfortunately, the idea is not really fleshed out or studied in any detail. In particular, it’s not clear whether it would still work in a less extreme scenario (all eight neurons fire in exact sequence).

- Poor comparison to state of the art. MaxEnt model is the only alternative approach tested. However, it is not clear that MaxEnt models are the state of the art. Latent variable models (e.g. Make et al. 2011) or more recent approaches based on autoencoders (Pandarinath et al. 2017; https://doi.org/10.1101/152884) are just among a few notable alternatives that the authors ignore.



Questions:

- Can sequences of arbitrary length be generated or is it fixed to 32 samples? If the latter, then how do envision the algorithm’s use in real-world scenarios that you propose such as microstimulation?



Minor:

- There is nothing “semicomvolutional” here. Just like images are multi-channel (color channels) 2D (x, y) observation, here the observations are multi-channel (neurons) 1D (time) observations. 

- Fig. 3E is not time (ms), but time (x20 ms bins). Potentially the firing rates are also off by a factor of 20?

- Why partition the training data into non-overlapping segments? Seems like a waste of training data not to use all possible temporal crops.
review_title: Some potentially interesting ideas, but many weaknesses

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: [Summary of paper] The paper presents a method for simulating spike trains from populations of neurons which match empirically measured multi-neuron recordings. They set up a Wasserstein-GAN and train it on both synthetic and real multi-neuron recordings, using data from the salamander retina. They find that their method (Spike-GAN) can produce spike trains that visually look like the original data, and which have low-order statistics (firing rates, correlations, time-lagged-correlations, total sum of population activity) which matches those of the original data. They emphasize that their network architecture is 'semi-convolutional', i.e. convolutional in time but not across neurons. Finally, they suggest a way to analyse the fitted networks in order to gain insights into what the 'relevant' neural features are, and illustrate it on synthetic data into which they embedded these features.

[Originality] This paper falls into the category of papers that do a next obvious thing ("GANs have not been applied to population spike trains yet"), and which do it pretty well: If one wants to create simulated neural activity data which matches experimentally observed one, then this method indeed seems to do that. As far as I know, this would be the first peer-reviewed application of GANs to multi-neuron recordings of neural data (but see https://arxiv.org/abs/1707.04582 for an arxiv paper, not cited here-- should be discussed at least).  On a technical level, there is very little to no innovation here -- while the authors emphasise their 'semi-convolutional' network architecture, this is obviously the right architecture to use for multivariate time-series data, and in itself not a big technical novel. Therefore, the paper should really be evaluate as an `application' paper, and be assessed in terms of i) how important the application is, ii) how clearly it is presented, and iii) how convincing the results are relative to state of the art. 

i) [Importance of problem, potential significance] Finding statistical models for modelling and simulating population spike trains is a topic which is extensively studied in computational neuroscience, predominantly using  model-based approaches using MaxEnt models, GLMs or latent variable models. These models are typically simple and restricted, and certainly fall short of capturing the full complexity of neural data. Thus, better, and more flexible solutions for this problem would certainly be very welcome, and have an immediate impact in this community.  However, I think that the approach based on GANs actually has two shortcomings which are not stated by the authors, and which possibly limit the impact of the method: First, statistical models of neural spike trains are often used to compute probabilities e.g. for decoding analyses— this is difficult or impossible for GANs. Second, one most often does not want to simulate data which match a specific recording, but rather which have specified statistics (e.g. firing rates and correlations)— the method here is based on fitting a particular data-set, and it is actually unclear to me when that will be useful.

ii) [Clarity] The methods are presented and explained clearly and cleanly. In my view, too much emphasis is given to highlighting the ‘semi-convolutional’ network, and, conversely, practical issues (exact architectures, cost of training) should be explained more clearly, possibly in an appendix. Similarly, the method would benefit from the authors releasing their code.

iii) [Quality, advance over previous methods] The authors discuss several methods for simulating spike trains in the introduction. In their empirical comparisons, however, they completely focus on a particular model-class (maximum entropy models, ME) which they label being the ‘state-of-the-art’. This label is misleading— ME models are but one of several approaches to modelling neural spike trains, with different models having different advantages and limitations (there is no benchmark which can be used to rank them...). In particular, the only ‘gain’ of the GAN over ME  models in the results comes from their ability of the GAN to match temporal statistics. Given that the ME models used by the authors are blind to temporal correlations, this is, of course (and as pointed out by the authors) hardly surprising. How does the GAN approach fair against alternative models which do take temporal statistics into account, e.g. GLMs, or simple moment-based method e.g. Krumin et al 2009, Lyamzin 2010, Gutnisky et al 2010— setting these up would be simple, and it would provide a non-trivial baseline for the ability of spike-GAN to outperform at least these models? While it true that GANs are much more expressive than the model-based approaches used in neuroscience, a clear demonstration would have been useful.

Minor comments: 
  - p.3: The abbreviation “1D-DCGAN” is never spelled out.
  - p.3: The architecture of Spike-GAN is never explicitely given.
  - p.3: (Sec. 2.2) Statistic 2) “average time course across activity patterns” is unclear to me -- how does one select the activity patterns over which to average? Moreover, later figures do not seem to use this statistic.
  - p.4: “introduced correlations between randomly selected pairs” -- How many such pairs were formed?
  - p.7 (just above Discussion) At the beginning of this section, and for Figs. 4A,B, the texts suggests that packets fire spontaneously with a given probability. For Figs. 4C-E, a particular packet responds to a particular input. Is then the neuron population used in these figures different from the one in Figs. 4A,B? How did the authors ensure that a particular set of neurons respond to their stimulus as a packet? What stimulus did they use?
  - p.8 (Fig. 4E) Are the eight neurons with higher importance those corresponding to the packet? This is insinuated but not stated.
  - p.12 (Appendix A) 
    + The authors do not mention how they produced their “ground truth” data. (What was its firing rate? Did it include correlations? A refractory period?)
    + Generating samples from the trained Spike-GAN is ostensibly cheap. Hence it is unclear why the authors did not  produce a large enough number of samples in order to obtain a 'numerical probability', just as they did for the ground truth data? 
    + Fig. S1B: The figure shows that every sample has the same empirical frequency. This indicates more a lack of statistical power rather than any correspondence between the theoretical and empirical probabilities. This undermines the argument in the second paragraph of p.12. In the other hand, if the authors did approximate numerical probabilities for the Spike-GAN, this argument would no longer be required.
  - p.13 Fig. S1A,B: the abscissas mention “frequency”, while the ordinates mention “probability”
  - p.25 Fig. S4: This figure suggests that the first layer of the Spike-GAN critic sometimes recognizes the packet patterns in the data. However, to know whether this is true, we would need to compare this to a representation of the neurons reordered in the same way and identified by packet. I.e. one expects something something like figure like Fig. 4A, with the packets lining up with the recovered filters when neurons are ordered the same way.

review_title: A new application of GANs: Simulating neural spike train recordings

Meta-review: This paper proposes a novel application of generative adversarial networks to model neural spiking activity.  Their technical contribution, SpikeGAN, generates neural spikes that accurately match the statistics of real recorded spiking behavior from a small number of neurons.

The paper is controversial among the reviewers with a 4, a 6 and an 8.  The 6 is short and finds the idea exciting but questions the utility of the proposed approach in terms of actually studying neural spiking.  The 4 and 8 are both quite thorough reviews.  4 seems to mostly question the motivation of using a GAN over a MaxEnt model and demands empirical comparison to other approaches.  8 applauds the paper as a well-executed pure application paper, applying recent innovations in machine learning to an important application with some technical innovation.  Overall the reviewers found the paper clear and easy to follow and agree that the application of GANs to neural spiking activity is novel.  In general, I find that such high variance in scores (with thorough reviews) indicate that the paper is exciting, innovative and might stir up some interesting discussion.  As such, and under the belief that ICLR is made stronger with interesting application papers, I feel inclined to accept as a poster.

Pros:
- A novel application of GANs to neural spiking data
- Addresses an important and highly studied application area (computational neuroscience)
- Clearly written and well presented
- The approach appears to model well real neural spiking activity from salamander retina

Cons:
- Known pitfalls of GANs aren't really addressed in the paper (mode collapse, etc.)
- The authors don't compare to state of the art models of neural spiking activity (although they compare to an accepted standard approach - MaxEnt)
- Limited technical innovation over existing methods for GANs
Decision: Accept (Poster)