Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors provide datasets of recorded neuronal activity from populations of neurons in various brain regions. The intend for these benchmarks to facilitate latent variable models of neuronal recordings and propose a type of held-out validation called co-smoothing for evaluation. 
strengths: The main strength of this work is the focus on providing a benchmark for unsupervised learning. This is often harder to do, than associating or annotating data with a few label classes. The authors also provide and test evaluation metrics for latent variable models of this type of data including co-smoothing and behavioral decoding. 
weaknesses: 1. This paper is very difficult to read for the general machine learning researcher. it assumes a lot of prior knowledge in a narrow area of neuroscience. For instance the specific readings that the authors provide are very different than the more common fMRI datasets used in computation and the authors do not state basic assumptions about the data types. For example it is unclear what pre-processing steps have to be taken to go from raw data to spike information, and presumably it is a lot since there are whole benchmark suites designed for this. 

2. The authors conflate machine learning terms. They call their LVMs "generative models," whereas they seem to merely be autoencoding or low-rank approximating the recording i.e., reconstructing the entire recording from lower dimensional latent variables. 

3. The held-out data is termed strangely. This is just the standard test/training split in ML, the authors just find compressed representations of recordings I believe and test reconstruction rather than any sort of "prediction," as the authors say. The term co-smoothing is a bit strange to me as well and the origin of that is not mentioned. 

4. Moreover, the authors don't seem to be seeking input from the ML community on what to do with latent representations, there is no mention of visualization, clustering, trajectory analysis or other tasks on the embedded space but only narrow references to tasks previously done in this subfield of neuroscience. This the variety of latent variables that they seek seem very limited to the types that are in references such as GLMs and gaussian processes. 


correctness: The dataset seems to be sound, as do the experiments. However there are not enough experiments examining latent variables generated by modern machine learning methods. 
clarity: The paper is not clear to the non-neuroscientist reader. 
relation to prior work: It is not clear to me what the relationship between Neural Latent Benchmarks and Neurodata without borders and DANDI is exactly. It seems these datasets are already available on DANDI . https://dandiarchive.org/dandiset/000070/ 
additional feedback: I would rewrite this submission for a more general audience.
review_title: Neuronal Spiking Recording benchmarks 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: I have no background in neuroscience but I enjoyed learning about this benchmark resource for latent variable modeling of neural population activity. 
strengths: The authors do a good job introducing a set of benchmark data consisting of controlled neuronal experiments with rhesus monkeys. The data are described clearly and the benchmark is motivated well. The use of a hold-out test set and competition-style format make the resource even more attractive from a benchmarking perspective. The paper is well-written and the repository includes code for reproducing their baseline reporting. The data are also well-illustrated. 
weaknesses: The biggest weakness is that the method evaluations lack any statistical comparisons or confidence estimates. As a result it is not possible to say whether any of these methods are significantly better than any other. The algorithms should be compared over multiple training instances with shuffled data, or a bootstrap analysis should be performed to estimate performance variability among the methods for each task. 

- it seems that this data was collected in 2010, and has been the subject of several research papers. With that in mind, the authors should do more to explain why such a benchmark is necessary by contrasting with findings from prior work with these data.


correctness: ML evaluation could be improved as suggested above
clarity: Yes
relation to prior work: Prior work is cited, but as I mention above it would be good to do more to distinguish and motivate this benchmark effort from previous work with these data.
additional feedback: - "MC_Maze as a Neuroscience MNIST" is repeated several times. MNIST is important to the history of deep learning of course, but IMO these days it is seen as an over-used / unrealistic benchmark with rampant over-fitting to the test set.
review_title: Interesting benchmark resource for neuroscience

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a standard benchmark for evaluating latent variable modeling (LVM) of neural population activity. Four diverse datasets are proposed along with new metrics and baseline models. 
strengths: 1. A standardized evaluation suite for latent variable modeling on neural activity data.
2. Propose new evaluation metrics for LVMs.

weaknesses: I think the proposed "behavior decoding accuracy" is a very important metric. Because the proposed benchmark is specifically for LVMs so it would be important to directly evaluate the quality of the latent representations. However, I have some concerns about this metric:
1. To my understanding, the behavioral data (monkey hand velocity) is also provided in the train/eval data. Are the LVMs supposed to use this data to guide the learning of the latent representation? If this is the case, then the latent representation could be overfitting to the behavioral data.
2. One specific kind of behavior, monkey hand velocity, is used for behavior decoding. This could introduce a bias toward this specific kind of behavior. Also according to Table 1, the AutoLFADS can already get a more than 0.9 R2 score, which leaves little space for other models to improve.

I also have some concerns regarding the related work, I'll elaborate below in the Relation To Prior Work section.
correctness: See above.
clarity: The paper well-written and easy to follow.
relation to prior work: I suspect that there are other neural activity datasets (not necessarily benchmarks) that are used previously in evaluating LVMs. However, I didn't find any related works discussed along this line. Please correct me if I missed something or the authors should give a comprehensive discussion on this. I think it is important to make the difference between this dataset and the existing datasets clear, or it will be hard to evaluate the contribution of this paper.

On the other hand, latent variable models usually refer to models that infer/learn latent variables from the observed variables. Not all unsupervised learning models or generative models are LVMs. Specifically, I don't think MLMs like BERT are LVMs. And the evaluation of MLMs does not correspond to the evaluation of "latent representations". I think the authors should do a more thorough literature review on the evaluation methods of LVMs in machine learning researches. One common evaluation method for LVMs is to interpolate the latent space along one direction and observe the meaningful transition in outputs. This does not necessarily correspond to a provided label or category information.
additional feedback: See above.
review_title: New benchmark for evaluating LVMs for neural activities

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces a benchmark for evaluation of latent variable models for neural population activity data. The authors conduct experiments on monkey by making it do a variety of different tasks and recording both its behavior and neural activity. The paper does a thorough job at evaluating different types of algorithms on the curated data and also use different evaluation criteria. Given the variety of tasks across which data is obtained and the existence of private data for model ranking to prevent overfitting to test data, I believe that the paper provides important contributions to the field. 
strengths: This work is quite significant for improved benchmarking of different algorithms that model neural activity data. There has been an increased interest in understanding of neural activity through latent variable models for both forward prediction as well as behavior prediction. Given this increased interest, a proper benchmark with diverse task candidates and evaluation metrics is quite important for future research in this area.

The existence of private data for providing a leaderboard and preventing overfitting by the research community is another positive trait of the provided benchmark.

The consistency of different evaluation metrics on the models considered provides confidence about the validity of the data and the metrics.
weaknesses: While the authors evaluate multiple relevant models on the data that they curate, they do not provide standard deviation corresponding to all the metrics. It would be nice to perform evaluation over multiple random seeds and provide standard deviation of all the metrics presented in the paper.

Throughout the start, the paper claims that progress in latent variable models is impeded by lack of standardized datasets and evaluation metrics. This feels like an over-claim because LVMs typically extend beyond neural activity data and there are a lot of standardized datasets and evaluation metrics that people have developed for evaluation of LVMs in general settings. It would be nice to establish early on that the paper concentrates on standardization of LVM evaluation on **neural activity data**.
correctness: The dataset is constructed in a sound way and the evaluation methods and experiment design are appropriate for the tasks introduced in the paper. The authors discuss in detail how each task that they provide is curated in the Appendix.
clarity: The paper is clearly written and is generally easy to follow through. The organization of evaluation metrics, baselines and codebase pipeline could be better.
relation to prior work: The authors do a good job at discussing existing work in the field and identifying the need for standardization of evaluation. They also do a pretty good job at reporting the performance of a wide variety of models ranging from more early approaches like Gaussian smoothing to Deep Neural Network based approaches.
additional feedback: It would be nice to see the discrete latents associated with SLDS on the provided data to see if some discrete interpretable latents are being observed on it.
review_title: Review of the Neural Latents Benchmark '21

Meta-review: The authors are proposing a dataset of spiking neural data recorded primarily from the motor areas of the primate brain while the primates are completing various reaching tasks. The neural recordings are also supplemented with hand kinematics data for the ground truth behavioural outcome of the neural activity. The authors have put together a nice evaluation challenge with private heldout data that will never be accessible to the users of their datasets, but which will be used privately on their servers for evaluating the submitted models. The dataset/leaderboard/competition setup can be a great way to push forward the developments of more accurate unsupervised models of spike activity.

The reviews are very mixed (4, 7, 5, 7) and all reviewers have quite low confidence. The main drawbacks brought up by the reviewers are:

Wrong/too-broad use of the term Latent Variable Model (LVM)
Lack of more traditional to the ML literature evaluation measures for the learnt latent spaces
Lack of statistical analysis of the results - are there statistically meaningful differences?
Is it fair for the competition to allow users access to the behavioural data - what if some models use this supervision signals while others don't?
The authors have addressed all of these points reasonably in my opinion during the discussion period. They have agreed to change how they refer to the baselines (not LVMs), they have added statistical tests, they have added a fair discussion on how the more traditional analysis of the learnt latents is not applicable in their use case, and how access to the behavioural data does not help these models pass all of their evaluation tests.

Having read the paper myself, and coming from the neuroscience background, I believe that the paper merits publication. It will be a great resource to encourage competition in developing good unsupervised learning methods for spiking neural data (which is being collected in more and more volume with the emergence of the better large scale recording methods), and gaining a better understanding of the latent dynamics in these populations of neurons. For this reason, I recommend acceptance.
Decision: Accept