Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors collect a multi-modal single cell dataset and formulate three tasks (modality prediction, matching and joint embeddings) using these data. The dataset is large and the collection is well-principled. This looks to be an excellent resource for computational biologists as it captures multiple 'layers' of molecular regulation at single-cell resolution.
strengths: * The authors present a large multi modal bone marrow benchmark dataset
* The authors formulate a collection of novel tasks enabled by the collection of these modalilities with relevant metrics.
* The authors cater for batch effects in the experimental measurements.
weaknesses: * Some of the biological details in the introduction can be tightened up though it is overall a good treatment of the background suitable for an ML audience. 
* I'm hesitant to comment more on the soundness of the paper. Perhaps other reviewers more familiar with the subject can be more probing.
correctness: * The link https://openproblems.bio/benchmark_dataset is broken
* The paper seems methodologically sound.
clarity: The paper is clear and well written
relation to prior work: Previous work is discussed, though I would contend blind evaluation competitions are prevalent in computational biology beyond the examples mentioned.
additional feedback: This appears to be a large initiative. It would be of interest to the field for the authors to describe in more detail the unique challenges and logistics behind co-ordinating such an effort. For example, they allude to the requirement for the maintenance of a cold chain for some the samples/reagents/
review_title: An impressive multi-modal dataset for single cell biology

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a new multimodal dataset of single-cell molecular data that includes RNA gene expression, cell surface protein markers, and DNA accessibility data. The data was obtained from the bone marrow mononuclear cells (BMMCs) of 9 donors and the data was gathered using two different commercially-available technologies (ATAC+GEX, ADT+GEX) at multiple laboratories. Specifically, the accessibility of 119254 genomic regions, the expression of 15189 genes, and the abundance of 134 surface proteins is measured. 

The dataset is formatted according to the Common Task Framework (CTF), which has been a standard dataset format for previous datasets such as the Critical Assessment of protein Structure Prediction (CASP) dataset. The proposed tasks are predicting one modality from another, matching cells based on their profiles in different modalities, and creating a representation of cells that captures multiple modalities, along with evaluation metrics for each task. The usage of the dataset is supported by an open-source framework that facilitates the centralized evaluation of methods.

strengths: Additional modalities, multi-site data generation, and focus on batch effects are strengths of this dataset over previous datasets. 

This dataset should be interesting to researchers working on multi-modal models and computational biology.  

Accessibility is a major strength of this benchmark dataset. The use of the Common Task Framework should facilitate the development of machine learning methods for this task, and  centralized evaluation should allow easy and fair comparisons between competing methods as well as provide benchmarks for future work after the conclusion of the challenge. 

The social implications is also a strength of this dataset, as advances in understanding cellular transcription and translation are important and may be beneficial to the study of disease, and this dataset may aid in this. 

weaknesses: One concern I have is the diversity of the dataset. It’s not clear to me that cell samples from only 9 donors is enough to create a representative sample. 

Another concern I have is that since the dataset only includes data from healthy individuals, this seems to limit the relevance of methods trained on this dataset to the study of disease.





correctness: The dataset is generally constructed in a sound way.
clarity: The paper is well-written and organized. 
relation to prior work: It is clear that the proposed dataset differs from existing datasets in that it was collected at multiple sites, has a private test split, and includes both developing and differentiated cell types. It also has three data modalities, as opposed to previous work at that scale which only included two modalities.
additional feedback: There is a typo on 57: “refered” should be “referred”.
review_title: Official Review of Paper313 by Reviewer RB4f

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Single-cell technologies have been successfully applied to basic and translational research questions and revealing the complex inner workings of the cell. Multi-modal single-cell technologies are emerging as effective tools for quantifying complex subcellular networks in health and disease. However, there are few large, standardized, multi-modal single cell benchmarks to stimulate algorithm and method development.
 
Lücken, Burkhardt, Cannoodt, Lance, et al. present a large, multi-institutional, multi-modal dataset profiling single cells from donor bone marrow samples. Among the many strengths of this contribution include the large multi-institutional multi-modal dataset collected using the best practices in the field. The tasks are well defined with clear evaluation metrics according to the Common Task Framework. The large, curated, and well documented multi-modal dataset represents an outstanding contribution to the field.

Updated review comments:
The authors have addressed many of the concerns raised by the reviewers and the revised manuscript is improved. I had similar concerns about specimen number and diversity, however I am familiar with how laborious and expensive it is to acquire and process the data, especially given the structure of the experiments. The baselines are sufficient to set the minimum performance and upper/lower bounds on performance since this is primarily a dataset contribution. I appreciate the thoughtful consideration of ethical issues and the additional internal review described by the authors addresses my concerns. The additional discussion of lessons learned from this benchmark will hopefully help others in the field. As described above, the dataset is large for single cell studies and well-curated with a specific task for benchmarking multi-modal inference in single-cell data. This will surely be a very interesting and useful dataset for multi-modal single-cell gene regulation.
strengths: The dataset is large, original, and was collected using the latest technology to profile DNA accessibility, RNA and protein expression from single cells. The adherence to the Common Task Framework is another strength with their clearly-defined tasks and evaluation metrics. The dataset was rigorously constructed using multiple institutions, multiple donors, expert annotation/processing, and an experimental design that nested batch effects to capture the many levels of biological and technical variation inherent in single-cell data. In addition, the free compute from Saturn Cloud will hopefully stimulate global participation and bright minds from all countries and socioeconomic groups. Overall, the data curation, processing, and expert-annotation represent a tour de force of molecular biology, single-cell technology, and computational genomics.
weaknesses: There is very little description of initial performance benchmarks for the proposed tasks, however I was able to find the "dummy baselines" on the Jupyter notebooks hosted on GitHub. As this is primarily seen to be a dataset contribution, state-of-the-art performance is not expected, but including some baseline metrics for the tasks could help readers unfamiliar with the field appreciate the difficulty of problem.

Ethical and societal issues as well as dataset limitations are discussed, but primarily within the supplemental Dataset Datasheet. The potential reidentification of research participants using genomic data in combination with public genealogical databases or public records is briefly discussed but the ethical and societal implications are not further discussed. Although the researchers will have no direct control over how the data is used once released to public scientific databases, it is recommended to discourage use that may harm the employability, insurability, reputation or financial standing of the research participants (it is common sense, but unfortunately needs to be explicit due to rare bad actors).

Also, more discussion of the dataset limitations would be appreciated. The challenges described in the text begin this discussion, but predominantly focus on the time, labor, and financial cost. What other lessons were learned during creating the benchmark? If the process had to be repeated, what would the authors do differently? How might future creators of benchmark datasets learn from and improve upon this example?

The introduction, though very well-written, could be tightened to allow for any of these topics above to be expanded upon.
correctness: The dataset is rigorously constructed applying the current best practices in the field. The design of the dataset collection with nested batch effects was well thought out. The benchmark tasks and evaluation metrics are well defined in the main text or appendix. The dataset is accessible on the associated website (https://openproblems.bio/neurips).
clarity: The manuscript was well-written with a detailed description of the prior work and dataset curation as well as the specific tasks involved in the challenge. The manuscript formatting adheres to the guidelines. No spelling or grammar errors were identified.
relation to prior work: The relevant prior work is discussed. This benchmark dataset represents a significant contribution with the potential to stimulate multiple lines of research. 
additional feedback: All feedback is described above.
review_title: Updated Review of "A sandbox for prediction and integration of DNA, RNA, and protein data in single cells"

Meta-review: There is uniform consensus among reviewers to accept the paper.
Decision: Accept