Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to establish a dataset / benchmark for gene regulatory network inference from single-cell data.
strength_and_weaknesses: The paper is hard to follow for me. From the abstract and introduction, it seems it aims at providing a dataset / benchmark for GRN inference that would be very easy to understand and use by researchers from outside of computational biology or network inference fields - something like DREAM 3/4/5 GRN challenges from a decade ago. However, starting from Section 3, the authors introduce a very specific formal structure (SCM) that also does not necessarily reflect the underlying biology (e.g. the DAG assumption) - it is not clear what purpose this serves. The choice of benchmark evaluation metrics (Section 4) seems somewhat ad-hoc (why STRING and not other network repository, why use just true positives instead of metrics based on precision-recall and ROC used in DREAM). On the other hand, the paper lacks details about how to actually use the dataset in practice, whether there is a clear leaderboard etc. - details that would be essential in establishing CausalBench as the benchmark for the field. It does not help that the github repository mentioned in the introduction is empty.
clarity,_quality,_novelty_and_reproducibility: The paper lacks clarity and novelty.
summary_of_the_review: The paper fails in achieving what a good benchmark/dataset paper should demonstrate: clarity, sound assumptions and design choices, and practicality. Reading it did not bring me any closer to being able to quickly, effortlessly evaluate a GRN inference method I may develop.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper propose a suite of tools for benchmarking network inference on both observational and interventional data single-cell gene expression data.  They run several network inference / causal discovery methods for recovering graphical relationships from scRNAseq data and evaluate the performance of the methods using existing putatative knowledge about specific gene-gene interactions as well as by comparing performance of observational and interventional data.  
strength_and_weaknesses: - I think the paper tackles an important problem and provides a great service by colating these datasets and metrics.  
- My main complaint is that the paper advertises it's contribution as providing a benchmark for network inference on large scale scRNA seq data although they admit that there are no gold standard datasets available from which to actually benchmark these methods.  This is a noted limitation, but my concern is that this is a really big limitation.  It's really hard to know whether the results are biologically meaningful, or really just noisy and reflect the difficulty of identifying useful benchmarks.
- Along these lines, I think the "Metrics" section needs a lot more clarification.  The authors refer to equality tests on "the two distributions" as well as Wasserstein distance on "the two empirical distributions".  What are these distributions of? There could be a lot more detail here regarding the data / statistics used to create these distributions and how the tests were calculated.
- While I understand some of the difficulties, it's hard to know how much to read into the TPR vs TP counts graphs.  The authors need to explicitly discuss the potential for false positives / false positive rates.  Are there any ways to identify putative false positives that can be used as a pseudo-benchmark?


clarity,_quality,_novelty_and_reproducibility: - The authors claim to have all code available in a github repository, but the repository seems to be empty.
- Overall, I think the paper lacks technical precision needed for publication.  There are many cases where wordy but vague descriptions are used in place of more precise statements.  E.g. "satisfactory results are only obtained on the RPE1 dataset".  What counts as satisfactory?  What is the reason the other results are unsatisfactory? At the same there is additional and unneeded notation and technical details regarding SCM.  In my opinion, this doesn't add much, since you are not describing a new SCM-based method and it is never referenced again.  Thus the background on SCMs seems superflous.
- Re-check for typos and language, e.g. "appraoch"

summary_of_the_review: Overall, I think the authors take on a challenging and important task of collating multiple datasets and implementing many causal discovery methods on these datasets.  Overall, I think the paper lacks clarity and justification around the statistical benchmarks employed.  As such, it's unclear how much value is actually added in the suite of benchmarks proposed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present CausalBench, a framework to benchmark causal gene regulatory network (GRN) inference methods on perturbational single-cell RNA sequencing data (scRNA-seq). It includes evaluation metrics, baseline implementations of relevant inference methods and access to perturbational scRNA-seq data. With CausalBench the authors evaluate the ability of recovering “silver-standard” ground truth networks and how they are affected by sample size.
strength_and_weaknesses: Strengths:
- Authors are aware of and nicely describe the assumptions and limitations of this evaluation task and data.
- Including both biological and statistical metrics for evaluation of method performance.
- Evaluation frameworks allow to easily test if novel methods improve over baselines, motivating the development of improved methods.

Weaknesses:
- Authors propose novel evaluation metrics but did not implement classical ones such as the comparison against ChIP-seq derived networks, see (Pratapa et al. 2020).
- Authors miss to evaluate classic methods, such as SCENIC (Aibar et al. 2017), that have been shown to be the top performers in other benchmarks (Pratapa et al. 2020). Although these methods are not causal, it would be beneficial to assess if the modeling of causality actually improves GRN inference.
- While the authors provide different metrics, it is still not clear which methods perform better than others consistently. A “consensus” score based on rankings would improve interpretation. Moreover, there is no discussion about which methods overperform the others and why.
- Although the authors are aware of the limitation of perturbation experiments, no quality control assessment is performed to test whether the perturbation actually worked. Samples where the perturbed gene still shows high levels of gene expression should be removed or at least accounted for.
- The partition of the data due to scalability issues raises concerns since, as the authors already mention, it breaks the no-latent-confounder assumption. Authors chose partition sizes such that the running time remains below 30 hours but do not mention anywhere the actual numbers for each method. In order to comply with this rule, some methods might use fractions of the feature space that are too small to generate valuable graphs.
- In figure 3, authors mention “significant” increase but do not perform any statistical test to corroborate it.


clarity,_quality,_novelty_and_reproducibility: The presented work is of good quality and is overall clearly explained, allowing reproducibility of the results. Regarding originality, some of the metrics that the authors are proposing are novel but others have been previously used. Moreover, while it is good to propose novel metrics, the authors miss to include some of the classical ones, which would make comparisons to previous work easier.

summary_of_the_review: The author's contributions to the GRN evaluation field are somewhat new but some aspects already exist in previous work. Moreover, the authors are missing some previous contributions, limiting the added cumulative value of it. On the other hand, the description of the problem, its assumptions and the description of the methods is overall clear and of good quality.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces CausalBench, a framework to benchmark gene regulatory inference methods using perturbational scRNAseq data. Although having access to perturbational scRNA data can help in discovering causal gene networks, generating large-scale perturbational data where every gene undergoes some genetic intervention (knockdown) is very cost prohibitive. There is a recent study that generates genome-wide perturbational scRNA data in only two cell types. The authors of CausalBench have used these data to benchmark some of the well-known network inference methods which use observational and interventional data. For the evaluation, they propose two approaches: biological and quantitative. For the biological evaluation, they use known gene-gene or protein-protein interaction data, and for the quantitative, they use some statistical tests to validate the effects of interventions. 
strength_and_weaknesses: Strength:

- This paper tries to take the first step towards benchmarking the causal gene regulatory network inference methods using large-scale perturbational scRNA-seq data, which is interesting and important for causal discovery in biology.

Weakness:

- The idea of having a framework to test the causal network inference methods is great if the benchmarking metrics are powerful and interesting enough. I think the metrics proposed in this paper are not that much interesting. For the biological metrics, as mentioned by the authors, the well-known gene-gene or protein-protein interaction data are not specific for the cell types that they are working. The underlying gene regulation rules are very cell-type specific and the inferred networks could not be reliably evaluated by general networks which have been derived from many cell types. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to read and understand. The novelty is only limited to introducing a a benchmarking bed for causal gene network inference methods using perturbational scRNAseq data.
summary_of_the_review: Causal gene network inference (DAGs) are very important in understanding the mechanisms of gene expression. However, learning such DAGs using only observational data is not possible. Recent Perturb-seq data provide helpful interventional data by genetically perturbing the target genes and measuring the expression of other genes. This data empowers the causal DAG learning methods, however, they should be tested and benchmarked via the same metrics to measure their efficacy. 

This paper takes the first step in the right direction by noticing the need for benchmarking causal network inference methods using Perturb-seq data. However, the benchmarking metrics are not comprehensive enough to be certain about the efficacy of such methods. My comments:

- Biological metric does not make so much sense to me. The underlying DAG driving the gene expression programs are very cell-type specific. So, the general gene/protein networks cannot be a good and reasonable metric to evaluate the correctness of the learned DAGs in K562 and RPE1 cells. 

- As a qualitative metric, the authors can use held out perturbations and see how well different methods can predict the effect of each perturbation on various genes. This has also been used in Lopez et al. (2022) (cited in the paper). 

- Why haven't the authors used the biggest Perturb-seq (2 M cells) in K562 cells as published in (Replogle et al., 2022)? It would be interesting to see how well/fast each method works in a very large dataset.

- Related to the previous comment, I think one another and yet interesting benchmarking metric would be how fast each method could be trained in each dataset. This kind of benchmarking has also been done in Lopez et al. (2022). By growing sequencing technology, it is not so far that we will have access to millions of cells and thousands of perturbations genome-wide, and the methods that could handle this huge computational barrier would be the winners. 

Minor commets:

Typos:   - we we observed --> we observed
             - in not large --> is not large
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses:   The authors propose a new benchmark for regulatory network inference/causal discovery based on single cell RNAseq data with interventional perturbations. While such a benchmark would be valuable for the community, the reviewers identified several issues with the authors' proposal and have recommended against accepting the paper. Two major issues were the benchmark does indeed reflect the underlying biology, and whether the proposed evaluation metrics are appropriate. 
justification_for_why_not_higher_score: Most reviewers recommended rejecting the paper.
Decision: Reject