Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery. Specifically, they introduce various genome-wide CRISPR screens within immunology that evaluate the causal effect of intervening on a large number of genes in model systems in order to identify the genetic perturbations that induce a desired phenotype. The work focuses on using counterfactual estimators of experimental outcomes to propose experimental hypotheses for validation in in vitro experiments with genetic interventions (CRISPR) in order to discover potential causal associations between biological entities that could be relevant for the development of novel therapeutics. 

The authors standardize two types of datasets: three standardized feature sets describing interventions and four difference in vitro genome-wide CRISPR experimental assays. Two model types are used (BNN and Random Forest Regression) and nine different acquisition functions. 

Results are presented in Figures 2 and 3. 

main_review: Strengths
1) The authors are commended for addressing the hard problem of causal inference in biological systems. 
2) The authors give a good explanation of the methodology, datasets, and metrics.
3) A good explanation of the genetic assays are provided

Weakness:
1) The title is misleading. The paper is much more about gene knockdown than drug discovery. As presented, causal connections between specific loss-of-function mutations and immunologic phenotypes may be drawn, but drug discovery is not directly related. 
2) The authors do not provide any novel methodologies within the field of active learning or drug development.
3) The assays are limited to a small subset of immunologic phenotypes.

summary_of_the_review: Overall, the authors provide a decent dataset and benchmark for genetic knockdown experimentation in the setting of active learning. However, I believe the paper would be better suited for datasets/benchmarks venue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a benchmarking suite for evaluating active learning strategies to experimental design in drug discovery. The authors focus on active learning for in vitro genetic experiments, where the goal is to identify which genes or proteins underlie a certain disease (e.g., by using CRISPR technologies). For this task, there are typically billions of potential hypotheses to test. The experimental design space for such in vitro cellular experiments is extremely vast; thus, even with help from modern computers and machine-learning algorithms (which can search through the entire space), it remains practically impossible to exhaustively test every possible hypothesis that lives in this biological hypothesis space.

The paper contributes a benchmarking suite that can be used to evaluate how well active learning strategies are able to learn useful experimental designs. The paper has the following contributions:

- A new experimental design benchmark suite of genetic experiments for CRISPR screens with unique challenges not found in other benchmarks. This includes demonstrating cost-efficiency by using only a small number of experiments relative to the total number of hypotheses tested.

- An evaluation methodology for experimental designs on GeneDisco data sets, including methods for evaluating cost efficiency and quality of information gain during active learning. 

- Initial evaluations apply this approach to two candidate causal inference algorithms: one based on a Bayesian neural network and another based on a Random Forest. Further methodological development is required in order to explore more sophisticated algorithms for active learning.

- The paper is published with supplementary material, including the two experimental design benchmark datasets and all source code necessary to reproduce the results of this paper.
main_review: The strengths of this paper include :

-  The paper focuses on an important aspect of experimental design for drug discovery, which has not previously been studied. This type of benchmarking is invaluable to make the best possible use of limited experimental resources.

- The datasets are carefully designed with input from researchers in genomics and CRISPR screens, minimizing potential sources of error (e.g., no duplication).

- It has the right level of detail; it makes specific recommendations that can be readily implemented by developers or researchers interested in working with similar data sets.

- Source code is made publicly available online.

The limitations include: 

- The benchmarks contain relatively little information about how far apart genes are located (i.e., there is no distance metric). In practice, this information can be very useful for active learning algorithms. 

- The authors study only two candidate algorithms so far; however it is still an important step forward because the novel dataset and methodology enable other groups to conduct extensive evaluations on their own machine learning algorithms.

- The problem formulation should be clarified and is difficult to understand as there are multiple different datasets presented. 

- A thorough justification for the assumptions utilized in the parametric models; specifically the outcome Y, should be disclosed.

- How do the authors define the ground truth when evaluating the active learning strategies?

- How do the authors define hits for the different active learning strategies?

- The results show rather uniform performance regardless of the active learning strategy employed. Can the authors comment on the reasoning around this phenomenon and whether it is an artifact of the problem formulation or datasets employed?

- It is unclear if biological replicates are included in the experiments. A change in this fundamental would have a very substantial impact on experimental design benchmarks and biology interpretations. If replicates were not used, they should be added to the text. 
summary_of_the_review: All in all, this paper proposes a benchmarking suite that can help developers of active learning algorithms improve their programs. While the paper only considers a limited set of models, active learning algorithms, and datasets, it is an important step towards establishing best practices for in vitro experimental design. With source code publicly available online, it provides necessary tools (and datasets) to advance the state of the art for experimental design in drug discovery studies that use CRISPR technologies.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a benchmark for experimental design in drug discovery based on previously public data sets. The benchmark includes three standardized feature sets describing possible interventions and four different genome-wide CRISPR experiment results (from which the counterfactual outcomes for an intervention on each gene are obtained). In addition, the paper uses the benchmark to compare different active learning strategies (nine acquisition functions in total), where the goal is to improve prediction accuracy and hit ratio of interesting genes among the interventions selected by a specific (batch) acquisition function.

main_review: Pros:
- I feel this kind of benchmark would be useful. Although the  data sets are already publicly available, it's nevertheless nice if someone else has done the required work and one can start modeling immediately.
- The presentation is mostly clear.

Cons/questions:
- The paper is straightforward, does not have any particular shortcomings, and is potentially useful to the community. However, I feel it lacks the kind of machine learning novelty and significance that would be expected from an ICLR publication, and hence some other forum could be more suitable (though I'm happy to discuss this with the other reviewers and the area chair).
- Figures 2 and 3 require a lot of zooming in before they are readable, and they are not at all legible in a printed-out version.
- I found the discussion on the comparison of acquisition methods a bit superficial.
summary_of_the_review: A straightforward paper that presents a useful-looking benchmark dataset for drug discovery. Machine learning novelty and significance is limited.

POST-REBUTTAL UPDATE:
After clarifications, it seems apparent that benchmark papers like this could fall in the scope of ICLR. I still find the benchmark potentially useful and something I might use in my own research. I do not have any particular shortcomings in mind, but I think a stronger demonstration of the ability to generate new machine learning insights using the benchmark would have been useful. I will update my score to borderline positive (6) to reflect these thoughts.
review_title: null

Meta-review: This paper introduces a benchmark for experimental design algorithms
for an important cellular biological question, causal discovery of
effective genetic knock-out interventions. It uses existing datasets.

The paper was discussed by the reviewers after the authors correctly
pointed out that methodological machine learning novelty is not a
necessary condition for accepting papers. Two reviewers increased
their scores and all are slightly positive. The benchmark was seen as
valuable, and one reviewer even commented they might use it in their
own research. However, the paper is still on the borderline as this
benchmark is only a first step. It has not been shown yet that machine
learning insights can be produced with it, as the authors have not 
actually used it for benchmarking yet. In other words, the benchmark
can be considered a potentially excellent idea which has not been
tested empirically yet.

This seems a highly promising research direction and the authors are
strongly encouraged to continue to providing the benchmarks and
releasing the method to the community so that others can help them in that.
Decision: Accept (Poster)