GeneDisco: A Benchmark for Experimental Design in Drug Discovery
Authors: Arash Mehrjou
Ashkan Soleymani
Andrew Jesson
Pascal Notin
Yarin Gal
Stefan Bauer
Patrick Schwab
Venue: ICLR 2022 Poster (2022)
Keywords: batch active learning
drug discovery
benchmark

Abstract: In vitro cellular experimentation with genetic interventions, using for example CRISPR technologies, is an essential step in early-stage drug discovery and target validation that serves to assess initial hypotheses about causal associations between biological mechanisms and disease pathologies. With billions of potential hypotheses to test, the experimental design space for in vitro genetic experiments is extremely vast, and the available experimental capacity - even at the largest research institutions in the world - pales in relation to the size of this biological hypothesis space. Machine learning methods, such as active and reinforcement learning, could aid in optimally exploring the vast biological space by integrating prior knowledge from various information sources as well as extrapolating to yet unexplored areas of the experimental design space based on available data. However, there exist no standardised benchmarks and data sets for this challenging task and little research has been conducted in this area to date. Here, we introduce GeneDisco, a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery. GeneDisco contains a curated set of multiple publicly available experimental data sets as well as open-source implementations of state-of-the-art active learning policies for experimental design and exploration.

Published as a conference paper at ICLR 2022
GENEDISCO: A BENCHMARK FOR EXPERIMENTAL DESIGN IN DRUG DISCOVERY
Arash Mehrjou', Ashkan Soleymani', Andrew Jesson', Pascal Notin", Yarin Gal Stefan Bauer', Patrick Schwab I GlaxoSmmthhKiine, Artificial Intelligence & Machine Learning 2 MIT, Department of Computer Science, University of Oxford
ABSTRACT
In vitro cellular experimentation with genetic interventions, using for example CRISPR technologies, is an essential step in early-stage drug discovery and target validation that serves to assess initial hypotheses about causal associations be- tween biological mechanisms and disease pathologies. With billions of potential hypotheses to test, the experimental design space for in vitro genetic experiments is extremely vast, and the available experimental capacity even at the largest research institutions in the world pales in relation to the size of this biological hypothesis space. Machine learning methods, such as active and reinforcement learning, could aid in optimally exploring the vast biological space by integrating prior knowledge from various information sources as well as extrapolating to yet unexplored areas of the experimental design space based available data. How- ever, there exist no standardised benchmarks and data sets for this challenging task and little research has been conducted in this area to date. Here, we introduce GeneDisco, a benchmark suite for evaluating active learning algorithms for exper- imental design in drug discovery. GeneDisco contains curated of multiple publicly available experimental data sets as well as open-source implementations of state-of-the-an active learning policies for experimental design and exploration.
INTRODUCTION
The discovery and development of new therapeutics is one of the most challenging human endeav-
ours with success rates of around 5% (Hay et al. 2014 Wong et al. 2019). timelines that span on
average over a decade (Dickson & Gagnon 2009
2004), and monetary costs exceeding two billion Berdigaliyev & Aljofan 2020). The successful
United States (US) dollars (DiMasi et al. 2016
discovery of drugs at an accelerated pace is critical to satisfy current unmet medical needs (Rawl.
ins 2004 Ringel al. 2020), and, with thousands of potential treatments currently in development (informa PLC 2018), increasing the probability of success of new medicines by establishing causal links between drug targets and diseases (Nelson al. 2015) could introduce additional hundreds of new and potentially life-changing therapeutic options for patients every year.
However, given the current estimate of around 20 protein-coding genes (Pertea al. 2018).
continuum of potentially thousands of cell types and states under multitude of environmental con- ditions Trapnell 2015) MacLean et al. 2018 Worzfeld et al. 2017), and tens of thousands of cel- lular measurements that could be taken (Hasin et al. 201 7 Chappell et al., 2018). the combinatorial space of biological exploration spans hundreds of billions of potential experimental configurations, and vastly exceeds the experimental capacity of even the world's largest research institutes. Machine learning methods, such as active and reinforcement learning, could potentially aid in optimally ex- ploring the space of genetic interventions by prioritising experiments that more likely to yield mechanistic insights therapeutic relevance (Figure []. but, given the lack of openly accessible curated experimental benchmarks, there does not yet exist date concerted effort to leverage the machine learning community for advancing research in this important domain.
To bridge the gap between machine learning researchers versed in causal inference and the challeng- ing task of biological exploration, we introduce GeneDisco, an open benchmark suite for evaluating batch active learning algorithms for experimental design in drug discovery. GeneDisco consists of

---

Published as a conference paper at ICLR 2022
Counterfactual Predictions
Experimental Design
Build Counterfactual Estimators
Generate Hypotheses
3
Run Experiments
covariates O Experimental do(1 T) Ty-O' outcome. y
intervention
control
Experimental
Data
Figure I: In the setting considered in this work, counterfactual estimators of experimental out- comes (step 1, left) are used to propose experimental hypotheses (step 2. center) for validation in in vitro experiments with genetic interventions (step 3. right), such as CRISPR knockouts, in order to discover potential causal associations between biological entities that could be relevant for the development of novel therapeutics. The trained counterfactual estimators experimental search towards the space of biological interest, and thus more efficiently explore the vast space of genetic interventions. After every cycle, experimental data are generated that could lead to mechanistic scientific discoveries forming the basis for new therapeutics development, and guide subsequent experiment eycles with enhanced counterfactual estimators.
be used to direct the
several curated datasets, tasks and associated performance metrics, open implementations of state- of-the-art active learning algorithms for experimental design, and an accessible open-source code base for evaluating and comparing new batch active learning methods for biological discovery.
Concretely, the contributions presented in this work are as follows:
We introduce GeneDisco, an open benchmark suite for batch active learning for drug dis- covery that provides curated datasets, tasks, performance evaluation and open source im- plementations of state-of-the algorithms for experimental exploration.
We perform an extensive experimental baseline evaluation that establishes the relative per- formance of existing state-of- of-ie-of-hh-attt methods on all the developed benchmark settings using a total of more than 000 central processing unit (CPU) hours of compute time.
We survey and analyse the current state-of-the -ftt----tt---tttte of active learning for biological explo- ration the context of the generated experimental results, and present avenues of height- ened potential for future research based on the developed benchmark.
2 RELATED WORK
Background. Drug discovery is challenging endeavour with (i) historically low probabilities of successful development into clinical-stage therapeutics (Hay et al., 2014 Wong al.) 2019), and, for many decades until recently (Ringel et al.) 2020), (ii) declining industry productivity commonly referred to as "Eroom's law" (Scannell et al. 2012). Seminal studies by Nelson et al. (2015) and King et al. (2019) respectively first reported and later independently confirmed that the probability of clinical success of novel therapeutics increases up to three-fold fold if a medicine's molecular target is substantiated by high-confidence causal evidence from genome-wide association studies (GWAS) Visscher et al. (2017). With the advent of molecular technologies for genetic perturbation, such as Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR) (Jehuda al.) 2018). there now exist molecular tools for establishing causal evidence supporting the putative mechanism of a potential therapeutic hypothesis by means of in vitro experimentation beyond GWAS early on during the target identification and target validation stages of drug discovery (Rubin et al. 2019 Itokawa et al. 2016, Harrer et al. 2019 Vamathevan al. 2019). Among other applications (Chen et al. 2018 Ekins al. : Mak & Pichikal (219), machine learning methods, such as active and reinforcement learning, could potentially aid in discovering the molecular targets with the highest therapeutic potential faster.
Machine Learning for Drug Discovery. There exist number of studies that proposed, advanced or evaluated the use of machine learning algorithms for drug discovery: Costa et al. (2010) used deci- sion tree meta-classifier to identify genes associated with morbidity using protein-protein, metabolic, transcriptional, and sub-cellular interactions as input. Jeon et al. (2014) used support vector ma- chine (SVM) on protein expressions to predict target or non-target for breast pancreatic or ovariam cancers, and Ament et al. (0188) used least absolute shrinkage and selection operator (LASSO)
2

---

Published as a conference paper at ICLR 2022
regularised linear regression transcription factor binding sites and transeriptome profiling to pre- dict transcriptional changes associated with Huntington's 's disease. In the domain of human muscle ageing, |Mmoshhna et al.) (018) used a SVM on deep features extracted from gene expression sig- natures in tissue samples from young and old subjects to discover molecular targets with putative involvement in musele ageing. More recently, (Stokes et al. (2020) utilised deep learning discover Halicin as a repurposed molecule with antibiotic activity in mice.
Reinforcement and Active Learning. The use of deep reinforcement learning for de novo molec-
ular design has been extensively studied (Olivecrona et al., 2017 Popova et al., 20188) Putin et al.
2018 Lim et al., 2018: Blaschke et al.] 2020; Gottipati et al.) 2020; Horwood
Noutahi 2020)
Active learning for de novo molecular design has seen less attention (Diit et al. [016; Green
Green et al.
2020j, however, active learning for causal inference has seen increasing application toward causal-
effect estimation (Deng et al.] 2011, [Schwab et al.) 2018)) (Snndin et al. 20199 (Schwab et al. 2020
Bhattacharyya et al. 2020, Parbhoo al.] |001]021]] Qin et al. [021 Jesson et al. 2021] | Chau et al.)
2021), and causal graph discovery (Ke et al. 2019; Tong & Koller 2001 Murphy 2001 Hauser &
Bühlmann 2014| Ghassami et al. 2018) Ness et al. 201 A Agrawal et al. 2019 Lorch et al. 2021
Annadani et al. 2021] Scherrer et al. 2021).
Benchmarks. Benchmark datasets play an important role in developing machine learning method-
ologies. Examples include ImageNet (Deng et al. 2009) or MSCOCO (Lin al. (014) for com- puter vision, as well as cart-pole (Barto et al. 1983) or reinforcement learning (Ahmed et al. 2020 Validation of active learning for causal inference methods depends largely on synthetic data exper- iments due to the difficulty or impossibility of obtaining real world counterfactual outcomes. For causal-effect active learning, real world data with synthetic outcomes such as IHDP (Hill (011) or ACIC2016 6 Dorie et al. 2019)) are used. For active causal discovery, in silico data such as DREAM4 (Prill et al. 2011) or the gene regulatory networks proposed by Marbach et al. (2009) are used. Non synthetic data has been limited to protein signalling networks (Sachs et al. (2055) thus far. Several benchmarks focus on some aspects of drug discovery, for instance BBoonn et al. (2019) focusing on de novo molecular screening. However, to our knowledge, no other benchmark addressing the target identification part of the drug discovery pipeline, which is what our work focuses on.
In contrast to existing works, we develop open benchmark to evaluate the use of machine learning for efficient experimental exploration an iterative batch active learning setting. To the best of our knowledge, this is the first study (i) to introduce curated open benchmark for the challenging task of biological discovery, and (ii) to comprehensively survey and evaluate state-of-the of-the-art active learning algorithms in this setting.
3 METHODOLOGY
Problem Setting. We consider the setting in which we are given dataset consisting of covariates X TRP with input feature dimensionality p E N and treatment descriptors T E TR9 with treatment descriptor dimensionality y E N+ that indicate similarity between interventions. Our aim is to estimate the expectation of the conditional distribution of an unseen counterfactual outcome Y E R and intervention do(T t). D. = 7(X x, do(T' = t)) s E| X = xr, do = t)]. This setting corresponds to the Rubin-Neyman potential outcomes framework (Rubin) 2005) adapted to the context of genetic interventions with larger number of parametric interventions. In the context of a biological experiment with genetic interventions, yt is the experimental outcome relative a non-interventional control (e.g.. change in pro-inflammatory effect) that is measured upon perturbation of the cellular system with intervention I, x is a descriptor of the properties of the model system and/or cell donor (e.g., the immuno-phan potnnuotppp of the cell donor), and t is a descriptor of the genetic intervention (c.g., a CRISPR knockout on gene STATI) that indicates similarity to other genetic interventions that could potentially be applied. In general, certain ranges of Y/ may be preferable for further pursuit of an intervention T = t that inhibits a given molecular target- often, but not necessarily always, larger absolute values that move the experimental result more are of higher mechanistic interest. We note that the use of empty covariate vector X = Zy with p- = 0 is permissible if all experiments are performed in the same model system with the same donor properties. In in vitro experimentation, the set of all possible genetic interventions P = (t, ( is typically known a-priori and of finite size (c.g., knockout interventions all 20000 protein-coding genes).
given observed covariates X
3

---

Published as a conference paper at ICLR 2022
Batch Active Learning. In the considered setting, reading out additional values for yet unexplored interventions t requires a lab experiment and can be costly and time-consuming. Lab experiments are typically conducted in parallelised manner, for example by performing batch of multiple inter- ventions at once in an experimental plate. Our overall aim is to leverage the counterfactual estimator 9 trained on the available dataset to simulate future experiments with the aim of maximising an objective function & in the next iteration with as few interventions as possible. For the purpose of this benchmark, we consider the counterfactual mean squared error (MSE) of 9 in predicting experimentally observed true outcomes Ye from predicted outcomes ÿ, as the optimisation objec- tive CASE MyE Ue). Depending on context, other objective functions, such as for example maximising the number of discovered molecular targets with certain properties of interest (e.g.. druggability (Keller et al., (2066) may also be sensible in the context of biological exploration. At every time point, a new counterfactual estimator 9 is trained with the entire available experimental dataset, and used to propose the batch of interventions to be explored in the next iteration with the batch size b E N+. When using CMSE, this setting corresponds to batch active learning with the optimisation objective of maximally improving the counterfactual estimator g.
Acquisiiion Function. An acquisition function D a(g(t), Divan) takes the model and the set of all available interventions Dí in cycle k input and outputs the set of interventions D6 that are most informative after the kth experimental cycle with cycle index k E [K] = [0 K] where K E N+ is the maximum number of cycles that be run. Formally speaking, the acquisition function a P(Dawai) xg - ((Davait) takes a subset of all possible interventions that have not been tried so far (Dvvit), together with the trained model (g) derived from the cumulative data collected over the previous cycles, and outputs a subset of the available interventions De that are likely to be most useful under L to obtain
better estimate of EE| | X = x, do(T ()] with & being the space of the models which can be, e.g. the space of models and (hyper- )aramettes..
4 DATASETS, METRICS & BASELINES
The GeneDisco benchmark curates and standardizes two types of datasets: three standardized feature sets describing interventions í (inputs to counterfactual estimators; Section[4.T) I.. and four different in vitro genome-wide CRISPR experimental assays (predicted counterfactual outcomes; Section 4.2) each measuring& a specific outcome y/ following possible interventions T. We perform an extensive evaluation across these datasets, leveraging two different model types (Section 4.3) and nine differ- Since all curated assay datasets contained only outcomes for
ent acquisition functions (Section[4.4) only one model system, we used the empty covariate set X 2o for all evaluated benchmark con- figurations. The metrics used to evaluate the various experimental conditions (acquisition functions and model types) include model performance (Figure 2 and the ratio of discovered interesting hits (Figure 3) as function of number of samples queried.
4.1 TREATMENT DESCRIPTORS
The treatment descriptors T characterize a genetic intervention and generally should correspond to data sources that are informative to a genes' functional similarity - i.e. defining which genes if acted upon, would potentially respond similarly to perturbation. The treatment descriptor 7 is the input to the model described in Section for a fixed vector of covariates x. Any dataset considered for use as a treatment descriptor must be available for all potentially available interventions Ppool in the considered experimental setting. In GeneDisco, we provide three standardised gene descriptor sets for genetic interventions, and furthermore enable users to provide custom treatment descriptors via a standardised programming interface:
Achilles. The Achilles project generated dependency scores across cancer cell lines by assaying 808 cell lines covering a broad range of tissue types and cancer types (Dempster et al. 2019). The genetic intervention effects are based on interventional CRISPR screens performed across the included cell lines. When using the Achilles treatment descriptors, each genetic intervention is summarized using a gene representation T' with q 808 corresponding to the dependency scores measured in each cell line. In Achilles, after processing and normalisation (see Dempster et al. (2019), the final dependency scores provided are such that the median negative control (non-essential gene effect for each cell line is O, and the median positive control (essential) gene effect for each cell line is -1. The rationale for using treatment descriptors based on the Achilles dataset is that genetic
4

---

Published as a conference paper at ICLR 2022
effects measured across the various tissues and cancer types in the 808 cell line assays included in (Dempster et al. 2019) could serve as a similarity vector in functional space that may extrapolate to other biological contexts due to its breadth.
Search Tool for the Retrieval of Interacting Genes/Proteins (STRING) Network Embeddings. The STRING (Szklarezyk et al., 2021) database collates known and predicted protein-protein in- teractions (PPIs) for both physical as well as for functional associations. In order to derive a vec- representation suitable serve as a genetic intervention descriptor T, utilised the network embeddings of the PPIs contained in STRING as provided by Cho et al. (2016) 2013) with dimen- sionality q 799. PPI network embeddings could be an informative descriptor of functional gene similarity since proteins that functionally interact with the same network partners may serve similar biological functions (Vazquez et al. 2003 Saha et al. 2014).
Cancer Cell Line Encyclopedia (CCLE). The CCLE (Nusinow et al. 2020) project collected quan- titative proteomics data from thousands proteins by mass spectrometry across 375 diverse cancer cell lines. The generated protein quantification profiles with dimensionality similarity of genetic interventions since similar expression profiles across a broad range biological contexts may indicate functional similarity.
420 could indicate
Custom Treatment Descriptors. Additional, user-defined treatment descriptors can be evaluated in GeneDisco by implementing the standardised dataset interface provided within.
4.2 ASSAYS
As ground-truth interventional outcome datasets, we leverage various genome-wide CRISPR screens, primarily from the domain of immunology, that evaluated the causal effect intervening on large number of genes in cellular model systems in order to identify the genetic perturbations that induce a desired phenotype of interest. In connection with the problem formulation in Sec- tion 3 the CRISPR screens are the random variables whose realized values are the outcome of the interventional experiments. In the following, we add more details about four CRISPR screens.
4.2.1 REGULATION OF HUMAN T CELLS PROLIFERATION
Experimental setting. This assay is based on Shifrut et al. 2018 After isolating CD8+ cells from human donors, /Shifrut al. (2018) performed genome- wide loss-of-function screen iden- tify genes that impact the proliferation of T cells following stimulation with T cell receptors.
Measurement. The measured outcome is the proliferation of ' cells in response to T cell receptor stimulation. Cells were labeled before stimulation with CFSE (a fluorescent cell staining dye). Proliferation of cells is measured 4 days following stimulation by FACS sorting (a flow cytometry technique to sort cells based on their fluorescence).
Importance. Human T cells play central role in immunity and cancer immunotherapies. The identification of genes driving T cell proliferation could serve as the basis for new preclinical drug candidates or adoptive T cell therapies that help eliminate cancerous tumors.
4.2.2 INTERLEUKIN PRODUCTION IN PRIMARY HUMAN CELLS
Experimental setting. This dataset is based on a genome-wide CRISPR interference (CRISPRi) screen in primary human T cells to uncover the genes regulating the production of Interleukin-2
(IL-2). CRISPRi screens test for foss-of-function genes by reducing their expression levels. IL-2 cytokine produced by CD4+ T cells and is major driver of 'T cell expansion during adaptive immune responses. Assays were performed on primary cells from different donors. The detailed experimental protocol is deseribed in SShhmidt et al. (2021).
Measurement. Log fold change (high/low sorting bins) in IL-2 normalized read counts (averaged across the two biological replicates for robustness). Sorting was done via flow cytometry after intracellular cytokine staining.
Importance.
1I.2 is central to several immunotherapies against cancer and autoimmunity.
5

---

Published as a conference paper at ICLR 2022
4.2.3 INTERFERON-Y PRODUCTION IN PRIMARY HUMAN T CELLS
Experimental setting. This data also based on Schmidt et al. (2021), except that this experiment was performed understand genes driving production of Interferon-y (IFN-y). IFN-y is cytokine produced by CD4+ and CD8 T cells that induces additional T cells.
Measurement. Log fold change (high/low sorting bins) in IFN-y normalized read counts (aver- aged across the two biological replicates for robustness).
Importance. IFN-y is critical to cancerous tumor killing and resistance to IFN-y is one escape mechanism for malignant cells.
4.2.4 VULNERABILITY OF LEUKEMIA CELLS TO NK CELLS
Experimental setting. This genome-wide CRISPR screen was performed in the K562 cell line to identify genes regulating the sensitivity of leukemia cells to cytotoxic activity of primary human NK cells. Detailed protocol described in Zhuang al. (2019).
Measurement. Log fold counts of gRNAs in surviving K562 cells (after exposition to NK cells) compared control (no exposition NK cells). Gene scores normalized fold changes for all gRNAs targeting this gene.
Importance. Better understanding and control over the genes that drive the vulnerability of leukemia cells to NK cells will help improve anti-cancer treatment efficacy for lcukemia patients, for example by preventing relapse during hematopocitic stem cell transplantation.
4.3 MODELS
Parametric or non-parametric models can be used to model the conditional expected outcomes, E[Y X x; do(T 0)] Parametric models assume that the outcome Y has density J | (,w) conditioned the intervention t and the parameters of the model (we drop zo for compact- ness). A common assumption for continuous outcomes is a Gaussian distribution with density fly f,w) = Wvy @(t:w), w), o²). which assumes that y is deterministic function of (t:ww) with ad- ditive Gaussian noise scaled by o²_ Bayesian methods treat the model parameters w as instances of the random variable Q E W and aim model the posterior density of the parameters given the data, [(w D). For high-dimensional, large-sample data, such as we explore here, a variational approxi- mation the posterior is often used, q(w (D) (MacKay 1992 Hinton & Van Camp, 1993 Barber & Bishop 1998 Gal & Ghahraman] 2016). In this work we use Bayesian Neural Networks (BNNs) to approximate the posterior over model parameters. A BNN gives (()
Ej= g(t; 1
where ((;:w/-1) is a unique functional estimator of EY X x, do = t)] induced by w g(w D:="): sample from the approximate posterior parameters given the mulative data at acquisition step k --1 1. We also use parameamerriic, may-Bsan,ian, Random Forest Regression (Breiman 2001). A Random Forest gives gi-1(()) = Eソ= 0 3 (t), where o)-((()) (t) is unique functional estimator of B|Y X x, do t)] indexed by the jth sample in the ensemble of m trees trained on D
We emphasize two main assumptions behind the above modeling choices for the problem formu- lation of Section 3 (1) A causal link between from the gene intervention to the phenotype screen exists. (2) Gaussian noise is reasonable choice for the output noise the interventional outcome continuous variable. We leave it to future work to explore heterogeneity in output noise and other likelihood functions.
In the following, we will define our acquisition functions terms of parametric models, but the definitions are easily adapted for non-parametric models well.
4.4 ACQUISITION FUNCTIONS
We included diverse set of the existing active learning methods to informative set of baselines for comparison and intuition activation for future improvements. Most of the Different acquisition function methods can be categorized in the following way:

---

Published as a conference paper at ICLR 2022
Random acquisition function is an integral part of whatever baseline set in active learning's performance criteria.
Random:
Uncertainty-bused acquisition functions: These categories are popular because of their casy-to-develop and low-computational properties while having great performance. Uncertainty- functions usually are employed in the shallow models that some sense of intrinsic uncertainty exists for them, e.g., random forests. Here, we adapted some of them to our notion of uncertainty (based on our regression task). These functions include topuncert arpuncertain ain and Margin Sample.
Deep Bayesian Active Learning acquisition functions: One way to bridge between classical uncertainty-bart algorithms and deep active learning methods is to leverage deep bayesian active learning. Bayesian Active Learning by Disagreement (BALD) calculates mutual information between samples and model parameters as a definition for uncertainty.
Density-basee acquisition functions: These functions try to select a sample that is the best representative of the whole dataset in terms of diversity. Based on different repesentativee power measures various methods exist that here Coreset and kmeansda are covered.
Adversarial learning attacks: By choosing adversarial samples as nominated samples for active learning, one may most of the adversarial learning algorithms acquisition functions. We implemented AdyBIM as a compelling example of this category.
Hybrid acquisition functions: These methods take into account both sample diversity and also prediction uncertainty of the model. In other words, they increase the diversity of data samples according to model properties. BADGE considers the gradient of the loss function w.r.t weights of the final layer while kmeansember looks at the data from their embeddings in the last layer of the deep learning model.
Each one of these acquisition functions (Random, BADGE, BALD (topuncertain and softuncertain). Coreset, Margin Sample (Margin), Adversarial Basic Iteractive Method (AdvBIM), h--mans Sampling (kmeansdata and kmeansembed)) included in the benchmark described in detail in Appendix B
EXPERIMENTAL EVALUATION
Setup. In order to assess current state-of-the-an methods on the GeneDisco benchmark, we per- form an extensive baseline evaluation of 9 acquisition functions, 6 acquisition batch sizes and 4 experimental assays using in excess of 200 CPU hours of compute time. Due the space limit, we include the results for 3 batch sizes in the main text and present the results for all batch sizes in the appendix. The employed counterfactual estimator 9 is a multi-layer perceptron (MLP) that has one hidden layer with ReLU activation and a linear output layer. The size of the hidden layer determined at each active learning cycle by k-fold cross validation against 20% of the acquired batch. At each cycle, the model is trained for at most 100 epochs but early stopping may interrupt training earlier if the validation error does not decrease. Each experiment is repeated with 5 random the
seeds assess experimental variance. To choose the number of active learning cycles, following strategy: the number of cycles are bounded to 40 for the acquisition batches of sizes 16, 32 and 64 due to the computational limits. For larger batch sizes, the number of cycles are reduced proportionally so that the same total number of data points are acquired throughout the cycles. At each cycle, the model is trained from scratch using the data collected up to that cycle, i.e. a trained model is not transferred to the future cycles. The test data is a random 20% subset of the whole data that is left aside before the active learning process initiates, and kept fixed across all experimental settings (i.e., for different datasets and different batch sizes) to enable a consistent comparison of the various acquisition functions, counterfactual estimator and treatment descriptor configurations.
Results.
The model performance based on the STRING treatment descriptors for different acquisi- tion functions, acquisition batch sizes and datasets are presented Figure 2 The same metries in
the Achilles and CCLE treatment descriptors are provided in Appendix C To showcase the effect
of the model class, we additionally repeat the experiments using a random forest model as an un- certainty aware ensemble model with reduced set of acquisition functions that are compatible with non-differentiable models (Appendix CC. The random forest model is also used in another set of experiments with other datasets (Sanchez et al., 2021 Zhu et al. 202T) which are fully postponed to
7

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt et al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
PPg)
2-75
batch size 16
2411 (Prag
batch size = -64
1.20
P4
batch size = 256
602
2a0ppe 30
The evaluation of the model trained with STRING treatment descriptors at each active learning cycle for 4 datasets and 3 acquisition batch sizes. In each plot, the x-axis is the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
Figure 2:
Shifrut al. 2018
Schmidt et al. 2021 (IFNg) Schmidt al. 2021 (1L-2)
Zhuang et al. 2019
201.
batch size
batch size 64
51)
030
batch size 256
Saden
501
100
300
Figure 3: The hit ratio of different acquisition for BNN model, different target datasets, and different acquisition batch sizes. We use STRING treatment descriptors here. The x-axis shows the number of data points collected far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
8

---

Published as a conference paper at ICLR 2022
the appendix due to the space constraint. To investigate the types of genes chosen by different acqui- sition functions, we defined a subset of potentially interesting genes as the top 5% with the largest absolute target value. These are the genes that could potentially be of therapeutic value due to their outsized causal influence on the phenotype of interest. The hit ratio out of the set of interesting genes chosen by different acquisition functions are presented in Figure descriptors, and in Appendix for Achilles and CCLE. Benchmark results of interest include that model-indeeenndet acquisition methods using diversity heuristics (random, kmeansdata) perform relatively better in terms of model improvement than acquisition functions based on model uncer- tainty (e.g., topuncertain, softuncertain) when using lower batch acquisition sizes than in regimes with larger batch acquisition sizes potentially due to diversity being inherently higher in larger batch acquisition regimes due to the larger set of included interventions in an intervention space with a limited amount of similar interventions. Notably, while diversity-focused, model-independent ac- quisition functions, such as random and kmeansdata, perform well in terms of model performance, they underperform in terms of interesting hits discovered as a function of acquired interventional samples (Figure 3). Based on these results, there appears to be a trade-off between model improve- ment and hit discovery in experimental exploration with counterfactual estimators that may warrant research into approaches to manage this trade-off to maximize long-term discovery rates.
the STRING treatment
DISCUSSION AND CONCLUSION
The ranking of acquisition functions in GeneDisco depends on several confounding factors, such as the choice of evaluation metric to compare different approaches. the characteristics of the dataset of interest, and the choice of the model class and its hyperparameters. An extrapolation of results ob- tained in GeneDisco to new settings may not be possible under significantly different experimental conditions. There is subtle interplay between the predictive strength of the model and the acqui- sition function used to select the next set of interventions, as certain acquisition functions are more sensitive the ability of the model to estimate its own epistemic uncertainty. An important message learned from Figures 2 and is the observation that in active learning. the best acquisition function is the one that gives more accurate model with the same budget of experiments. However, in drug discovery. learning an accurate predictive model has secondary importance. The primary objective is to find the optimal set of targets for potential medicine. This objective may be unaligned even in the opposite direction of training the predictive model. For example, even though the Kmeans acquisition function that chooses the centroids in the data domain has a decent performance in the active learning task (finding the most informative labeled dataset for training the predictive model), does not perform well in choosing the most interesting targets (those that moves the phenotype maximally.) A potential reason could be the intuition that the interesting targets may be close to each other in terms of the information content they provide to the predictive model. The acquisition function then considers them as redundant and does not choose in the next iteration. Hence, AL algorithms must be used with care in drug discovery. We postpone further study of a more suitable objective function for the acquisition function to future work.
From a practical standpoint, GeneDisco assumes the availability of a labeled set that is sufficiently representative to train and validate the different models required by the successive active learning cycles. However, model validation might be challenging when this set is small (c.g., during the early active learning cycles) when the labelling process is noisy. Label noise is common in in- terventional biological experiments, such as the ones considered in GeneDisco. Experimental noise introduces additional trade-offs for consideration not considered in GeneDisco, such as choosing the optimal budget allocation between performing experiment replicates (technical and biological) to mitigate label noise or collecting more data points via additional active learning cycles.
GeneDisco addresses the current lack of standardised benchmarks for developing batch active learn- ing methods for experimental design in drug discovery. GeneDisco consists of several curated datasets for experimental outcomes and genetic interventions, provides open source implementa- tions of state-of-the-ee assessment of these methods across a wide range of hyperparameter settings. We aim to attract the broader active learning community with an interest in causal inference by providing a robust and user-friendly benchmark that diversifies the benchmark repertoire over standard vision datasets. New models and acquisition functions for batch active learning in experimental design are of criti- cal importance to realise the potential of machine learning for improving drug discovery. As future research, we aim to expand GeneDisco to enable multi-modal learning and support simultancous optimization across multiple output phenotypes of interest.
acquisition functions for batch active learning, and includes a thorough
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
PS. AM and SB are employees of GlaxoSmithKline (GSK) and PS is also a shareholder of GSK. PN is supported by GSK and the UK Engineering and Physical Sciences Research Council (EPSRC ICASE award no. 18000077).
REPRODUCIBILITY STATEMENT
This work introduces a new curated and standardized benchmark, GeneDisco, for batch active learn- ing for drug discovery. The benchmark includes four publicly available datasets, which have previ- ously been published in peer review process. Using a total of more than 20,000 central processing unit (CPU) hours compute time, we perform an extensive evaluation of state-of-the- thettht-rrrttt acquisition functions for batch active learning on the GeneDisco benchmark, across wide range of hyperpa- rameters. To the best of our knowledge, this is the first comprehensive survey and evaluation of active learning algorithms on real-world interventional genetic experiment data. Similar to develop- ments in other fields e.g. for the learning of disentangled representations (Locatello et al. 2019) or generative adversarial networks (Lucic et al. 2017), we hope that our large scale experiments across a diverse set of real-world datasets provide an evidence basis to better understand the settings in which different active learning approaches work do not work for drug discovery applications.
All used models and acquisition functions are described in detail and referenced in Section 4.4 and For all introduced datasets, include detailed description and the details on the train, test and validation splits at the beginning of Section 5
Section 3
For all experimental results report the range of syyper-parameen considered and the methods of selecting hyper-pae parameters as well as the exact number of training and evaluation runs (Appendix (
We additionally provide error bars over multiple random seeds and the code was executed on cloud cluster with Intel CPUs. We provide detailed results for all investigated settings in the appendix (C
REFERENCES
Raj Agrawal, Chandler Squires, Karren Yang, Karthikeyan Shanmugam, and Caroline Uhler. Abed- strategy: Budgeted experimental design for targeted causal structure discovery. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3400-3409, PMLR, 2019.
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Yoshua Bengio, Bernhard Schölkopf, Manuel Wüthrich, and Stefan Bauer. Causalworld: A robotic manipulation bench- mark for causal structure and transfer learning. arXiv preprint arXivv2010.042266 2020.
Seth A Ament, Jocelynn R Pearl, Jeffrey P Cantle, Robert M Bragg, Peter J Skene, Sydney R Coffey, Dani E Bergey, Vanessa C Wheeler, Marcy MacDonald, Nitin S Baliga, et al. Transcriptional regulatory networks underlying gene expression changes in huntington's 's disease. Molecular Sys- tems Biology, 1443), 2018
Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Ben- gio, and Stefan Bauer. Variational causal networks: Approximate bayesian inference over causal structures. arXiv preprini arXiv:2106.07635 2021.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint 1r0.v:1906.03671, 2019.
David Barber and Christopher M Bishop. Ensemble learning in bayesian neural networks. Nato ASI Series Computer and Systems Sciences, 168:215-238, 1998.
Andrew Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics. (5):834-846, 1983.
Nurken Berdigaliyev and Mohamad Aljofan. An overview of drug discovery and development. Future Medicinal Chemistry, 12(10):939-94 2020.
10

---

Published as a conference paper at ICLR 2022
Arnab Bhattacharyya, Sutanu Gayen, Saravanan Kandasamy, Ashwin Maran, and Vinodchandran N Variyam. Learning and sampling of atomic interventions from observations. In International Conference on Machine Learning, pp. 842-853, PMLR, 2020.
Thomas Blaschke, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Memory-assiste rein- forcement learning for diverse molecular de novo design. Journal of Cheminformatics, 12(1): 1-17, 2020.
Leo Breiman. Random forests. Machine Learning, 45(1):5 2001.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3): 1096-1108, 2019.
Lia Chappell, Andrew JC Russell, and Thierry Voet. Single-cell (multi) omics technologies. Annual Review of Genomics and Human Genetics, 19:15-41, 2018.
Siu Lun Chau, Jean-Frrançois Ton, Javier González, Yee Whye Teh, and Dino Sejdinovic. Bayesimp: Uncertainty quantification for causal data fusion. arXiv preprint arXiv:2106.03477 2021.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning drug discovery. Drug Discovery Today. 23(6):1241-122 1250, 2018.
Hyunghoon Cho, Bonnie Berger, and Jian Peng. Diffusion component analysis: unraveling func- tional topology in biological networks. In International Conference Research in Computa- tional Molecular Biology, pp. 62-64. Springer, 2015.
Hyunghoon Cho, Bonnie Berger, and Jian Peng. Compact integration of multi-network topology for functional analysis of genes. Cell Systems, 3(6):540-544, 2016.
Pedro R Costa, Marcio Acencio. and Ney Lemke. A machine learning approach for genome-wide
prediction of morbid and druggable human genes based on systems-level data. In BMC Genomics, volume 11, pp. 1-15. Springer, 2010.
Joshua M. Dempster, Jordan Rossen, Mariya Kazachkova, Joshua Pan, Guillaume Kugener, David E. Root, and Aviad Tsherniak. Extracting biological insights from the project achilles genome-
scale crispr screens in cancer cell lines. bioRxiv, 2019. doi: 10.1101/720243. URL https:
org/ www.tenttetttt/eeeeeeeee 200.//2019//00
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Icee, 2009.
Kun Deng, Joelle Pineau, and Susan Murphy. Active learning for personalizing treatment. In 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp- 32-39. IEEE, 2011.
Michael Dickson and Jean Paul Gagnon. Key factors in the rising cost of new drug discovery and development. Nature Reviews Drug Discovery, 3(5):417-44) 429, 2004.
Michael Dickson and Jean Paul Gagnon. The cost of new drug discovery and development. Discov- ery Medicine, 4(22):172-177) 2009.
Joseph A DiMasi, Henry G Grabowski, and Ronald W Hansen. Innovation in the pharmaceutical industry: new estimates of r&d costs. Journal of Health Economics, 47:20-33, 2016.
Atray Dixit, Oren Parnas, Biyu Li, Jenny Chen, Charles P Fulco. Livnat Jerby-Amon, Nemanja D Marjanovic, Danielle Dionne, Tyler Burks, Raktima Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell ma profiling of pooled genetic screens. Cell, 167(7): 1853-1866, 2016.
Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus do-it- yourself methods for causal inference: Lessons learned from a data analysis competition. Statis- fical Science, 34(1):43-66 2019.
11

---

Published as a conference paper at ICLR 2022
Sean Ekins, Ana C Puhl, Kimberley M Zorn, Thomas R Lane, Daniel P Russo, Jennifer J Klein, Anthony Hickey, and Alex M Clark. Exploiting machine learning for end-to-end drug discovery and development. Nature Materials, 18(5):435-4444 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference Machine Learning, pp. 1050-1059,
2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192, PMLR, 2017.
AmirEmad Ghassami, Saber Salehkaleybar, Negar Kiyavash, and Elias Bareinboim. Budgeted periment design for causal structure learning. In International Conference on Machine Learning, pp. 1724-1733. PMLR, 2018.
Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Simon Blackburn, Karam Thomas, Connor Coley, Jian Tang, et al. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International Conference on Machine Learning, pp. 3668-3679, PMLR, 2020.
Darren VS Green, Stephen Pickett, Chris Luscombe, Stefan Senger, David Marcus, Jamel Mes- lamani, David Brett, Adam Powell, and Jonathan Masson. Bradshaw: a system for automated molecular design. Journal of Computer-aided Molecular Design, 34(7):747-765, 2020.
Stefan Harrer, Pratik Shah, Bhavna Antony, and Jianying Hu, Artificial intelligence for clinical trial design. Trends in Pharmacological Sciences, 40(8):577-591, 2019.
Yehudit Hasin, Marcus Seldin, and Aldons Lusis. Multi-omics approaches to disease. Genome Biology, 18(1):1-15, 2017.
Alain Hauser and Peter Bühlmann. Two optimal strategies for active learning of causal models from interventional data. International Journal of Approximate Reasoning, 55(4):926-95) 99096,9999-99 2014.
Michael Hay, David W Thomas, John L Craighead, Celia Economides, and Jesse Rosenthal. Clinical development success rates for investigational drugs. Nature Biotechnology, 32(1):40-51, 2014.
Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the de- scription length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, pp. 5-13, 1993.
Julien Horwood and Emmanuel Noutahi. Molecular design in synthetically accessible chemical space via deep reinforcement learning. ACS omega, 5(51):32284 (155)33288-32299, 2020.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprini arXiv: 2011.
informa PLC. Pharma R&D Annual Review 2018. 2018. Accessed: 2021-10-03.
Kentaro Itokawa, Osamu Komagata, Shinji Kasai, Kohei Ogawa, and Takashi Tomita. Testing the causality between cyp9m 10 and pyrethroid resistance using the talen and crispr/cas9 technologies. Scientific Reports, 6(1):1-10. 2016.
Ronen Ben Jehuda, Yuval Shemer, and Ofer Binah. Genome editing induced pluripotent stem cells using crispr/cas9. Stem Cell Reviews and Reports, 14(3):323-333 2018.
Jouhyun Jeon, Satra Nim. Joan Teyra, Alessandro Datti, Jeffrey L Wrana, Sachdev S Sidhu, Jason Moffat, and Philip M Kim. A systematic approach to identify novel cancer drug targets using machine learning, inhibitor design and high-th -ighhhhrouhhput screening. Genome Medicine, 6(7):1-18, 2014.
12

---

Published as a conference paper at ICLR 2022
Andrew Jesson, Panagiotis Tigas, Joost van Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal. Causal-bald: Deep bayesian active learning of outcomes to infer treatment-effect from observational data. Advances in Neural Information Processing Systems, 34, 2021.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv: 191000105 2019.
Thomas H Keller, Arkadius Pichota, and Zheng Yin. A practical view of 'druggability'. Current Opinion in Chemical Biology, 10(4)3357-361, 2006.
Emily King. Wade Davis, and Jacob F Degner. Are drug targets with genetic support twice as likely to be approved? revised estimates of the impact of genetic support for drug mechanisms on the probability of drug approval. PLoS Genetics, (5(12);e 1008489, 2019.
Andreas Kirsch, Sebastian Farquhar, and Yarin Gal. simple baseline for batch active learning with stochastic acquisition functions, 2021.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
Jaechang Lim, Seongok Ryu, Jin Woo Kim, and Woo Youn Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of Cheminformatiis, 10(1):1-9, 2018.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114 PMLR, 2019.
Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. Dibs: Diffcrentiable bayesian structure learning. arXiv preprint arXiv:2/05.//883 2021.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. arXiv preprint arXiv:1711.10337, 2017.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural Compu- lation, 4(3):448-477 1992.
Adam L. MacLcan, Tian Hong, and Qing Nic. Exploring intermediate cell states through the lens of single cells. Current Opinion in Systems Biology, 9:32-41, 2018. ISSN 2452-3100. doi:
https://di..or//10 http://doo.og/11000011///ccooooo (t016//ccosh.2018.0..000 URL https: www. www.sceeeeecc.... rect. com/ Mathematic Modelling.
Kit-Kay Mak and Mallikarjuna Rao Pichika. Artificial intelligence in drug development: present status and future prospects. Drug Discovery Today, 24(3):773-780, 2019.
Polina Mamoshina, Marina Volosnikova, Ivan V Ozerov, Evgeny Putin, Ekaterina Skibina, Franco Cortese, and Alex Zhavoronkov. Machine learning on human muscle transcriptomic data for biomarker discovery and tissue-specific drug target identification. Frontiers in Genetics, 9:242, 2018.
Daniel Marbach, Thomas Schaffter, Claudio Mattiussi, and Dario Floreano. Generating realistic in silico gene networks for performance assessment of reverse engineering methods. Journal of Computational Biology, 16(2):229 22099,992299200000 2009.
Kevin P Murphy. Active learning of causal bayes net structure. 2001.
Matthew R Nelson, Hannah Tipney, Jeffery L Painter, Judong Shen, Paola Nicoletti, Yufeng Shen, Aris Floratos, Pak Chung Sham, Mulin Jun Li, Junwen Wang, et al. The support of human genetic evidence for approved drug indications. Nature Genetics, 478))8566866)) 2015.
13

---

Published as a conference paper at ICLR 2022
Robert Osazuwa Ness, Karen Sachs, Parag Mallick, and Olga Vitek. A bayesian active learning experimental design for inferring signaling networks. In International Conference on Research in Computational Molecular Biology, pp. 134-156. Springer, 2017.
David P Nusinow, John Szpyt, Mahmoud Ghandi, Christopher M Rose, E Robert McDonald III, Marian Kalocsay, Judit Valé-naa Ellen Gelfand, Devin K Schweppe, Mark Jedrychowski, et al. Quantitative proteomics of the cancer cell line encyclopedia. Cell, 180(2):387-402 2020.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 91): 2017.
Sonali Parbhoo, Stefan Bauer, and Patrick Schwab. NCoRE: Neural Counterfactual Representation Learning for Combinations of Treatments. arXiv preprint vrXiv:2103.11773
2021. 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2888 2011.
Mihaela Pertea, Alaina Shumate, Geo Pertea, Ales Varabyou, Florian P Breitwieser, Yu-Chi Chang, Anil K Madugundu, Akhilesh Pandey, and Steven L Salzberg. Chess: a new human gene catalog curated from thousands of large-scale ma sequencing experiments reveals extensive transcrip- tional noise. Genome Biology, 19(1):1-14, 2018.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science Advances, 4(7))eaap7885, 2018.
Robert Prill, Julio Saez-Rodriguez, Leonidas Alexopoulos, Peter Sorger, and Gustavo Stolovitzky. Crowdsourcing network inference: The dream predictive signaling network challenge. Science Signaling, 4:mr7, 09 2011. doi: 10. 10.11266siisgnnal22002222
Evgeny Putin, Arip Asadulaev, Yan Ivanenkov, Vladimir Aladinskiy, Benjamin Lenghez.,engeling, Alán Aspuru-Guzik, and Alex Zhavoronkov. Reinforced adversarial neural computer for de novo molecular design. Journal of Chemical Information and Modeling, 58(6): 194-1204, 2018.
Tian Qin, Tian-Zuo Wang, and Zhi-Hua Zhou. Budgeted heterogeneous treatment effect estimation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8693-8702. URL attps: : https::/prrceeedingg mlr. www.pppes..vilv http///33////// /1tps///139//n221b...
PMLR. 18-24 Jul 2021.
html
Michael D Rawlins. Cutting the cost of drug development? Nature Reviews Drug Discovery, 3(4): 360-364, 2004.
Michael S Ringel, Jack W Scannell, Mathias Baedeker, and Ulrik Schulze. Breaking eroom's law. Nature Reviews Drug Discovery, 2020.
Dan Roth and Kevin Small. Masein-based active learning for structured output spaces. In European Conference on Machine Learning, pp.
Springer, 2006.
Adam Rubin, Kevin R Parker, Ansuman Satpathy, Yanyan Qi, Beijing Wu, Alvin J Ong, Maxwell R Mumbach, Andrew L Ji, Daniel S Kim, Seung Woo Cho, al. Coupled single- cell crispr screening and epigenomic profiling reveals causal gene regulatory networks. Cell, 176 (1-2):361-376, 2019.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331, 2005.
Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721): 523-529, 2005.
14

---

Published as a conference paper at ICLR 2022
Sovan Saha, Piyali Chatterjee, Subhadip Basu, Mahantapas Kundu, and Mita Nasipuri. Fumpred-
1: Protein function prediction from a protein interaction network using neighborhood analysis. Cellular and Molecular Biology Letters, 19(4):65-669) 2014.
Carlos G Sanchez, Christopher M Acker, Audrey Gray, Malini Varadarajan, Cheng Song, Nadire R Cochran, Steven Paula, Alicia Lindeman, Shaojian An, Gregory McAllister, et al. Genome wide crispr screen identifies protein pathways modulating protein levels in neurons. Communica- lions biology, 4( 4(1):1-14, 2021.
Jack W Scannell, Alex Blanckley, Helen Boldon, and Brian Warrington. Diagnosing the decline in pharmaceutical r&d efficiency. Nature Reviews Drug Discovery, 11(3):191-200, 2012.
Nino Scherrer, Olexa Bilaniuk, Yashas Annadani, Anirudh Goyal, Patrick Schwab, Bernhard Schölkopf, Michael Mozer, Yoshua Bengio, Stefan Bauer, and Nan Rosemary Ke. Learning neural causal models with active interventions. arXiv preprint
(tfi::20090022999 2021.
Ralf Schmidt, Zachary Steinhart, Madeline Layeghi, Jacob W. Freimer, Vinh Q. Nguyen, Franziska
Blaeschke, and Alexander Marson. Crispr activation and interference screens in primary human t
cells decode cytokine regulation. bioRxiv, 2021. doi: 10.1101/202100551144470000 URL https:
content
Patrick Schwab, Lorenz Linhardt, and Walter Karlen. Perfect Match: A Simple Method for
Learning Representations For Counterfactual Inference With Neural Networks. arXiv preprint arXiv:1810.00656. 2018.
Patrick Schwab, Lorenz Linhardt, Stefan Bauer, Joachim M Buhmann, and Walter Karlen. Learn- ing Counterfactual Representations for Estimating Individual Ressones Curves. In AAAI Conference on Artificial Intelligence, 2020.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 1708.0047700 2017.
Eric Shifrut, Julia Carnevale, Victoria Tobin, Theodore L. Roth, Jonathan M. Woo, Christina T. Bui, P. Jonathan Li, Morgan E. Diolaiti, Alan Ashworth, and Alexander Marson. Genome- crispr screens in primary human t cells reveal key regulators of immune function. Cell, 175(7):1958- 1971.e15, 2018. ISSN 0092-8674. doi: http://cc....................te.
htRpttss:
www..scien.cdirecc... http:///ceeettccccccccccceeee/re
Jonathan M Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar A deep learning approach to antibiotic discovery. Cell, 180(4))688-002 2020.
et al.
liris Sundin, Peter Schulam, Eero Siivola, Aki Vehtari, Suchi Saria, and Samuel Kaski. Active learning for decision-making from imbalanced observational data. In International Conference on Machine Learning, pp. 6046-6055, PMLR, 2019.
Damian Szklarczyk, Annika L Gable, Katerina C Nastou, David Lyon, Rebecca Kirsch, Sampo Pyysalo, Nadezhda T Doncheva, Mare Legeay, Tao Fang, Peer Bork, et al. The string database in 2021: customizable protein protein-protein networks, and functional characterization of user-uploaded gene/measurement sets. Nucleic Acids Research, (9(D1):)005-DD)
2021.
Simon Tong and Daphne Koller. Active learning for structure in bayesian networks. In International Joint Conference on Artificial Intelligence, volume 17, pp. 863-869. Citeseer, 2001.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint 2017.
Cole Trapnell. Defining cell types and states with single-cell genomics. Genome Research, 25(10): 1491-1498, 2015.
Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learn- ing in drug discovery and development. Nature Reviews Drug Discovery, (866):433-477,
2019. 2019.
15

---

Published as a conference paper at ICLR 2022
Alexei Vazquez, Alessandro Flammini, Amos Maritan, and Alessandro Vespignani. Global protein function prediction from protein-protent interaction networks. Nature biotechnology. 21(6):697- 700, 2003.
Peter M Visscher, Naomi Wray, Qian Zhang, Pamela Sklar, Mark I McCarthy, Matthew A Brown, and Jian Yang. 10 years of GWAS discovery: biology, function, and translation. The American Journal of Human Genetics, 101(1):5-25 -2, 2017.
Chi Heem Wong. Kien Wei Siah, and Andrew W Lo. Estimation of clinical trial success rates and related parameters. Biostatistics, 20(2):273-266, 2019.
Thomas Worzfeld, Elke Pogge Strandmann, Magdalena Huber, Till Adhikary, Uwe Wagner, Silke Reinartz, and Rolf Müller. The unique molecular and cellular microenvironment of ovarian cancer. Frontiers in Oncology, 7:24, 2017.
Yunkai Zhu, Fei Feng, Gaowei Hu, Yuyan Wang, Yin Yu, Yuanfei Zhu, Wei Xu, Xia Cai, Zhiping Sun, Wendong Han, et al. A gideme-wide crispr screen identifies host factors that regulate sars- cov-2 entry. Nature communications, 12(1):1-11, 2021.
Xiaoxuan Zhuang, Daniel Veltri, and Eric O. Long. Genome-wide crispr screen reveals cancer cell resistance to nk cells induced by nk-derived ifn-y. Frontiers in Immunology, 10:2879, 2019. ISSN 1664-3224, doi: 10.3389/fmmm..c
URL
www.friiniirsin.
org/article/1o. / fi 1mmu. 2019. 02879
16

---

Published as a conference paper at ICLR 2022
A NOTATIONS
Here the list of notations used in this manuscript.
Pooot: The pool of unlabeled data.
D Day Acquired data at k-th AL cycle.
D D Cumulative training data after k-th cycle of AL.
vat Validation data.
Dies: Held-out test data.
b: Acquisition batch size.
K: Total number of AL eycles.
k 11, ..... K]: Index of the AL cycle.
[K] = [1,2 [1,2....,K] K].
e E E: Inherent noise (aleatoric uncertainty).
T T: Treatment variable.
E XX æ, do(TT t]]: The conditional expected outcomes.
@(::w): The model parameterised by Ω to estimate E[(Y == xx 1)]-
X: random variable X with distribution X F(X) and density f(x).
B ACQUISITION FUNCTIONS CONT.
Random. As baseline we look at random acquisition. Random acquisition at cycle & can be seen as uniformly sampling data from D
Roai
Random ( Divaii) (f1,...tof
tsj navail
Here, the acquisition function samples b elements without replacement from the set of Ravail ele- ments. The set element (t,) on the left of the semicolon, and the probability of the element being is the right of the semicolon. This convention will be used again below.
acquired Anal
BADGE. BADGE looks maximize the diversity of acquired samples, but, in contrast to Coreset, additionally takes the uncertainty of the prediction into account (Ash et al.) 2019). If the true label y were observed, BADGE would proceed by maximizing the diversity of samples based on the gradient of the loss function I with respect to the weights the final layer of the most recently trained model - - (y, (t; Intuitively, it asks how much would parameters change we observed the labeled outcome for this example? However, the true label is not yet observed. Ash al. (2019) explore BADGE in the classification setting. For
class problem, where f(y E, w) = Bernoulliy (t;:w)), they propose using the class with the highest predicted probability, y = argmax,eo,, f t, f,w), to approximate the gradient as T " (t): This does not directly translate to the regression setting, under our modelling assumptions the with the highest predicted likelihood corresponds exactly to (t: w) which would lead to a loss of zero, and gradients of zero. As starting point, we instead take ÿ as random sample from f(y t,w) = Ni g(t:w), o²). We then use the same k-mmans++ algorithm as Ash et al. (201)) to approximate:
GBADGE ( (t), D (
A al(0, 1)) ((
(2)
argmin argmax argmin [f te aad D mad t oun UD UDD
dūk-1
where is again the Euclidean distance.
Bayesian Active Learning by Disagreement (BALD). Given uncertainty model, such as
a BNN or Random Forest we can now take an information theoretic approach to selecting interven- tions from the pool data. Houlshy et al. (001)) frame active learning as looking maximize the
17

---

Published as a conference paper at ICLR 2022
information gain about the model parameters if observe the outcome Y given model inputs. Formally, the information gain is given by the mutual information between the random variables Y and ! given the intervention and acquired training data Dum up until acquisition step k:
I(; Demml) HY (,mmm' HY 2rm
(3)
HY 1, (, ) HY w.,)..
Under the assumed model we have
0² sww (Dtmm [b(t:)) Dim [(t:w))
I(; c, (
2 log
(4)
o²
which leads to the following estimator setting a² =1
i( 0 1.001.2000 Dim')) = log
Σ
-
m1 _
(5)
We look at two acquisition functions for BALD. First, consider the naive batch acquisition CBBALD proposed by Gal et al. (217)) which acquires the the top b examples from D
aBALD g (t), Divat)
argmax (t....))))
Eix: t,, Doum').
(6)
This method will be referred to as topuncertain in the plots later. And second, we consider aSoBBALD which randomly samples b interventions from D weighted by a tempered softmax function (Kirsch al. 2:2T):
I(Y; 44, D
dSSotBBLD (f), D = (o) ~ 4::
exp
Temp
I(Y; 12 tt, D
exp
Temp
where Temp > 0 is user defined constant. This method will be referred to as softuncertain in the plots. As Temp + co, OSoRBALD will behave more like Organdom- And as Temp O, OrsoRBALD will behave more like OBBLL-
Coreset. Coreset acquisition looks to maximize the diversity of acquired samples. This is done by finding the data points in D Di that are furthest from the labelled data points in DD-11 The robust K-centers algorithm of [Sener & Savarese (2017) approximates a solution to:
aCORESET (t), (
argmin argmax argmin Att,,ty) (1..)))))) will GEat mat 4 tyepf uD!m
(8)
Euclidean distances, Att. tj), are calculated between the output of the penultimate layer of g(t; w).
Margin Sample. Margin sampling is designed for classifiers where selection based on the dis- tance of a sample from the classifiers decision boundary (Roth & Small 2006). As proxy. the difference between the predicted probability of the most and second most probable classes is used. The distance between the most probable and the second most probable classes for a multi-class clas- sification problem can be seen as how confident the model about the label of that class. However, The concept of decision boundary is ill-defined for regression tasks. One option to approximate margin sampling could be to model the aleatoric uncertainty of the model by predicting the condi- tional variance of the outcome o² (t; and select data based on the magnitude of this value. Here, we instead look at the difference in the maximum and minimum values of the predicted outcome as measure of the model's confidence and select data based on the magnitude of this value. Formally, we have
(e M(Y 1 t,, D = max (ā(t; 1)
min ((ttw/111) jE(L...m)
(9)
and the acquisition function:
O'Margin ( ((), Dr D
argmax (
M(Y; 14, D
(10)
Note that this approximation is similar to BALD under the assumption of a uniformly distributed outcome: fly () = U(y ((;w))).
18

---

Published as a conference paper at ICLR 2022
Ad Adversarial Basic Iteractive Method (AdvBIM). Some of the adversarial algorithms can act as active learning acquisition functions by nominating the adversarial samples. Here, we extended the famous Adversarial BIM method for our regression task as an example. BIM was introduced by (Kurakin et al.] 2016) to iteratively perturb adversarial samples to maximize the cost function J subject an Ip norm constraint
2 = t, ( = clip,,e (86-1) + sign( V Ro- (0,880)) (12 ,)))
(11)
(intermediate results are clipped to stay in e-neighbourhood of the primary data point 1). This technique bypasses the intractable problem of finding the distance from the decision boundary by iteratively perturbing the features until crossing the boundary (Tramèr et 2017). In our regression task, we perturb the features in the gradients' direction to increase the conditional variance of the outcome, i.c.,
t t,t (00) clip,, (t + sign V 20-1) Var for (1.....m),,
(12)
where ||4; - 4112 < 7 * ||1|/ with the hyperparameter y. After creating adversarial samples for each data point in avail AddversariaaBIMM acquires the samples by
AdversarialBIM (0" (t), Divaai)
U 4, Gval
argmin ((?!") .ty);
(13)
EyE
where A is the euclidean distance.
k-means Sampling. This method nominates samples by returning the closest sample to each cen- ter of the unlabeled data clusters. In order to do so, one may run Kmeans++ clustering algorithm with the number of clusters equal to b over either the unlabeled data points D & the output of the penultimate layer of ((::w). We refer to the former as kmeansdata and to the latter as kmeensembed in the experiments. Assuming .663 are the centers of the clustering, we have
a Kmeans 0 k- (t), Divan) = U argmin ((ss,
(14)
where A is euclidean distance over the data points the penultimate layer of g(l:w).
C DETAILED EXPERIMENTAL RESULTS
C. BAYESIAN NEURAL NETWORK (BNN) MODEL
We provide here detailed experimental results across all hyperparameter settings. The result of fig. that was presented for 3 batch sizes are provided for batch sizes in fig. 4 Similarly, the results of fig. 33 are provided for additional batch sizes in fig. 7 In addition, both fig. 2 and fig. B report the results for the STRING treatment descriptors. All experiments are repeated for two other sets of input treatment descriptors (Achilles and CCLE) whose results are provided in figs. g and 9
C.2 RANDOM FOREST MODEL
In addition to the BNN model, we carried out thorough analyses for different model class. The experiments are repeated for the random forest as an uncertainty aware ensemble model. The un- certainty in random forests. similar to other ensemble methods, originated from the prediction made by each model instance in the ensemble. We use the random forest implementation in the Scikit-learn package (Pedregosa et al. 2011) with 100 trees and set the option max _deaddddeeoeeeeeeeeee so that the depth of the trees are determined automatically. The performance of the model trained over the active learning cycles can seen in fig. 10 for different acquisition functions, different batch sizes, different target datasets, and the STRING treatment descriptors. Similarly, the hit ratio of the interesting genes for a random forest model is reported in fig. 12 The same experiment was repeated for CCLE treatment descriptors whose results provided in fig. and fig. [3] Notice that random forest experiments are done with a reduced set of acquisition functions that could be adjusted to the random forest model.
19

---

Published as a conference paper at ICLR 2022
C.3 IN-DEPTH DESCRIPTION OF THE Hit Ratio EXPERIMENT
Here we elaborate more on the purpose and the message of the hit ratio experiment whose results reported in figs. to 9 T2 and 13 for various settings. The purpose of these experiments is to compare the performance of different acquisition functions in different settings of batch sizes and input/output datasets hit the gene targets that are known be interesting by genomics experts. To choose the set of interesting genes, we sort them based on their absolute target values. Then we choose the top 5% of this list that corresponds to both extremes of positive and negative values (both extremes are considered to be good targets by experts.) The experiments are repeated for 5 different random seeds to obtain the error bars.
20

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
11-10
batch size -6
S.c
P4
iPhcc
2011
batch size -22
P4
Pt
batch size 64
csrsaacce NC
D2 12
g+
batch size = 128
8,12
6.5
0
Murber cranpentyy
102
batch size = 256
Cepparar Aag aibe Fasdon (hit
120
batch size = 512
142
01
500
Figure 4: The evaluation of the model trained with STRING treatment descriptors at each active learning cycle for 4 datasets and 6 acquisition batch sizes. In each plot, the x-axis the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
21

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt et al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
(emplinte 25111
Natertiial Puckbes SDhang
batch size -6
302
203
Corester
iPhcs
batch size 32
Corvater 2010
Crepussar
batch size = 64
DKC
batch size 128
1.50
100
26
Munde Cf D1
120
batch size = 2.5
Creaater dhte
20%
2012
102
batch size = 512
0.00
130
20d Rusbe' Juspar PC
Figure 5: The evaluation of the model trained with Achilles treatment descriptors at each active learning cycle for 4 datasets and 6 acquisition batch sizes. In each plot, the x-axis is the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
22

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt ct al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al, 2019
batch size 16
oh DS
Oor persotot of Acculation Fureters Stramn she 2999
batch size = 32
betgs
batch size = 64
batch size 128
BLiCE
batch sizee = 256
Coppertor
(PA))
batch size = 512
Competion wlwyylitee Rontena (Thcc
0.00
107
2.20
202
Figure 6: The evaluation of the model trained with CCLE treatment descriptors at each active learn- ing cycle for datasets and 6 acquisition batch sizes. In each plot, the x-xis the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
23

---

Published as a conference paper at ICLR 2022
Shifrut al. 2018
Schmidt et al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
Carpersco Frceeatarr Fulctoo (ihuing eta. 2016
16
batch size =
2013
batch size 32
000
batch size = -64
000
3300
c00
batch size = 128
Darol rastica ponstte 2-12
115
Exra
30
89
200
X30
batch size 256
C'qpuitor Fuadbrs Shany EX.. 2018
201
TD
TD
Surtie P4
batch size 512
Coppormn Aocustion Fonters (lobi dra 201)
Fucctem (hhingg eisl 29A
Figure 7: The hit ratio of different acquisition for BNN model, different target datasets, and different acquisition batch sizes. We use STRING treatment descriptors here. The x-axis shows the number of data points collected so far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
24

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
Purttara
Docastion Prodlia pictradt 11-4)
batch size -6
batch size = 32
batch size 64
undor chaagger
batch size 128
batch size = 256
604
10.0
134
batch size = 512
Darpateon 2018)
2016
122
Nartaa D1
Nartal Cianmpei P4
240
300
Figure& 8: The hit ratio of different acquisition for BNN model, different target datasets, and different acquisition batch sizes. We use Achilles treatment descriptors here. The x-axis shows the number of data points collected so far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
25

---

Published as a conference paper at ICLR 2022
Shifrut al. 2018
Schmidt et al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
Fulctor (Phsuay
batch size -66
batch size 32
lfe potice pittaee
372
Caroeloo ciN.gattottooaoeeo.o ghalny $18.200
batch size 64
andore Lohagget
batch size 128
Humber Coroone
122
2999
batch size = 256
Hower
batch size 512
Shaagg 2018
24
Figure 9: The hit ratio of different acquisition for BNN model, different target datasets, and different acquisition batch sizes. We use CCLE treatment descriptors here. The x-axis shows the number of data points collected so far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
26

---

Published as a conference paper at ICLR 2022
Shifrut al. 2018
Schmidt et al. 2021 (IFNg)
Schmidt et al. 2021 (1L-2)
Zhuang et al. 2019
Conpatmen Aspusttor Funduns (hatggg ou 2016
16
batch size =
3 4:1
corperson of
batch size = 32
hatch size = 64
(Ffic
2011
batch size 128
Fomptore 2011
cas
batch size = 2.56
Consation
Calareer dhogesio Fands Duvedll 2011
conpanson ETA.UURORT CODOO
batch size 512
Coesarar dh Mtataa (1962
0.21
143
Ampistor Puetoe Civnngate (9%
0.20
115
0.20
20
301
Figure 10: The evaluation of the random forest model trained with STRING treatment descriptors at each active learning cycle for 4 datasets and 6 acquisition batch sizes. In each plot, the x-axis is the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
27

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt et al. 2021 (IFNg)
Schmidt et al. 2021 (IL-2)
Zhuang et al. 2019
Comperaa Poientiool Fus WCw.eeeCCCAaaoooooo 410. 2016
batch size = 16
113
122
8
115
<40
145
batch size = 32
batch size 64
(Ffict
20111
1.5
140
batch size 128
Nunber of
batch size 256
2015
batch size = 512
Copportor 20111.2
Ampinttor Puetoe (hrenn 291
2:10
0.20
0.00
175
0.2
20
Martar dangie
3.6%
201
Figure 11: The evaluation of the random forest model trained with CCLE treatment descriptors at each active learning cycle for 4 datasets and 6 acquisition batch sizes. In each plot, the x-axis is the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
28

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt al. 2021 (IFNg)
Schmidt al. 2021 (IL-22
Zhuang et al. 2019
batch size = 16
Norte aarpptn)
21 29%
batch size = 32
batch size = -64
oiter Fustien (Dibu
siter Fandim Seneter 20 iPho)
dfey siee Fandoo Seeed a 3D
Campolt
Vanones
P
Exppin P)
P4
-30
batch size 128
Fantors & 20 iPlc)
Compatton d'Aquittrr picttess Shangea (019)
1.5
hatch size= 255
wlice Biies 220 51:30
Aopunter Phangg dia 2013
Scate
FD
A Simpln P4
batch size 512
Creparis Peeettioriaeeer elu 201)
Purk (Pheng (id 2015
6%
1221
Norder
2008
PC
Figure 12: The hit ratio of different acquisition for random forest model, different target datasets, and different acquisition batch sizes. We use STRING treatment descriptors here. The x-axis shows the number of data points collected so far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
29

---

Published as a conference paper at ICLR 2022
Shifrut et al. 2018
Schmidt al. 2021 (IFNg)
Schmidt al. 2021 (IL--2
Zhuang et al. 2019
batch size = 16
Norte aarpptn
21 Phampote 291
batch size = 32
batch size = -64
Faodom Senbre 20 iPh
dfey site Fastom Seeed e 32
Campott
370
43%
X00
Exppin P)
P4
batch size 128
Fastber Serecise 301 (Pfcg
Compatton d'Aquittrr picctess Shangea (1/0)
1.3
hatch size= 256
(Ph)
Aepunter phanig dia 2013
144
A of Simpln P4
batch size 512
Creparin Aecattion plubil 271)
Purk (Pheng 2015
04
1221
Norder
200
PC
Figure 13: The hit ratio of different acquisition for random forest model, different target datasets, and different acquisition batch sizes. We use CCLE treatment descriptors here. The x-axis shows the number of data points collected so far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
30

---

Published as a conference paper at ICLR 2022
D ADDITIONAL DATASETS
We tested the GeneDisco pipeline on two additional datasets which are included this appendix due space constraints. The relative performance of the random forest model with the acquisition functions that are compatible wit this model class are shown in Figure 14 for the predictive accuracy task and Figure for the hit ratio task.
D.1 MODULATION OF TAU PROTEINS IN NEURONS
Experimental setting. This assay is based on Sanchez et al. (202 in which authors have con- ducted genome-wide CRISPR screens in two SH-SY5Y neuroblastoma cell lines to identify genes that, when knocked out, either increased decreased expression of endogenous tau proteins.
Measurement. After editing (for 21 or 30 days), cells are FACS sorted based on low tau expression (low quartile 25% and high tau expression (high quartile 25%). Statistical significance of gRNA enrichment is determined via a redundant siRNA activity (RSA; log p-value) analysis. RSA up scores were used to find genes enriched in the 25% of cells with highest tau protein, while RSA down scores were used to find genes enriched in the 25% of cells with the lowest tau protein.
Importance. While the exact mechanism leading the buildup of Tau proteins is unknown, their accumulation is correlated with several neurodegenerative pathologies (eg.. Alzheimer's s disease, progressive supranuclear palsy, and frontotemporal dementia). The genes identified in this screen may therefore help gaining a better understanding of the underlying disease pathways as well as leading to potential treatments.
D.2 REGULATION OF ENDOSOMAL ENTRY IN CELLS FOR SARS-COV-2
Experimental setting. This dataset is based on the genome-wide CRISPR screen described in Zhu
et (2021). The assay was designed to identify endosomal entry-specific regulators -AV.-CoV. 2 virion in AS42-ACE2 cells.
Measurement. The top candidates from the CRISPR screen were determined according to their MAGeCK score (- 10).
Importance. The endosomal pathway one of the two pathways (along with fusion at the plasma membrane) used by SARS-CoV-2 to infect cells. A better understanding of mechanisms under- pinning this pathway may help identify targets for the development of new SARS-CoV-2 antiviral therapeutics.
31

---

Published as a conference paper at ICLR 2022
Sanchez et al. 2021
Zhu et al. 2021
Sanchez et al. 2021
7hu et al. 2021
batch size 16
20111
Cospulben Aupeste Purber Dolnod 4.. 1645
3
032
batch size = 32
Standhad 12211
0.0
batch size 64
2011
(1.7g
batch size 128
Congarase Shaet
Concomaan
batch size 256
Companer oldey wew F1-25
030
CCLE Teatures
CCLE features
STRING features
STRING features
Figure 14:
The evaluation of the random forest model trained with CCLE and STRING treatment
descriptors at each active learning cycle for the datasets (Sanchez et al. 2021 and (Zhu et al. 2021) and also for 5 acquisition batch sizes. In each plot, the x-axis is the active learning cycles multiplied by the acquisition bath size that gives the total number of data points collected so far. The y-axis is the test MSE error evaluated on the test data.
32

---

Published as a conference paper at ICLR 2022
Sanchez et al. 2021
Zhu et al. 2021
Sanchez et al. 2021
Zhu et al. 2021
batch size 16
Conperioo AtAceut Struet 29112
Crapecton offecten foiner 4.d. (52 9112
Camsr fet chiquttic wostettchhhccteo (hoang
batch size = 32
batch size = 64
Centeree 2010
2.12
500
0.00
430
200
batch size 128
Aldeei oe Shoeeca 20111
Concetan Crcpestinn sctors dutend eta, S012
608
Scide
210
03C
batch size = 256
CCLE features
CCLE features
STRING features
STRING features
Figure 15: The hit ratio of different acquisition for random forest model, different target datasets, and different acquisition batch sizes. We use STRING and CCLE treatment descriptors here. The x-axis shows the number of data points collected so far during the active learning cycles. The y-axis shows the ratio of the set of interesting genes that have been found by the acquisition function up until each cycle.
33


Reviews:
[{"review_id": "z1sWYNn1_Qr", "reviewer": "Reviewer_NhJz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) The authors are commended for addressing the hard problem of causal inference in biological systems. \n2) The authors give a good explanation of the methodology, datasets, and metrics.\n3) A good explanation of the genetic assays are provided\n\nWeakness:\n1) The title is misleading. The paper is much more about gene knockdown than drug discovery. As presented, causal connections between specific loss-of-function mutations and immunologic phenotypes may be drawn, but drug discovery is not directly related. \n2) The authors do not provide any novel methodologies within the field of active learning or drug development.\n3) The assays are limited to a small subset of immunologic phenotypes.\n", "summary_of_the_paper": "The authors introduce a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery. Specifically, they introduce various genome-wide CRISPR screens within immunology that evaluate the causal effect of intervening on a large number of genes in model systems in order to identify the genetic perturbations that induce a desired phenotype. The work focuses on using counterfactual estimators of experimental outcomes to propose experimental hypotheses for validation in in vitro experiments with genetic interventions (CRISPR) in order to discover potential causal associations between biological entities that could be relevant for the development of novel therapeutics. \n\nThe authors standardize two types of datasets: three standardized feature sets describing interventions and four difference in vitro genome-wide CRISPR experimental assays. Two model types are used (BNN and Random Forest Regression) and nine different acquisition functions. \n\nResults are presented in Figures 2 and 3. \n", "summary_of_the_review": "Overall, the authors provide a decent dataset and benchmark for genetic knockdown experimentation in the setting of active learning. However, I believe the paper would be better suited for datasets/benchmarks venue. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZQ6qNmJW8u9", "reviewer": "Reviewer_6EvA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- I feel this kind of benchmark would be useful. Although the  data sets are already publicly available, it's nevertheless nice if someone else has done the required work and one can start modeling immediately.\n- The presentation is mostly clear.\n\nCons/questions:\n- The paper is straightforward, does not have any particular shortcomings, and is potentially useful to the community. However, I feel it lacks the kind of machine learning novelty and significance that would be expected from an ICLR publication, and hence some other forum could be more suitable (though I'm happy to discuss this with the other reviewers and the area chair).\n- Figures 2 and 3 require a lot of zooming in before they are readable, and they are not at all legible in a printed-out version.\n- I found the discussion on the comparison of acquisition methods a bit superficial.", "summary_of_the_paper": "The paper presents a benchmark for experimental design in drug discovery based on previously public data sets. The benchmark includes three standardized feature sets describing possible interventions and four different genome-wide CRISPR experiment results (from which the counterfactual outcomes for an intervention on each gene are obtained). In addition, the paper uses the benchmark to compare different active learning strategies (nine acquisition functions in total), where the goal is to improve prediction accuracy and hit ratio of interesting genes among the interventions selected by a specific (batch) acquisition function.\n", "summary_of_the_review": "A straightforward paper that presents a useful-looking benchmark dataset for drug discovery. Machine learning novelty and significance is limited.\n\nPOST-REBUTTAL UPDATE:\nAfter clarifications, it seems apparent that benchmark papers like this could fall in the scope of ICLR. I still find the benchmark potentially useful and something I might use in my own research. I do not have any particular shortcomings in mind, but I think a stronger demonstration of the ability to generate new machine learning insights using the benchmark would have been useful. I will update my score to borderline positive (6) to reflect these thoughts.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "79tQpHphePc", "reviewer": "Reviewer_ozjG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this paper include :\n\n-  The paper focuses on an important aspect of experimental design for drug discovery, which has not previously been studied. This type of benchmarking is invaluable to make the best possible use of limited experimental resources.\n\n- The datasets are carefully designed with input from researchers in genomics and CRISPR screens, minimizing potential sources of error (e.g., no duplication).\n\n- It has the right level of detail; it makes specific recommendations that can be readily implemented by developers or researchers interested in working with similar data sets.\n\n- Source code is made publicly available online.\n\nThe limitations include: \n\n- The benchmarks contain relatively little information about how far apart genes are located (i.e., there is no distance metric). In practice, this information can be very useful for active learning algorithms. \n\n- The authors study only two candidate algorithms so far; however it is still an important step forward because the novel dataset and methodology enable other groups to conduct extensive evaluations on their own machine learning algorithms.\n\n- The problem formulation should be clarified and is difficult to understand as there are multiple different datasets presented. \n\n- A thorough justification for the assumptions utilized in the parametric models; specifically the outcome Y, should be disclosed.\n\n- How do the authors define the ground truth when evaluating the active learning strategies?\n\n- How do the authors define hits for the different active learning strategies?\n\n- The results show rather uniform performance regardless of the active learning strategy employed. Can the authors comment on the reasoning around this phenomenon and whether it is an artifact of the problem formulation or datasets employed?\n\n- It is unclear if biological replicates are included in the experiments. A change in this fundamental would have a very substantial impact on experimental design benchmarks and biology interpretations. If replicates were not used, they should be added to the text. ", "summary_of_the_paper": "The authors propose a benchmarking suite for evaluating active learning strategies to experimental design in drug discovery. The authors focus on active learning for in vitro genetic experiments, where the goal is to identify which genes or proteins underlie a certain disease (e.g., by using CRISPR technologies). For this task, there are typically billions of potential hypotheses to test. The experimental design space for such in vitro cellular experiments is extremely vast; thus, even with help from modern computers and machine-learning algorithms (which can search through the entire space), it remains practically impossible to exhaustively test every possible hypothesis that lives in this biological hypothesis space.\n\nThe paper contributes a benchmarking suite that can be used to evaluate how well active learning strategies are able to learn useful experimental designs. The paper has the following contributions:\n\n- A new experimental design benchmark suite of genetic experiments for CRISPR screens with unique challenges not found in other benchmarks. This includes demonstrating cost-efficiency by using only a small number of experiments relative to the total number of hypotheses tested.\n\n- An evaluation methodology for experimental designs on GeneDisco data sets, including methods for evaluating cost efficiency and quality of information gain during active learning. \n\n- Initial evaluations apply this approach to two candidate causal inference algorithms: one based on a Bayesian neural network and another based on a Random Forest. Further methodological development is required in order to explore more sophisticated algorithms for active learning.\n\n- The paper is published with supplementary material, including the two experimental design benchmark datasets and all source code necessary to reproduce the results of this paper.", "summary_of_the_review": "All in all, this paper proposes a benchmarking suite that can help developers of active learning algorithms improve their programs. While the paper only considers a limited set of models, active learning algorithms, and datasets, it is an important step towards establishing best practices for in vitro experimental design. With source code publicly available online, it provides necessary tools (and datasets) to advance the state of the art for experimental design in drug discovery studies that use CRISPR technologies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a benchmark for experimental design algorithms
for an important cellular biological question, causal discovery of
effective genetic knock-out interventions. It uses existing datasets.

The paper was discussed by the reviewers after the authors correctly
pointed out that methodological machine learning novelty is not a
necessary condition for accepting papers. Two reviewers increased
their scores and all are slightly positive. The benchmark was seen as
valuable, and one reviewer even commented they might use it in their
own research. However, the paper is still on the borderline as this
benchmark is only a first step. It has not been shown yet that machine
learning insights can be produced with it, as the authors have not 
actually used it for benchmarking yet. In other words, the benchmark
can be considered a potentially excellent idea which has not been
tested empirically yet.

This seems a highly promising research direction and the authors are
strongly encouraged to continue to providing the benchmarks and
releasing the method to the community so that others can help them in that.
Decision: Accept (Poster)