GeneDisco: A Benchmark for Experimental Design in Drug Discovery

1 INTRODUCTION The discovery and development of new therapeutics is one of the most challenging human endeavours with success rates of around 5 % ( Hay et al. , 2014 ; Wong et al. , 2019 ) , timelines that span on average over a decade ( Dickson & Gagnon , 2009 ; 2004 ) , and monetary costs exceeding two billion United States ( US ) dollars ( DiMasi et al. , 2016 ; Berdigaliyev & Aljofan , 2020 ) . The successful discovery of drugs at an accelerated pace is critical to satisfy current unmet medical needs ( Rawlins , 2004 ; Ringel et al. , 2020 ) , and , with thousands of potential treatments currently in development ( informa PLC , 2018 ) , increasing the probability of success of new medicines by establishing causal links between drug targets and diseases ( Nelson et al. , 2015 ) could introduce an additional hundreds of new and potentially life-changing therapeutic options for patients every year . However , given the current estimate of around 20 000 protein-coding genes ( Pertea et al. , 2018 ) , a continuum of potentially thousands of cell types and states under a multitude of environmental conditions ( Trapnell , 2015 ; MacLean et al. , 2018 ; Worzfeld et al. , 2017 ) , and tens of thousands of cellular measurements that could be taken ( Hasin et al. , 2017 ; Chappell et al. , 2018 ) , the combinatorial space of biological exploration spans hundreds of billions of potential experimental configurations , and vastly exceeds the experimental capacity of even the world ’ s largest research institutes . Machine learning methods , such as active and reinforcement learning , could potentially aid in optimally exploring the space of genetic interventions by prioritising experiments that are more likely to yield mechanistic insights of therapeutic relevance ( Figure 1 ) , but , given the lack of openly accessible curated experimental benchmarks , there does not yet exist to date a concerted effort to leverage the machine learning community for advancing research in this important domain . To bridge the gap between machine learning researchers versed in causal inference and the challenging task of biological exploration , we introduce GeneDisco , an open benchmark suite for evaluating batch active learning algorithms for experimental design in drug discovery . GeneDisco consists of several curated datasets , tasks and associated performance metrics , open implementations of stateof-the-art active learning algorithms for experimental design , and an accessible open-source code base for evaluating and comparing new batch active learning methods for biological discovery . Concretely , the contributions presented in this work are as follows : • We introduce GeneDisco , an open benchmark suite for batch active learning for drug discovery that provides curated datasets , tasks , performance evaluation and open source implementations of state-of-the-art algorithms for experimental exploration . • We perform an extensive experimental baseline evaluation that establishes the relative performance of existing state-of-the-art methods on all the developed benchmark settings using a total of more than 20 000 central processing unit ( CPU ) hours of compute time . • We survey and analyse the current state-of-the-art of active learning for biological exploration in the context of the generated experimental results , and present avenues of heightened potential for future research based on the developed benchmark . 2 RELATED WORK Background . Drug discovery is a challenging endeavour with ( i ) historically low probabilities of successful development into clinical-stage therapeutics ( Hay et al. , 2014 ; Wong et al. , 2019 ) , and , for many decades until recently ( Ringel et al. , 2020 ) , ( ii ) declining industry productivity commonly referred to as “ Eroom ’ s law ” ( Scannell et al. , 2012 ) . Seminal studies by Nelson et al . ( 2015 ) and King et al . ( 2019 ) respectively first reported and later independently confirmed that the probability of clinical success of novel therapeutics increases up to three-fold if a medicine ’ s molecular target is substantiated by high-confidence causal evidence from genome-wide association studies ( GWAS ) Visscher et al . ( 2017 ) . With the advent of molecular technologies for genetic perturbation , such as Clustered Regularly Interspaced Short Palindromic Repeats ( CRISPR ) ( Jehuda et al. , 2018 ) , there now exist molecular tools for establishing causal evidence supporting the putative mechanism of a potential therapeutic hypothesis by means of in vitro experimentation beyond GWAS early on during the target identification and target validation stages of drug discovery ( Rubin et al. , 2019 ; Itokawa et al. , 2016 ; Harrer et al. , 2019 ; Vamathevan et al. , 2019 ) . Among other applications ( Chen et al. , 2018 ; Ekins et al. , 2019 ; Mak & Pichika , 2019 ) , machine learning methods , such as active and reinforcement learning , could potentially aid in discovering the molecular targets with the highest therapeutic potential faster . Machine Learning for Drug Discovery . There exist a number of studies that proposed , advanced or evaluated the use of machine learning algorithms for drug discovery : Costa et al . ( 2010 ) used decision tree meta-classifier to identify genes associated with morbidity using protein-protein , metabolic , transcriptional , and sub-cellular interactions as input . Jeon et al . ( 2014 ) used a support vector machine ( SVM ) on protein expressions to predict target or non-target for breast pancreatic or ovarian cancers , and Ament et al . ( 2018 ) used least absolute shrinkage and selection operator ( LASSO ) regularised linear regression on transcription factor binding sites and transcriptome profiling to predict transcriptional changes associated with Huntington ’ s disease . In the domain of human muscle ageing , Mamoshina et al . ( 2018 ) used a SVM on deep features extracted from gene expression signatures in tissue samples from young and old subjects to discover molecular targets with putative involvement in muscle ageing . More recently , Stokes et al . ( 2020 ) utilised deep learning to discover Halicin as a repurposed molecule with antibiotic activity in mice . Reinforcement and Active Learning . The use of deep reinforcement learning for de novo molecular design has been extensively studied ( Olivecrona et al. , 2017 ; Popova et al. , 2018 ; Putin et al. , 2018 ; Lim et al. , 2018 ; Blaschke et al. , 2020 ; Gottipati et al. , 2020 ; Horwood & Noutahi , 2020 ) . Active learning for de novo molecular design has seen less attention ( Dixit et al. , 2016 ; Green et al. , 2020 ) , however , active learning for causal inference has seen increasing application toward causaleffect estimation ( Deng et al. , 2011 ; Schwab et al. , 2018 ; Sundin et al. , 2019 ; Schwab et al. , 2020 ; Bhattacharyya et al. , 2020 ; Parbhoo et al. , 2021 ; Qin et al. , 2021 ; Jesson et al. , 2021 ; Chau et al. , 2021 ) , and causal graph discovery ( Ke et al. , 2019 ; Tong & Koller , 2001 ; Murphy , 2001 ; Hauser & Bühlmann , 2014 ; Ghassami et al. , 2018 ; Ness et al. , 2017 ; Agrawal et al. , 2019 ; Lorch et al. , 2021 ; Annadani et al. , 2021 ; Scherrer et al. , 2021 ) . Benchmarks . Benchmark datasets play an important role in developing machine learning methodologies . Examples include ImageNet ( Deng et al. , 2009 ) or MSCOCO ( Lin et al. , 2014 ) for computer vision , as well as cart-pole ( Barto et al. , 1983 ) or reinforcement learning ( Ahmed et al. , 2020 ) . Validation of active learning for causal inference methods depends largely on synthetic data experiments due to the difficulty or impossibility of obtaining real world counterfactual outcomes . For causal-effect active learning , real world data with synthetic outcomes such as IHDP ( Hill , 2011 ) or ACIC2016 Dorie et al . ( 2019 ) are used . For active causal discovery , in silico data such as DREAM4 ( Prill et al. , 2011 ) or the gene regulatory networks proposed by Marbach et al . ( 2009 ) are used . Non synthetic data has been limited to protein signalling networks ( Sachs et al. , 2005 ) thus far . Several benchmarks focus on some aspects of drug discovery , for instance Brown et al . ( 2019 ) focusing on de novo molecular screening . However , to our knowledge , no other benchmark addressing the target identification part of the drug discovery pipeline , which is what our work focuses on . In contrast to existing works , we develop an open benchmark to evaluate the use of machine learning for efficient experimental exploration in an iterative batch active learning setting . To the best of our knowledge , this is the first study ( i ) to introduce a curated open benchmark for the challenging task of biological discovery , and ( ii ) to comprehensively survey and evaluate state-of-the-art active learning algorithms in this setting . 3 METHODOLOGY Problem Setting . We consider the setting in which we are given a dataset consisting of covariates X ∈ Rp with input feature dimensionality p ∈ N and treatment descriptors T ∈ Rq with treatment descriptor dimensionality q ∈ N+ that indicate similarity between interventions . Our aim is to estimate the expectation of the conditional distribution of an unseen counterfactual outcome Yt ∈ R given observed covariates X = x and intervention do ( T = t ) , ŷt = ĝ ( X = x , do ( T = t ) ) ≈ E [ Y | X = x , do ( T = t ) ] . This setting corresponds to the Rubin-Neyman potential outcomes framework ( Rubin , 2005 ) adapted to the context of genetic interventions with a larger number of parametric interventions . In the context of a biological experiment with genetic interventions , yt is the experimental outcome relative to a non-interventional control ( e.g. , change in pro-inflammatory effect ) that is measured upon perturbation of the cellular system with intervention t , x is a descriptor of the properties of the model system and/or cell donor ( e.g. , the immuno-phenotype of the cell donor ) , and t is a descriptor of the genetic intervention ( e.g. , a CRISPR knockout on gene STAT1 ) that indicates similarity to other genetic interventions that could potentially be applied . In general , certain ranges of yt may be preferable for further pursuit of an intervention T = t that inhibits a given molecular target - often , but not necessarily always , larger absolute values that move the experimental result more are of higher mechanistic interest . We note that the use of an empty covariate vector X = x0 with p = 0 is permissible if all experiments are performed in the same model system with the same donor properties . In in vitro experimentation , the set of all possible genetic interventions Dpool = { ti } npool i=1 is typically known a-priori and of finite size ( e.g. , knockout interventions on all 20 000 protein-coding genes ) . Batch Active Learning . In the considered setting , reading out additional values for yet unexplored interventions t requires a lab experiment and can be costly and time-consuming . Lab experiments are typically conducted in parallelised manner , for example by performing a batch of multiple interventions at once in an experimental plate . Our overall aim is to leverage the counterfactual estimator ĝ trained on the available dataset to simulate future experiments with the aim of maximising an objective function L in the next iteration with as few interventions as possible . For the purpose of this benchmark , we consider the counterfactual mean squared error ( MSE ) of ĝ in predicting experimentally observed true outcomes yt from predicted outcomes ŷt as the optimisation objective LMSE = MSE ( yt , ŷt ) . Depending on context , other objective functions , such as for example maximising the number of discovered molecular targets with certain properties of interest ( e.g. , druggability ( Keller et al. , 2006 ) ) may also be sensible in the context of biological exploration . At every time point , a new counterfactual estimator ĝ is trained with the entire available experimental dataset , and used to propose the batch of b interventions to be explored in the next iteration with the batch size b ∈ N+ . When using LMSE , this setting corresponds to batch active learning with the optimisation objective of maximally improving the counterfactual estimator ĝ . Acquisition Function . An acquisition function Dk = α ( g ( t ) , Dkavail ) takes the model and the set of all available interventions Dkavail in cycle k as input and outputs the set of interventions Dk that are most informative after the kth experimental cycle with cycle index k ∈ [ K ] = [ 0 . . K ] where K ∈ N+ is the maximum number of cycles that can be run . Formally speaking , the acquisition function α : P ( Davail ) ×G → P ( Davail ) takes a subset of all possible interventions that have not been tried so far ( Davail ) , together with the trained model ( ĝ ) derived from the cumulative data collected over the previous cycles , and outputs a subset of the available interventions Dk that are likely to be most useful under L to obtain ĝ ∈ G as a better estimate of E [ Y | X = x , do ( T = t ) ] with G being the space of the models which can be , e.g . the space of models and ( hyper- ) parameters . 4 DATASETS , METRICS & BASELINES The GeneDisco benchmark curates and standardizes two types of datasets : three standardized feature sets describing interventions t ( inputs to counterfactual estimators ; Section 4.1 ) , and four different in vitro genome-wide CRISPR experimental assays ( predicted counterfactual outcomes ; Section 4.2 ) , each measuring a specific outcome yt following possible interventions T . We perform an extensive evaluation across these datasets , leveraging two different model types ( Section 4.3 ) and nine different acquisition functions ( Section 4.4 ) . Since all curated assay datasets contained only outcomes for only one model system , we used the empty covariate set X = x0 for all evaluated benchmark configurations . The metrics used to evaluate the various experimental conditions ( acquisition functions and model types ) include model performance ( Figure 2 ) and the ratio of discovered interesting hits ( Figure 3 ) as a function of number of samples queried . 4.1 TREATMENT DESCRIPTORS The treatment descriptors T characterize a genetic intervention and generally should correspond to data sources that are informative as to a genes ’ functional similarity - i.e . defining which genes if acted upon , would potentially respond similarly to perturbation . The treatment descriptor T is the input to the model described in Section 3 for a fixed vector of covariates x . Any dataset considered for use as a treatment descriptor must be available for all potentially available interventions Dpool in the considered experimental setting . In GeneDisco , we provide three standardised gene descriptor sets for genetic interventions , and furthermore enable users to provide custom treatment descriptors via a standardised programming interface : Achilles . The Achilles project generated dependency scores across cancer cell lines by assaying 808 cell lines covering a broad range of tissue types and cancer types ( Dempster et al. , 2019 ) . The genetic intervention effects are based on interventional CRISPR screens performed across the included cell lines . When using the Achilles treatment descriptors , each genetic intervention is summarized using a gene representation T with q = 808 corresponding to the dependency scores measured in each cell line . In Achilles , after processing and normalisation ( see Dempster et al . ( 2019 ) ) , the final dependency scores provided are such that the median negative control ( non-essential ) gene effect for each cell line is 0 , and the median positive control ( essential ) gene effect for each cell line is -1 . The rationale for using treatment descriptors based on the Achilles dataset is that genetic effects measured across the various tissues and cancer types in the 808 cell line assays included in ( Dempster et al. , 2019 ) could serve as a similarity vector in functional space that may extrapolate to other biological contexts due to its breadth .

Summary: The authors propose a benchmarking suite for evaluating active learning strategies to experimental design in drug discovery. The authors focus on active learning for in vitro genetic experiments, where the goal is to identify which genes or proteins underlie a certain disease (e.g., by using CRISPR technologies). For this task, there are typically billions of potential hypotheses to test. The experimental design space for such in vitro cellular experiments is extremely vast; thus, even with help from modern computers and machine-learning algorithms (which can search through the entire space), it remains practically impossible to exhaustively test every possible hypothesis that lives in this biological hypothesis space.  The paper contributes a benchmarking suite that can be used to evaluate how well active learning strategies are able to learn useful experimental designs. The paper has the following contributions:  - A new experimental design benchmark suite of genetic experiments for CRISPR screens with unique challenges not found in other benchmarks. This includes demonstrating cost-efficiency by using only a small number of experiments relative to the total number of hypotheses tested.  - An evaluation methodology for experimental designs on GeneDisco data sets, including methods for evaluating cost efficiency and quality of information gain during active learning.   - Initial evaluations apply this approach to two candidate causal inference algorithms: one based on a Bayesian neural network and another based on a Random Forest. Further methodological development is required in order to explore more sophisticated algorithms for active learning.  - The paper is published with supplementary material, including the two experimental design benchmark datasets and all source code necessary to reproduce the results of this paper.