DiscoBAX - Discovery of optimal intervention sets in genomic experiment design
Authors: Clare Lyle
Arash Mehrjou
Pascal Notin
Andrew Jesson
Stefan Bauer
Yarin Gal
Patrick Schwab
Venue: Submitted to ICLR 2023 (2023)
Keywords: Optimal experiment design
Bayesian Algorithm Execution
Active learning
Genetic intervention
Drug design

Abstract: The discovery of novel therapeutics to cure genetic pathologies relies on the identification of the different genes involved in the underlying disease mechanism. With billions of potential hypotheses to test, an exhaustive  exploration of the entire space of potential interventions is impossible in practice. Sample-efficient methods based on active learning or bayesian optimization bear the promise of identifying interesting targets using the least experiments possible. However, genomic perturbation experiments typically rely on proxy outcomes measured in biological model systems that may not completely correlate with the outcome of interventions in humans. In practical experiment design, one aims to find a set of interventions which maximally move a target phenotype via a diverse set of mechanisms in order to reduce the risk of failure in future stages of trials. To that end, we introduce DiscoBAX — a sample-efficient algorithm for the discovery of genetic interventions that maximize the movement of a phenotype in a direction of interest while covering a diverse set of underlying mechanisms. We provide theoretical guarantees on the optimality of the approach under standard assumptions, conduct extensive experiments in synthetic and real-world settings relevant to genomic discovery and demonstrate that DiscoBAX outperforms state-of-the-art active learning and Bayesian optimization methods in this task. Better methods for selecting effective and diverse perturbations in biological systems could enable researchers to potentially discover novel therapeutics for a range of genetically-driven diseases.

Under review as a conference paper at ICLR 2023
DISCOBAX: DISCOVERY OF OPTIMAL INTERVENTION SETS IN GENOMIC EXPERIMENT DESIGN
Anonymous authors Paper under double-blind review
ABSTRACT
The discovery of therapeutics treat genetically-driven pathologies relies on identifying genes involved in the underlying disease mechanism. With billions of potential hypotheses to test, an exhaustive exploration of the entire space of po- tential interventions is impossible in practice. Sample-efficient methods based on active learning or Bayesian optimization bear the promise of identifying targets of interest using as few experiments as possible. However, genomic perturba- tion experiments typically rely on proxy outcomes measured in biological model systems that may not completely correlate with the results of interventions in hu- mans. In practical experiment design, one aims to find a set of interventions that maximally move a target phenotype via diverse mechanism set to reduce the risk of failure in future stages of trials. To that end, we introduce DiscoBAX sample-efficient algorithm for genomic intervention discovery that maximizes the desired movement of a phenotype while covering a diverse set of underlying mechanisms. We provide theoretical guarantees the optimality of the approach under standard assumptions, conduct extensive experiments in synthetic and real- world settings relevant to genomic discovery, and demonstrate that DiscoBax out- performs state-of of-the-art active learning and Bayesian optimization methods in this task. Better methods for selecting effective and diverse perturbations in bio- logical systems could enable researchers to discover novel therapeutics for many genetically-driver diseases.
INTRODUCTION
Genomic experiments probing the function of genes under realistic cellular conditions are the cor- nerstone of modern early-stage drug target discovery and validation; moreover, they are used to identify effective modulators of one or more disease-relevan cellular processes. These experiments, for example using Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR) (Jehuda et al., 2018) perturbations, are both time and resource-intensive (Dickson & Gagnon, 2004; 2009; DiMasi et al., 2016; Berdigaliyev & Aljofan, 2020). Therefore, an exhaustive search of the bil- lions of potential experimental protocols covering all possible experimental conditions, cell states, cell types, and perturbations (Trapnell, 2015; Hasin et al., 2017: Worzfeld et al., 2017: Chappell al., 2018; MacL_ean al., 2018; Chappell al., 2018) is infeasible biomedical research institutes. Furthermore, to mitigate the chances of failure in subsequent stages of the drug design pipeline, it is desirable for the subset of precursors selected in the target iden- tification stage to operate on diverse underlying biological mechanisms (Nica et al., 2022). That way, if promising candidate based on in-vitro experiments triggers unexpected issues when tested in-vivo (e.g., undesirable side effects), other lead precursors relying on different pathways might be suitable replacements that are not subject to the same issues. Mathematically, finding a diverse set of precursors corresponds to identifying and sampling from the different modes of the black-box objective funetion mapping intervention representations to the corresponding effects on the disease phenotype ($ 2). Existing machine learning methods for iterative experimental design (c.g., active learning, Bayesian optimization) have the potential to aid in efficiently exploring this vast biological intervention space. However, to knowledge, there is method geared toward identifying the modes of the underlying black-box objective function to identify candidate interventions that are both effective and diverse 6).
for the world's largest
To this end, we introduce DiscoBAX a sample-efficient Bayesian Algorithm eXecution (BAX)
method for discovering genomic intervention sets with both high expected change in the target phe-

---

Under review as a conference paper at ICLR 2023
high
onn onle
DiecoBex
Sorthuutoo of
Diversky-seeking
number samples
Valun-scoking
20000 Ne Sorroo
number of samples
Figure I: We compare DiscoBAX (orange star) existing diversity-secking (dark grey circle) and value-seeking (light grey triangle) batch active learning policies. DiscoBAX aims to recover maxi- mally diverse set of interventions with values above a pre-defined threshold from a given underlying distribution. This aim contrasts with value-seeking strategies focusing maximizing value and diversity-seeking strategies focusing on maximizing coverage. expect DiscoBAX to design ge- nomic experiments yielding high value findings that maximize mode coverage. As discussed in $ 1, the diversity of selected interventions is highly desirable to increase the chances that at least some of these interventions will succeed in subsequent stages of the drug discovery pipeline.
notype and high diversity to maximize chances of success in the following stages of drug develop- ment (Figure 1), which formalize set-valued maximization problem (Equation 4), After pro- viding theoretical guarantees on the optimality of the presented approach under standard conditions, we perform a comprehensive experimental evaluation in both synthetic and real-world datasets. The experiments show that DiscoBAX outperforms existing state-of-the-ee of-the-art active learning and Bayesian optimization methods in designing genomic experiments that maximize the yield of findings that could lead to the discovery of new potentially treatable disease mechanisms.
Our contributions are as follows:
We formalize the gene target identification problem ($ 3) and discuss limitations of existing methods in addressing this problem ($ 6).
We develop DiscoBAX a sample-efficient BAX method for maximizing the rate of signif- icant discoveries per experiment while simultaneously probing for a wide range of diverse mechanisms during a genomic experiment campaign ($ 4).
We provide theoretical guarantees that substantiate the optimality of DiscoBAX under stan- dard assumptions ($ and Appendix A).
We conduct comprehensive experimental evaluation covering both synthetic well as real-world experimental design tasks that demonstrate that DiscoBAX outperforms existing state-of-the-an methods for experimental design in this setting 5).
2 BACKGROUND AND NOTATION
Genomic experimentation is early stage in drug discovery where geneticists assess the effect of genomic interventions on moving set of disease-relevant phenotypes to determine suitable drug targets. In an abstract language, we assume a black-box function, f -> IR. that maps each gene, E G, to the value, fgg) corresponding to the magnitude of phenotypic change under gene knock out. The set, G. finite, | m < 00, because there are a limited number of protein-encoding genes in the human genome (z 20, 000) (Pertea et al., 2018), and is formalizable by either the set of integers or one-hot vectors with dimension m. However, biologically informed embeddings, X G + X, are often preferred to represent genes for their potential to capture genetic, functional relationships. We assume that gene embeddings, X(g) = x E X C Rdd, are d-dimensional variables, with distinct members, X m, thus, f(g) and f(x) interchangeably.
2

---

Under review as a conference paper at ICLR 2023
In drug development, a candidate target must meet several criteria proceed to subsequent stages in the development pipeline. For example, engaging the target - down- or up-regulating the gene must move the phenotype significantly in the desired direction. Such genes are called "top-movers" of the phenotype. We can define the K top-movers for given phenotype as members of the set, X' = the K largest values of (xa): each evaluation of the phenotype change, f. requires a CRISPR-Cas9 knockout experiment in the lab, which makes exhaustive experimentation infeasible even for the most resourceful institutions. Hence in practice, the experimentation budget is limited to T < m experiments. Instead of choosing the K top-movers (requiring phenotype change knowledge, f(x), for all inputs x E X)), a more practical approach is to form the subset, X. C X. of genes that when knocked out lead to change in the phenotype. [(x), larger than a selected threshold value, e, i.e. X. = {x E X: f(x) 2 c}.
{x1, X2,..., Xm ). corresponding
(x..). However,
Bayesian Algorithm Execution (BAX), proposed by Neiswanger et al. (2021), is a method to esti- mate the output, A == O (f). of an algorithm, A, run on a function, f. by evaluating the function on budgeted set of inputs, {x, E X. Estimating a computable property is done by positing a probabilistic model for f for estimating OA. Data is acquired by searching for the value x X that maximizes the mutual information, I(Y OA DD), between the function output, Yx. and the algorithm output. OΛ. BAX assumes that functional output instances, Yx. of the function, f. can be observed for each acquired x. The acquisition of data is sequential, where the information gain maximization procedure leads to a dataset of observations, D; == [(x,, Yx,) ). at step E [T]. BAX can be used in conjunction with a number of algorithms, such as determining the superlevel set (i.c. Xo), computing integrals, or finding local optima of f. Given that genomic experimentation seeks find diverse set of genes corresponding to the modes of f, the BAX framework is well suited to our task.
Concretely, BAX acquisition functions select points by maximizing the expected information gain (EIG) obtained from each point about the output of the algorithm. Crucial to the applicability of BAX to our problem setting is the tractability of accurate approximators of the EIG for algorithms which, like the one we will propose, return subset of their inputs. The exact computation of the EIG for arbitrary algorithms is not generally tractable; however, Neiswanger et al. (2021) present an approximation that only requires the computation of the entropy of the distribution over function values conditioned on algorithm outputs.
EIG; (x, D2) = H (f)(())) (H,
(1)
When the model P is a Gaussian Process, both of these quantities are straightforward to compute: the first the entropy of the GP's predictive distribution at x, and we can estimate the second by conditioning posterior on the values of elements in the set S. Monte Carlo approximation of this quantity is possible when the model P does not permit closed form.
PROBLEM SETTING
A primary challenge in the drug discovery pipeline is the discrepancy in outcomes between in vitro experimental data and in vivo diseases. Where In vitro experimental data can quamtify the effect of a gene knockout on specific aspect of a cellular phenotype in a petri dish, in vivo interactions between the drug and the organism may lead to weaker effect sizes or toxicity. The drug discovery pipeline consists of stages that start by testing a set of candidate interventions and then procedes by selecting a subset of promising candidates to pass on for further development. For example, one might test a broad range of gene knockouts on cell cultures and then select a subset to evaluate in animal models. These trials can be expensive, so it is desirable to weed out potentially ineffective or toxic candidates before this phase. To do so, researchers can leverage heuristic score functions that predict the "drug-like-ness" or likelihood of toxicity of a compound (Jiménez-Lung et al., 2020). Considering a diverse set of candidate interventions, where each intervention applies to a different mechanism in the disease phenotype, is also of use because it increases the likelihood of at least one candidate succeeding in the subsequent phase.
We formalize this problem an optimization problem where the optimizer has access to measure- ment correlated with the quantity of interest; however, it is noise augmented to emulate the primary objective function. We formalize our search space (i.e., the set of available genes, though in princi- ple this could be any set) g = (g.....,ge gm J. for which we have some phenotype measurement fip We will primarily refer Sip function from features to phenotype changes, but it is equivalent
3

---

Under review as a conference paper at ICLR 2023
to expressing Sip a function on genes G. The subscript 'ip' stands for intermediate phenotype as it is not the actual clinical measurement caused by the gene knockout. Instead, it is a measure- ment known to correlate with disease pathology and is tractable in the lab setting (see Appendix B for detailed formalization). In this paper, we will assume the phenotype change is a real number fip(x) E RR; however, given suitable modeling assumptions, it is possible extend our approach to vector-valued phenotype readouts. We also define a function called disease outcome, fout, which composed of Sip and factors outside the biological pathway, such as toxicity of a molecule that engages with a target gene. The noise component, 1), encapsulates all these extra factors.
We consider two tractable formulations of the relationship between the disease outcome, fouts and the in vitro phenotype, fip.
1. Multiplicative Bernoulli noise:
four fip(x))((x)
where (xx E (0, E G. and 1) is sampled from Gaussian process classification model. This setting presents a simplified model of drug toxicity: 17 corresponds to a binary indicator of whether or not the drug is revealed to exhibit unwanted side effects in future trials. The multiplicative noise model assumes that the downstream performance of an the
intervention is monotone with respect its effect on the phenotype, conditional compound not exhibiting toxicity in future trials. In our experiments, we assume exhibits correlation structure over inputs corresponding GP classification model, and construct the kernel Kx of this GP to depend on some notion of distance in the embedding space X.
2. Additive Gaussian noise:
fxu((x:)) m7) fi)(x) + 7xx) 17 ~~~ GP(O, Kx)
where 17 G R is drawn from Gaussian process model with kernel Kx. In this case, we assume that the unforeseen effects of the input x are sufficiently numerous to resemble Gaussian perturbation of the measured in vitro phenotype (x).
Notice that in the above models. noise is an umbrella term for everything that affects the fitness of a target but is not part of the biological pathway from the gene to the phenotype change. Therefore, the choice of noise distribution and how it affects the outcome is modelling assumption that is intended to capture coarse inductive biases known to the researcher. We additionally seek out a set of interventions S G of some fixed size [S] A whose elements cause the maximum expected change (for some noise distribution) in the disease outcome. In other words, we seek an interven- tion that best moves the disease phenotype, which will be the best candidate drug. This goal is distinct from either sampling the super-level-sets of fip or finding the set S with the best average performance. Instead, we explicitly seek to identify a set of points whose toxicity unintended side effects will be minimally correlated, maximizing the odds that at least one will succeed in the subsequent trials. We thus obtain set-valued maximization problem
max E. max fou (x: 17) SCX
(4)
XES
This compact formula is critical to attain our overarching objective: identifying interventions with both a large impact on the phenotype of interest and with high diversity to increase the chance of success of some of them in the subsequent steps of the drug discovery pipeline. An illustrative example is provided Figure
the Appendix to provide further intuition into this formula.
The general formulation of this problem is NP-hard (Goel et al., 2010); therefore, we propose tractable algorithm that provides constant-factor approximation of the optimal solution by lever- aging the submodular structure of the objective under suitable modeling assumptions. Given such an algorithm, our task the active learning problem of optimally querying the function, fip- given a limited number of trials, T, to accurately estimate the algorithm's 's output on the ground-truth dataset.
Importantly, this formulation allows us to decouple modeling the measured phenotype, fip- from modeling the noise n. For example, we might make the modeling assumption that we sample fip from a GP with some kernel k and that y is a Bernoulli random variable indicating the safety of the compound.
4

---

Under review as a conference paper at ICLR 2023
4 METHOD
Various methods exist for efficiently optimizing black-box functions; however, problem setting violates several assumptions underlying these approaches. In particular, while we assume access to intermediate readouts Sip- the actual optimization target of interest four observable. Further, seek to find a sel of interventions that maximize its expected value under some modeling assump- tions. These two properties render a broad range of prior art inapplicable. Active sampling methods do not prioritize high-value regions of the input space. Bayesian optimization methods assume ac- cess to the ground-truth function outputs (or noisy observation thereof). And Bayesian algorithm execution approaches based on level-set sampling may not sufficiently decorrelate the hidden noise the outcome.
We propose intervention set selection algorithm in Bayesian algorithm execution procedure that leverages the modeling assumptions we characterize in the previous section. This method, Sub- set Discovery via Bayesian Algorithm Execution (DiscoBAX), consists of two distinct parts. (1) subset-selection algorithm obtaining a 1-/ee-factor approximation tion 3, and (2) an outer BAX loop that queries the phenotype readings to maximize the information gain about the output of this algorithm. In Section 4.1, we present the idealized form of DiscoBAX and show that it attains approximately optimal solution. Our approach is easily adaptable in- corporate approximate posterior sampling methods, enabling its use with deep neural networks on high-dimensional datasets. We outline this practical implementation in Section 4.2.
the set that optimizes equa-
4.1 ALGORITHM
Subset maximization: we first address the problem of identifying subset S X which maximizes the value E,, Imaxxes (x; x; 7)]] As mentioned previously, the exact maximization of this objective intractable. To construct tractable approximation, under which the value of an intervention is bounded by zero four (x; max(foul 17), 0). This choice is motivated by the intuition that any intervention with negative expected value the phenotype is equally useless as it will not be considered in later experiment iterations, and so do not need distinguish between harmful interventions. The resulting function S(S)
propose submodular surrogate objective,
E,, [maxxxs fäu (x; 1)] will be submodular, and thus Algorithm 1, the greedy algorithm, will provide al 1/e approximation of the optimal solution.
Observation The score function f P(G) + IR defined by
f(S)
En XES
max
max((,, four fou 1)
(5)
is submodular.
We provide proof of this result in Appendix A. In practice, we can estimate the expected value in this objective using Monte Carlo (MC) samples over the noise distribution 77. Where MC sampling is too expensive, heuristic that uses threshold to remove points whose values under are too highly correlated can also obtain comparable results with reduced computational burden.
Algorithm 1 SubsetSelect (Multiplicative Noise)
Algorithm 2 DiscoBAX
Require: integer > 0, set X, distribution
Require: finite sample set X, budget T, Monte Carlo parameter & E N D
P(n), sampled Fip X >
- 0
$+
Saunt (x: xx == (x))(()x
for i T do
sample 11
P(fip
for i < < do
S, <- Subseet SubsetSelect
Vj ,,...,
x, <- arg max EIG "G,"(X()') (x,
- SU (arg max Eyl max (x;
=EXiS yESU(a)
query (x;(
D D=DL ((xi, (in(x:)
end for return
end for return
5

---

Under review as a conference paper at ICLR 2023
Active sampling: because do not assume prior knowledge of the phenotype function fip. require a means of selecting potential interventions for querying its value at a specified input x. In practice, running these experiments may incur cost, and so is desirable to minimize the number of queries necessary to obtain an accurate estimate of the optimal intervention set. BAX (Neiswanger al., 2021) presents rithm using a minimal number of queries to the dataset of interest. In our setting, this allows us to approximate the output of Algorithm over the set (X, fip(t))) without incurring the cost of evalu- ating the effect of every knockout intervention in G. Concretely, this procedure takes as input some probabilistic model P which defines distribution over phenotype readings fip conditioned on the data D, seen so far and from which it is possible to draw samples.
effective active sampling approach approximate the output of an algo-
remark on the efficiency of subset maximization & active sampling- It has to be emphasized that subset selection is a function called within each active sampling cycle. Hence, the above observation about submodularity refers specifically to Algorithm 1 rather than its incorporation If sample efficiency is not concern this algorithm could be run on the set of all inputs and provide the exact solution.
Algorithm 2.
We outline this procedure in Algorithm 2, and refer to Section 2 for additional details. In the batch acquisition setting, we form batches of size B at each cycle by selecting the B points with the highest EIG values.
4.2 PRACTICAL IMPLEMENTATION IN HIGH DIMENSIONS
When working with high-dimensional input features, we typically leverage Bayesian Neural Net- works in lieu of Gaussian Processes. We sample from the parameter distribution via Monte Carlo dropout (MCD) (Gal & Ghahramani, 2016), and rely on Monte Carlo simulation to estimate the quantities introduced in Algorithm 2. In particular, the entropy of the posterior distribution is ob- tained as follows:
M
H Myx = Epty, D. [log p(yx (Di) ~
Σ 2=1
log P(y, fa)
(6)
M
where the samples {v = [ are obtained by sampling from the distribution over model parameters with MCD to obtain the parameter samples {
EXPERIMENTS
In the experimental evaluation of DiscoBAX, specifically seek to answer the following questions: 1) Does DiscoBAX allow us to reach a better trade-off between recovery of the top interventions and their diversity (Table and 2)? 2) Is the method sample-efficient, i.e., identifies global optima in fewer experiments relative to random sampling or naive optimization baselines (Figure and 5)? 3) Is the performance of DiscoBAX sensitive to various hyperparameter choices (Appendix D.3)? To address these questions, we first focus on experiments involving synthetic datasets ($ 5.1) in which we know the underlying ground truth objective function. We then conduct experiments across several large-scale experimental assays from the GeneDisco benchmark Mehrjou et al. (2021) that cover a diverse set of disease phenotypes.
5.1 SYNTHETIC DATA
We begin with a concrete example to illustrate the distinction between the behavior DiscoBAX and existing methods. The dataset we consider is a one-dimensional regression task on a mixture-of- Gaussians density function fmog. We construct fmog such that it exhibits several local optima at a variety of values, necessitating a careful trade-off between exploration and exploitation to opti- mize the DiscoBAX objective. Crucially, exploitation in this setting requires not only an accurate estimation of the global optimum but also an accurate estimation of the local optima. We provide evaluations additional datasets in Appendix D.I. We consider the following baseline acquisi- tion functions which select the optimal point x" to query at each iteration, letting μu()) denote the posterior mean over fx)(x) and (x) its variance. We evaluate random sampling, UCB-like acqui- sition function, BAX on super-level set and top-k algorithms, Thompson sampling, and uncertainty maximization baselines. Full details are provided in Appendix D.

---

Under review as a conference paper at ICLR 2023
Mixture Geussians Illustration
Activn Sampling
Thomoson Semsling
LCC
Discnffex (ours)
MMM
30
Score 0.412 000
Score 0.395 0.003
Score 0.390 3.003
Score 0421 D.033
-422
-23
-25
BAX
Lovelsot' BAX
Lovelcetto.6 BAX
Ground Truth
mmm
8 30
00
0.399 0.003
0.403 0.003
0.403 3.023
Score 0.424 0.003
Figure 2: Illustration of failure modes of benchmark acquisition functions in our problem setting: existing methods struggle accurately capture both the high- and low-valued local optima. We use a batch size equal to one for all methods.
In Figure 2, we visualize the solutions found by each approach after 30 iterations. We further evaluate the score of each method, computed as E,, maxxes (ip(()))(() (x)))x), where rj is drawn from a Bernoulli distribution whose logits determined by an affine transformation of a sample from GP with zero mean and radial basis function covariance kernel. This construction ensures a high correlation between the values of nearby inputs and reward sets S whose elements are distant from each other. To select S, we use the learned posterior mean je from each acquisition strategy as input to Algorithm and set S be equal its output. We observe that most baselines over-exploit the high-value local optima, leading to inaccuracies on the lower optima. As a result, Algorithm unable select the optimal subset elements from the lower-value modes and the model score suffers. The active sampling baseline yields more uniform sampling distribution over inputs that results in a relatively uniform distribution of errors. While DiscoBAX does not perfectly estimate the value of the target function, its sampling strategy yields reasonably accurate estimates of all of the local optima.
5.2 GENEDISCO DATASET
Datasets & baselines. The GeneDisco benchmark (Mehrjou et al., 2021) is comprised of five large-scale gedome-wide CRISPR assays and compares the relative strengths of nine active learning algorithms (eg., Margin sampling, Coreset) for optimal experimental design. The objective of the different methods is to select the set of interventions (ie., genetic knockouts) with the largest impact on the corresponding disease phenotype. We include all existing baselines from the GeneDisco benchmark, as well as eight additional approaches: UCB, qUCB, qEI, gPOI, Thompson sampling, Top-K BAX, Levelset BAX, and DiscoBAX.
Metries & approach. We define the of optimal interventions the ones in the top percentile of the siiiinennallyymmauueee phenotype (referred to as 'Top-K interventions'). We use the Top- K recall metric assess the ability of the different methods identify the best interventions. To quantify the diversity across the set of optimal interventions, we first cluster these interventions in a lower-dimensional subspace (details provided in Appendix C). We then measure the proportion selected) by given algorithm over
of these clusters that are recalled (i.c., any of its members the different experiment cycles. The overall score of an approach is defined as the geometric mean between Top-K recall and the diversity metric. For all methods and datasets, we perform 25 con- secutive batch acquisition cycles (with batch size 32). All experiments are repeated 10 times with different random seeds.
Results & discussion. We observe that, across the different datasets, DiscoBAX enables to identify more diverse set of optimal interventions relative baselines (Table 1). It does so in sample-
7

---

Under review as a conference paper at ICLR 2023
Interferon Gamma
Interferon Gamma
12%
60%
Cumulative T recall
10%
50%
Diversity
8%
40%
8%
30%
20%
10%
0%
0%
Acquisition methods
10
15
20
25
0
10
15
20
25
DaseBAX
Active learning cycles
Active learning cycless
Top
Levelset
Leukemia cells
Leukemia NK cells
UCS
50%
Thempson samp
6%
10%
Almulative Diversity
30%
4%
Cumulstive
20%
10%
0%
Active learning cycles
Active learning cycles
Figure 3: Top-K recall and Diversity score Vs acquisition cycles The two top plots are for the Interferon y assay (Schmidt et al., 2021), and the two bottom plots are based on the Leukemia assay (Zhuang et al., 2019).
efficient manner as it achieves higher diversity throughout the different acquisition cycles (Fig.3). Note that sample-effiiencyy is empirical observation here not theoretical property of the al- gorithm since possible to construct adversarial datasets where BAX method will attain no better performance than random sampling. Interestingly, it tends to recall higher share of optimal interventions on several assays as well, which may be the result of very steep extrema in the cor- responding datasets. We also find the performance of DiscoBAX to be relatively insensitive to the choice of hyperparameters (Appendix D.3). Lastly, we note that when the input feature space (ic., the intervention representation) does not correlate much with the disease phenotype of interest, the model heing learned tends to perform poorly and we observe no lift between the different methods and random sampling (eg., the SARS-CoV-2 assay from Zhu et al. (2021) Appendix D.2).
Table 1: Performance comparison on GeneDisco CRISPR assays We report the aggregated per- formance of DiscoBAX and other methods on all assays from the GeneDisco benchmark. All other baselines and the breakdown per assay are provided in Appendix D.2.
Method
Category
Top-K recall
Diversity score
Overall score
Random
29.3% (1.4%)
4.9% (0.3%)
12.0% (0.6%)
Thompson Sampling
Bandits
27.5% (1.5%)
4.8% (0.4%
11.5% (0.7%)
UCB
Bayesian Optim.
33.5% (2.0%)
5.9% (0.5%
14.1% (1.0%)
Coreset
Active learning
39.3% (1.9%)
5.5% (0.3%)
14.7% (0.8%)
Levelset Bax
BAX
35.4% (2.2%)
6.3% (0.4%)
15.0% (0.9%)
Top-K Bax
BAX
38.8% (2.3%)
6.8% (0.6%)
16.2% (1.2%)
DiscoBax (ours)
BAX
44.1% (2.2%)
7.8% (0.5%)
18.6% (1.1%)
8

---

Under review as a conference paper at ICLR 2023
6 RELATED WORK
Prior works have studied the application of genomic discovery and method development for diverse larget generation.
Bayesian optimization: Bayesian optimization (BO) is concerned with finding the global opti- mum of a function with the fewest number of function evaluations (Snock et al., 2012; Shahriari et al., 2015). Since this target function is often preensive-o-evaluutee, one typically uses a Gaussian process as a surrogate function (Srinivas al.). The candidates for function evaluation are then determined through so-called acquisition function, which is often expressed as the expected util- ity over the surrogate model. Typical choices include the expected improvement (Močkus, 1975, El) and probability of improvement (Kushner, 1964, PI) utility functions. Recent work includes variational approaches Song et al. (2022) which yield a tractable acquisition function whose limit- ing behavior is equivalent to PI. Prior work tried to obtain diversity in Bayesian optimization e.g. through a batch setting (Kirsch et al., 2019) or oujti-obeecctiv optimization (Hernández- al., 2016). Bayesian optimization has been applied biological problem settings such as small molecule optimization (Korovina et al., 2020) or automatic chemical design (Griffiths & Hernández- Lobato, 2017).
Optimal experiment design broadens the scope of Bayesian Optimization: rather than simply max- imizing& a parametric function, the task is adaptively identify an optimal set of experiments to efficiently reach some goal (Robbins, 1952: Chernoff, 1959). Applying machine learning to auto- mate hypothesis generation and testing goes back multiple decades (King et al., 2004). Optimal experiment design is amenable to Bayesian optimization (Greenhill et al., 2020) and reinforcement learning approaches (Kandasamy et al., 2019). Most related to our work is Bayesian Algorithm Exe- cution (BAX) Neiswanger et al. (2021) that extends the goal of experiment design from only finding the maximum of a function to estimating more general properties such as level sets by computing the expected information gain (EIG) which is the mutual information between the evaluation of an input point and the statistics related that property.
Active learning While many probabilistic models like Gaussian processes provide principled un- certainty estimates (Rasmussen, 2003), modern neural network architectures often rely on heuristics only provide approximations approaches (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017). Active learning based approaches use the uncertainty estimates for maximizing expected information gains of model parameters (Houlsby et al., 201 1). Recently, more and more approaches have used active learning based on model uncertainties of neural networks for biomedical applica- tions.
The upper confidence bounds seen in BO originate in the bandit setting (Lai & Robbins, 1985), which one can extend the widely-used UCB algorithm to Gaussian processes (Grünewälder et al., 2010: Srinivas et al.). While both bandits and BO seek to find the maximum of a function, the two problem settings leverage different notions of optimality. BO seeks identify the argmax, whereas bandits seek to minimize the number of sub-optimal queries. Related to bandits and BO, some efforts are made to formulate active learning as a reinforcement learning problem (Slade & Branson, 2022; Casanova et al., 2020; Konyushkova et al., 2017; Pang et al., 2018).
Bandits:
7 CONCLUSION
We have introduced a mathematical formalization of the drug discovery problem that captures the noise induced by moving from in vitro to in vivo experiments. We proposed novel algorithm based on Bayesian Algorithm Execution and illustrated its utility on many illustrative synthetic datasets. We have further evaluated this class of methods against the real-world large-scale assays from the GeneDisco benchmark, where they help identify diverse top interventions better than existing base- lines. Future work could see the extension of the current framework to explicitly account for the fact that experimental cycles happen in batches. Further, we assume in this work that distant rep- resentations of interventions implied different underlying biological mechanisms proper causal formulation of the problem would allow us to tell apart causally connected pathways more cleanly. Finally, it is typical practice to measure several potential intermediate phenotypes of interest to capture different aspects of interest, which requires an extension of our approach to the setting of multiple objectives.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
We clearly state our modelling assumptions throughout Sections 2 to 4. We provide proof for our theoretical claims in Appendix A. All experimental results reported in Section and appendix D can be reproduced using the code available at: https: / sttps://giis /giihub. com/ antn/mous33m solaris 20-22023-250 -iclr. Hyper-parameter sweeps for the BAX methods for GeneDisco are pre- sented in Table 3.
REFERENCES
Nurken Berdigaliyev and Mohamad Aljofan. An overview of drug discovery and development. Future Medicinal Chemistry, 12(10):939 947, 2020.
Arantxa Casanova, Pedro O. Pinheiro, Negar Rostamzadeh, and Christopher J. Pal. Reinforced ac- tive learning for image segmentation. In International Conference on Learning Representations, 2020. URL https:tpp........ : //pp///eeeewww..oo net/ wtt.://onmmmmmnoooooaaea
Lia Chappell, Andrew JC Russell, and Thierry Voet. Single-cell (multi) omics technologies. Annual Review of Genomics and Human Genetics, 19:15-41, 2018.
Henman Chernoff. Sequential design of experiments. The Annals of Mathematical Statistics, 30(3): 755-770, 1959.
Joshua M. Dempster, Jordan Rossen, Mariya Kazachkova, Joshua Pan, Guillaume Kugener, David E. Root, and Aviad Tshemiakk Extracting biological insights from the project achilles genome- scale crispr screens in cancer cell lines. bioRxiv, 2019. doi: 10.1101/720243. URL https:
//www. biorxiv. org/ wontent/earlyyye.e /20l1/221111 2019/001/00000 00/0/333/33 /211331//22233
Michael Dickson and Jean Paul Gagnon. Key factors in the rising cost of new drug discovery and development. Nature Reviews Drug Discovery, 355):417-429 2004.
Michael Dickson and Jean Paul Gagnon. The cost of new drug discovery and development. Discov- ery Medicine, 4(22):172-179, 2009.
Joseph DiMasi, Henry Grabowski, and Ronald W Hansen. Innovation the pharmaceutical industry: new estimates of r&d costs. Journal of Health Economics, 47:20-33, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Ashish Goel, Sudipto Guha, and Kamesh Munagala. How to probe for extreme value. ACM Trans. Algorithms,, 7(1), dec 2010. ISSN 1549-6325. doi: 10.1145/1882237.1862550 URL
https:/ htns:://doi.rggg /1gg 11445/5888277.8822288
Stewart Greenhill, Santu Rana, Sunil Gupta, Pratibha Vellanki, and Svetha Venkatesh. Bayesian optimization for adaptive experimental design: A review. IEEE Access, 1:13977-13948, 2020.
doi: 10.1109/ACCES. 20200.966222
Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design. arXiv preprint 1709:::779.0555000 2017.
Steffen Grünewälder, Jean-Y Yes Audibert, Manfred Opper, and John Shawe-Taylor. Regret bounds for gaussian process bandit problems. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 273-280. JMLR Workshop and Conference Proceedings, 2010.
Ychudit Hasin, Marcus Seldin, and Aldons Lusis. Multi-omics approaches to discase. Genome Biology, 18(1):1-15, 2017.
Daniel Hernández-Lood Lobato, Jose Hernandez-Lobate Amar Shah, and Ryan Adams. Predictive tropy search for multi-objective bayesian optimization. In International conference on machine learning, pp. 1492-1501. PMLR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: arXiv:1112.5745, 2011.
Ronen Ben Jehuda, Yuval Shemer, and Ofer Binah. Genome editing in induced pluripotent stem cells using crispr/cas9. Stem Cell Reviews and Reports, 14(3):323-336, 2018.
José Jiménez-Luna, Francesca Grisoni, and Gisbert Schneider. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence, 2(10):573-584, 2020.
Kirthevasan Kandasamy, Willie Neiswanger, Reed Zhang, Akshay Krishnamurthy, Jeff Schneider, and Barnabas Poczos. Myopic posterior sampling for adaptive goal oriented design of exper- iments. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3222-3232. PMLR, 09-15 Jun 2019.
Ross King, Kenneth Whelan, Ffion M Jones, Philip GK Reiser, Christopher H Bryant, Stephen H Muggleton, Douglas B Kell, and Stephen G Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427(6971):2477-22 2004.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Ad- vances neural information processing systems, 30, 2017.
Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing- Chembo: Bayesian optimization of small organic molecules with synthesizable recommendations. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings Machine Learning Research, pp. 3393-3403. PMLR, 26-28 Aug 2020. URL https: http:///occeeeenngcceeee mlr press / vv00//00/vv ww10//krrvviiiio www.oooooinaaanoo html.
H. Kushner. new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86(1):97-106, mar 1964. doi: 10.1115/1. 3653121. URL https: //tpi. www.aii.ogggagaooo wtt.i/or.//...112/.1111 11155221.3363122..
T.L Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4-22, 1985. ISSN 0196-8858. doi: http::/doi.org//
101601196-88588339900002 URL
httpe/eeeeiieet.eeeeeeeeeeeeeeee com/science/
axticleeppi/111189555550000
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Adam I.. Macl_en, Tian Hong, and Qing Nie. Exploring intermediate cell states through the lens of single cells. Current Opinion in Systems Biology, 9:32-41, 2018. ISSN 2452-3100, doi: wotp.///......./...oooo...boobb accienne/eertieeee/////222222220
URL https:/ www. sciencedirect com/ Mathematic Modelling.
Arash Mehrjou, Ashkan Soleymani, Andrew Jesson, Pascal Notin, Yarin Gal, Stefan Bauer, and Patrick Schwab. Genedisco: A benchmark for experimental design in drug discovery. arXiv preprint arXiv:21 2021.
Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization techniques IFIP technical conference, pp. 400-404. Springer, 1975.
Willie Neiswanger, Ke Alexander Wang, and Stefano Ermon. Bayesian algorithm execution: Esti- mating computable properties of black-box functions using mutual information. In International Conference Machine Learning, pp. 8005-8015. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng-Hao Liu, Maksym Korablyov, Michael M Bronstein, and Yoshua Bengio. Evaluating generalization in gflownets for molecule design. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy Hospedales. Meta-learning transferable active learning policies by deep reinforcement learning. arXiv preprint ht0b::8006.47999 2018.
Mihaela Pertea, Alaina Shumate, Geo Pertea, Ales Varabyou, Florian P Breitwieser, Yu-Chi Chang, Anil Madugundu, Akhilesh Pandey, and Steven L Salzberg. Chess: a new human gene catalog curated from thousands of large-scale ma sequencing experiments reveals extensive transcrip- tional noise. Genome Biology. 19(1): 2018.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.
Carlos G Sanchez, Christopher M Acker, Audrey Gray, Malini Varadarajan, Cheng Song, Nadire R Cochran, Steven Paula, Alicia Lindeman, Shaojian An, Gregory McAllister, et al. Genome-wide crispr screen identifies protein pathways modulating tau protein levels in neurons. Communica- tions biology, 4(1):1-14, 2021.
Ralf Schmidt, Zachary Steinhart, Madeline Layeghi, Jacob W. Freimer, Vinh Q. Nguyen, Franziska Blaeschke, and Alexander Marson. Crispr activation and interference screens in primary human t cells decode cytokine regulation. bioRxiv, 2021. doi: 10. 10.1101//2000 20222 05 11. 443701.
htts::::
//www. httww..birrxte bww.riiv.iiv........... org/
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
Emma Slade and Kim M Branson. Deep reinforced active learning for multi-class image classifica- tion. arXiv preprint arXiv:2206. 13391, 2022.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Jiaming Song, Lantao Yu, Willie Neiswanger, and Stefano Ermon. A general recipe for likelihood- free Bayesian optimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 Proceedings of Machine Learning Research, pp. 20384-20404. PMLR, 17-23 Jul 2022. URL https: http///ecccceeeeeeeeeeeeepo mlr. pwwessvvll vvlss//vvlee///// / .html.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimiza- tion in the bandit setting: No regret and experimental design.
Cole Trapnell. Defining cell types and states with single-cell genomics. Genome Research, 25(10): 1491-1498, 2015.
Thomas Worzfeld, Elke Pogge von Strandmann, Magdalena Huber, Till Adhikary, Uwe Wagner, Silke Reinartz, and Rolf Müller. The unique molecular and cellular microenvironment of ovarian cancer. Frontiers in Oncology, 7:24, 2017.
Yunkai Zhu, Fei Feng, Gaowei Hu, Yuyan Wang, Yin Yu, Yuanfei Zhu, Wei Xu, Xia Cai, Zhiping Sun, Wendong Han, al. A gide crispr screen identifies host factors that regulate sars- cov-2 entry. Nature communications, 12(1):1-11, 2021.
Xiaoxuan Zhuang, Daniel P. Veltri, and Eric O. Long. Genome-wide crispr screen reveals cancer cell resistance to nk cells induced by nk-derived ifn-y. Frontiers in Immunology, 10:2879, 2019. ISSN 1664-3224. doi: 10. 10.3338//mmm..200 2019..mm....1........ 02879. URL https: https://wwww...e Fro.iierniii.ii...
oxg/article/100 3 fimmu. .2019, 02879.
12


Reviews:
[{"review_id": "dr0GXZ4RVma", "reviewer": "Reviewer_JiqB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with the iterative selection of an optimal set of targets for genetic interventions based on a scalar readout. The methods section described a particular instance of BAX, that models the singularity of the biological problem that is considered. A few aspects that makes it different from the existing InfoBAX approach is (1) the subset maximization target function and (2) a practical uncertainty model with BNN instead of GPs. Then, the paper provides a set of experiments on synthetic data, and on the openly available GeneDisco benchmark. ", "summary_of_the_review": "This paper provides an interesting application of BAX, and investigate a practical variant to batch learning of genetic perturbations. Beyond this, the technical contributions may be considered weak, and the empirical results are not exactly strong either (please refer to my previous sections for actionable items). Regarding the experimental results, I completely understand that it might be caused by noise in some datasets (e.g., COVID). In this case, I do think that the authors should carefully present ablation studies so that readers can understand what features of DiscoBAX helps in increase performance compared to BAX (again, see above for suggestions). In the light of this, I am currently leaning towards rejection of this manuscript, but very much looking forward to discussions with authors, and other reviewers.", "strength_and_weaknesses": "Strength:\n(1) Novelty *in its application*: this is one of the (relatively) few papers that tackle this important scientific problem (other example I could find is https://arxiv.org/pdf/2207.12805.pdf).\n(2) The paper shows improved results in the recent GeneDisco benchmark, for a few of the datasets.\n\nWeaknesses:\n(1) My main issue is that I am having a hard time understanding the *technical* novelty of the paper. A lot of the concepts explained in Section 4 seems to belong more to a background section, as they are not contributions of this particular work, but from the BAX paper. As far as I understand, the contributions are (a) working with sets, and performing greedy set optimization and (b) using BNN for uncertainty quantification. However, in the results section, I don't see a clear explanation that those changes are improving the performance. It seems that (a) improves diversity? (if so, it should be made clear), but (b) has no comparison to different flavors of BNNs, GPs etc..\n(2) there is no theoretical analysis on how the (1 - 1/e) approximation error propagates through the BAX procedure. Then, it is not correct for the author to write \"theoretical guarantees on the optimality of the approach\". Similarly, the authors write that DiscoBAX is \"sample-efficient\", but I have seen no proofs of sample efficiency, and the experimental results are not necessarily strong enough to claim this.\n(3) There is some improvement over BAX, but only in a few datasets (2 / 4). \n(4) There should be a comparison to other BNN flavors, and at the very least a discussion of uncertainty modeling w/ neural networks\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rb4R8-gV2xJ", "reviewer": "Reviewer_3hGy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a new probabilistic algorithm (\"DiscoBAX\") for subset selection that aims to approximately optimize phenotype movement in genomic intervention and can be useful in drug discovery tasks according to the authors. The method identifies a set of interventions whose elements will trigger maximum expected change in the disease outcome with respect to for some noise distribution. Performance is assessed based on synthetic data as well as public benchmarking data, and the algorithm is shown to outperform alternative approaches.\n\n", "summary_of_the_review": "The work provides an interesting approach to an optimization problem that is initially motivated by certain genomic drug discovery tasks. Overall the reporting is clear and appears reproducible. The algorithmic innovation is in place but focuses on a relatively specific task definition where the applied motivation remains questionable, and it is not clear whether overfitting has been sufficiently controlled as the algorithm appears rather flexible. The paper has figure numbering problems. \n\n\n", "strength_and_weaknesses": "Strengths:\n+ probabilistic approach is neat for uncertainty characterization\n+ practical combination of task, method derivation, and algorithm implementation \n+ Openly licensed source code for the algorithms available\n\nWeaknesses:\n- Figure numberings are broken, text cites figure up to Fig.5, manuscript only has 3 figures, refs to Figs 3-4 are missing; not clear how to follow\n- The problem motivation is hypothetical; it is not clear that \"maximizing change\" is what one would like to achieve in genomic drug discovery; on the other hand the algorithm is of interest regardless but perhaps best evaluated on its own right as a target optimization task.\n- I am not sure if potential overfitting is sufficiently addressed in this work; explanation on this part could be strengthened ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C2XkF7OOJF", "reviewer": "Reviewer_efcP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of designing an experiment where there are two stages: in the first (in vitro) stage our task is to (efficiently) design an experiment that will have good chance of success in the second (in vivo) stage. The authors formalize this elegantly in equation (3).\n\nThe proposed solution is discoBAX: an algorithm that actively searches the in-vitro function for a set of optimal yet diverse points that have the best shot at having a successful outcome in the in vivo stage. \n\nThe paper explores discoBAX on a synthetic and real-world dataset.\n\n", "summary_of_the_review": "A really interesting setting to be working in, and what seems like a good idea, but a lack of empirical investigation into the method lets the paper down. Clarity is a problem and it's difficult to assess which parts of the work are a novelty. I could change my \"technical novelty' score if the authors could clarify.", "strength_and_weaknesses": "Strengths\n--\nAfter spending some time with the paper, I came to appreciate the formalisation of the setting that is being worked in. I particularly enjoyed section 3 which laid out the problem. I appreciate the authors use of the appendices to keep long proofs out of the paper (the submodular proof for example) and also the clear presentation of the algorithms used. Posting the code anonymously gives me some confidence that the work is sound. \n\nWeaknesses\n--\nthe paper fails to deliver what is promised at the start of the experimental section - an investigation of discoBAX's sample efficiency, sensitivity to hyper-parameter settings, and comparison with the top-intervention method. As a result I feel that the paper is really rather unfinished! I am keen to understand more about discoBAX and am disappointed that the investigation is not deeper. \n\nThe 'related work' section at the back does not cover BAX - which appear to be prior art? this section feels like filler, I would have preferred the space be used to investigate discoBAX further. \n\nI did not find a satisfactory discussion (or empirical evaluation) of the use of the relu'd score function, relative to the original objective. I get that this makes the optimization submodular, but have we lost anything? In which cases does this cause a problem?\n\nIt took me some time to work out what was going on - I think more discussion around Figure 1 is needed, and perhaps some commentary on how this is connected to the invitro/in-vivo setting. I liked the clarity of eq 3, but this felt like a long time coming, and would have been better placed at the start of the paper perhaps?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A0TK4ZgnQK", "reviewer": "Reviewer_uzVy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of finding a `diverse` set of genomic interventions that maximize a phenotype of interest. Formally, the problem can be modeled as optimizing expensive black-box functions over sets of inputs from a given design space. Bayesian Algorithm eXecution (BAX) is a recently proposed framework that allows estimating required properties of black-box functions using information gain based input acquisition strategies. The paper provides a BAX (Bayesian `Algorithm` eXecution) style approach to solve the problem where the key idea is to consider the subset maximization of a chosen score function as the `Algorithm`. At each iteration, an Expected Information Gain acquisition objective is optimized to select the next points for evaluation. This objective is parameterized by multiple sample outputs of the subset maximization algorithm. Experiments are performed on synthetic and GeneDisco benchmarks.", "summary_of_the_review": "I find the overall approach to be quite interesting while addressing an important problem. However, some of the design choices and baseline comparisons require more work to improve the quality of the paper for an ICLR publication. ", "strength_and_weaknesses": "- The problem considered in the paper is important with real-world implications. I found the biological motivation of the problem well-written with good intuitions for a non-domain person.\n\n- The overall idea of using BAX style approach for this problem setting is fairly interesting and novel. \n\nHowever, I have few questions  to understand some of the details better and some suggestions that will hopefully improve the paper's contributions:\n\n- It is mentioned in the paper that the discrepancy between the disease outcome $f_{out}$ and intermediate phenotype $f_{ip}$ is captured by the noise distribution $\\eta$. It is motivated as an important distinguishing factor between the problem setting of this paper compared to that of existing approaches like Bayesian optimization. However, there is little clear description about estimating this quantity or principles behind choosing a certain distribution. A short remark is mentioned after observation 1 in few lines but that seems limited for such an important motivation of the problem setting. Please describe the concrete implementation/algorithmic details of the noise distribution and how it is estimated in the experiments. \n\n- The proof of submodularity of the score function in appendix A is missing some description. For example, $dis(g, \\eta)$ is not defined. Please expand the proof and explain the reasoning behind the inequalities. If it is `straightforward` (as mentioned in the first line of the proof), can it be considered a major contribution of the paper? \n\n- The score function is chosen to be the best value in a set (averaged over noise distribution eta). This seems like an optimistic choice. For the algorithm to find good robust points, should we not consider the worst value in the set as the score function?\n\n- Please add a description of the computational complexity of the proposed approach. Since multiple instances of the subset maximization algorithm needs to be run in each iteration for estimating the expected information gain quantity, the computational complexity can be quite high. A wall-clock time comparison of the proposed approach with existing baselines will be very useful, especially since the top-k recall gap is relatively small on most of the benchmarks (other than Leukemia/NK cells).\n\n- As mentioned in section 5.2, batch size of 32 points are selected for evaluation in each acquisition cycle. How is the batch of points selected? Is it accomplished by greedy optimization of the EIG acquisition function? If yes, what are the accuracy losses of doing this greedy optimization. \n\n- The choice of UCB as Bayesian Optimization (BO) baseline strategy seems surprising. There is a large literature on Bayesian optimization algorithms (please see [1-4] and references therein) for the batch evaluation setting that should be the right comparison here. Something like qEI (q-Expected Improvement) is easy to setup and implemented in popular packages like BoTorch [1] (https://botorch.org/tutorials/). The points suggested by qEI are also known to find highly diverse points in the context of batch optimization. Please consider improving this comparison (by including qEI for instance) because BO algorithms are directly applicable and the most relevant baseline in the setting. \n\nReferences \n\n\n[1] Balandat, Maximilian, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G. Wilson, and Eytan Bakshy. \"BoTorch: a framework for efficient Monte-Carlo Bayesian optimization.\" Advances in neural information processing systems 33 (2020): 21524-21538.\nGonzález, J., Dai, Z., Hennig, P., & Lawrence, N. (2016, May). Batch Bayesian optimization via local penalization. In Artificial intelligence and statistics (pp. 648-657). PMLR.\n\n[2] Wu, J., & Frazier, P. (2016). The parallel knowledge gradient method for batch Bayesian optimization. Advances in neural information processing systems, 29.\n\n[3] Azimi, J., Fern, A., & Fern, X. (2010). Batch bayesian optimization via simulation matching. Advances in Neural Information Processing Systems, 23.\n\n[4] Gong, C., Peng, J., & Liu, Q. (2019, May). Quantile stein variational gradient descent for batch Bayesian optimization. In International Conference on Machine Learning (pp. 2347-2356). PMLR.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject