Efficient Biological Data Acquisition through Inference Set Design

## Abstract

Abstract In drug discovery, highly automated high-throughput laboratories are used to screen a large number of compounds in search of effective drugs. These experiments are expensive, so we might hope to reduce their cost by experimenting on a subset of the compounds, and predicting the outcomes of the remaining experiments. In this work, we model this scenario as a sequential subset selection problem: we aim to select the smallest set of candidates in order to achieve some desired level of accuracy for the system as a whole. Our key observation is that, if there is heterogeneity in the difficulty of the prediction problem across the input space, selectively obtaining the labels for the hardest examples in the acquisition pool will leave only the relatively easy examples to remain in the inference set, leading to better overall system performance. We call this mechanism inference set design , and propose the use of an uncertainty-based active learning solution to prune out these challenging examples. Our algorithm includes an explicit stopping criterion that stops running the experiments when it is sufficiently confident that the system has reached the target performance. Our empirical studies on image and molecular datasets, as well as a real-world large-scale biological assay, show that deploying active learning for inference set design leads to significant reduction in experimental cost while obtaining high system performance.

## 1 Introduction

Automated high-throughput screening (HTS) laboratories have enabled scientists to screen large compound libraries to find effective therapeutic compounds and screen whole-genome CRISPR knockouts to understand the effects of genes on cell function (Mayr & Bojanic, ; Wildey et al., ; Blay et al., ; Tom et al., ; Fay et al., ). However, conducting experiments on every compound or CRISPR guide in these vast libraries remains very resource-intensive. With typical screening libraries holding on the order of $10^{5}$ to $10^{6}$ compounds (Hughes et al., ) and the number of possible small molecules estimated at $10^{60}$ (Bohacek et al., ), the disparity between our screening capabilities and all which we could explore is staggering. Reducing experimental costs without compromising the quality of the generated data would allow us to accelerate biology and pharmaceutical research and expand the set of molecules considered for testing.

To avoid costs scaling with the number of perturbations, we can train a model from a subset of the target library that has been tested in the lab, and then predict experimental outcomes for the remainder of the library using the trained model (Naik et al., ; Reker & Schneider, ; Dara et al., ), thereby building a *hybrid screen* of the library.
This approach entails three interrelated questions: (1) which subset of the library should we use to maximise the accuracy of the predictions?, (2) how do we select this subset without access to the experimental outcomes?, and (3) how do we ensure that we select a large enough subset to meet a target level of accuracy for the predictions?

This problem is similar to an active learning problem in that we want to select examples that maximize prediction accuracy, but instead of aiming to minimize generalization error, here we only care about prediction on a particular, *finite*, set of experiments from a library.
The fact that the library is finite introduces an important difference: the learner can influence the set of examples on which it is evaluated by strategically selecting examples.
If we accept that the difficulty of predicting outcomes varies among compounds, such that some examples are inherently harder to predict, either due to their complex properties (Bengio et al., ), because of the partial observability of the system (Saleh et al., ; Krenn et al., ) or due to noise in the labeling function (Frénay & Verleysen, ; Lukasik et al., ), the learner can select these examples into the training set to avoid having to predict their outcomes. Conversely, if an example can be reliably predicted by the model, we can save experimental costs by *not* including it in the training set. We call this mechanism through which a learner can influence performance *inference set design*.

We propose an active learning-based solution to hybrid screening that uses the model’s confidence to guide the selection of experiments and leverages the mechanism of inference set design to improve the system’s performance on the target set.
Our algorithm includes a practical stopping criterion that terminates the search for new molecules once a lower bound on a target accuracy threshold is exceeded.
We show in Lemma 1 that this bound provides probabilistic guarantee on the performance of the algorithm as long as the model is weakly calibrated, such that examples for which the model is more uncertain receive a lower predicted probability than those for which it is more certain.

To validate our method, we conduct series of empirical studies on image and molecular datasets, as well as a real-world case study in drug discovery.
The results demonstrate inference set design significantly reduces experimental costs while improving overall system performance.
Importantly, this is true even when the generalization benefits of active learning-based acquisition functions are marginal compared to random search.
This has important practical implications for active learning: if a problem is a hybrid screen—in the sense that one only needs good performance on a fixed, finite set of experiments—then evaluating generalization error dramatically understates the benefits of active learning.
By combining simple active learning acquisition functions with an appropriate stopping criterion, it is possible to make large scale screening far more efficient.

## 2 Methods

In this section, we present the problem of efficiently evaluating a large, *finite* set of experimental designs, such as a screening compound library. We formalize the problem as a subset selection problem, and show that it can be solved using an active learning strategy. Active learning is effective in this setting because it essentially selects its own inference set; we call this strategy, *inference set design* (ISD), and show that it allows to reach high levels of performance on a target set of samples. Finally, we propose a stopping criterion to monitor the model’s performance and trigger termination of the acquisition process. A pseudo-code of the algorithm is presented in Appendix A.

### 2.1 Problem: Minimizing Experimental Costs with Hybrid Screens

We are motivated by the problem of efficient data acquisition of compound libraries for drug discovery applications.
Each “experimental readout” (or label) $y$ requires us to run an experiment, which has some associated cost. We assume that we have a predefined target set $\mathcal{X}_{\text{target}}=\{x_{i}\}^{N_{\text{target}}}_{i=1}$ of experimental designs for which we want to acquire a readout. For example, in drug discovery, a common set of experimental designs, $\mathcal{X}_{\text{target}}$, would be a library of compounds, $\mathcal{D},$ to be tested on some set of cell types, $\mathcal{C}$, such that $\mathcal{X}_{\text{target}}=\mathcal{D}\times\mathcal{C}$. For each $x\in\mathcal{X}_{\text{target}}$, the associated experimental readout, $y$, would corresponds to the drug’s effect on the corresponding cell type. In this section, we assume binary labels, $y\in\{0,1\}$ for simplicity, but also explore extensions to continuous outcomes in the experiments section.

One way to reduce the acquisition costs of screening the entire target set is to train a predictive model $\hat{f}$ on some subset of the target set, $\mathcal{X}_{\text{obs}}\subseteq\mathcal{X}_{\text{target}}$, which we call the *observation set*, and then use this model’s predictions in place of the real labels on the remaining samples. Because experiments are typically run in multiple rounds, at each round, $t$, the target set $\mathcal{X}_{\text{target}}$ can be partitioned into two mutually exclusive subsets,

- •
The observation set $\mathcal{X}_{\text{obs}}^{t}$, which contains the union of all experimental designs that have already been tested in the lab and for which the readouts, $\mathcal{Y}_{\text{obs}}^{t}$, have been observed. This set grows over time as new acquisitions are made.
- •
The inference set $\mathcal{X}_{\text{inf}}^{t}:=\mathcal{X}_{\text{target}}\setminus\mathcal{X}_
{\text{obs}}^{t}$, which consists of the remaining experimental designs in the target set that have not yet been tested. This set shrinks over time as samples are selected from it for acquisition.

Once acquisition is stopped, the combination $(\mathcal{X},\mathcal{Y})_{\text{obs}}\cup(\mathcal{X},\hat{\mathcal{Y}})_{
\text{inf}}$ of both the labeled pairs from the observation set and the predicted pairs from the inference set represent the output of such a system. We call this procedure a hybrid screen, providing a hybrid set of labels for the entire target set – some measured, some predicted. The system’s performance is evaluated based on its accuracy across the entire target set, which consists of perfectly accurate labels $y$ from the observation set and the inferred labels $\hat{y}$ for the inference set. Formally, the system accuracy at each round is,

$$ $\text{ACC}_{\text{sys}}^{t}:=\frac{1}{|\mathcal{X}_{\text{target}}|}\sum_{x\in \mathcal{X}_{\text{target}}}\mathbf{1}(x\in\mathcal{X}_{\text{obs}}^{t})+ \mathbf{1}(\hat{y}=y\mid x\in\mathcal{X}_{\text{inf}}^{t})$ (1) $$

Figure: Figure 1: Hybrid screen employing active learning as an inference set design strategy. The goal is to produce a “hybrid dataset”, composed of the labels collected in the observation set and the predictions of the model made on the inference set. The system’s performance is reported as the “system accuracy”, measured as a combination of observed labels on the observation set and of predictions made on the inference set. As samples are acquired and added to the observation set, the contribution of the accuracy of the prediction model gradually decreases. The aim of the active agent is to select informative examples while pruning out the most difficult ones in order to reach high system performance without having to acquire all the samples.
Refer to caption: x1.png

In this formulation, acquiring all the labels would give 100% accuracy, but at significant expense. We instead want a system that reaches an accuracy of at least $\gamma$, while incurring minimal experiment cost. This can be formulated as the following combinatorial optimization problem,

$$ $\min_{\mathcal{X}_{\text{obs}}\subseteq\mathcal{X}_{\text{target}}}|\mathcal{X }_{\text{obs}}|\quad\text{such that}\quad\text{ACC}_{\text{sys}}>\gamma$ (2) $$

where $|\mathcal{X}_{\text{obs}}|$ denotes the number of samples in the observation set.

A natural greedy heuristic to solve this problem is to treat it as an active learning problem. Like standard active learning, we can sequentially select subsets of data from labelling and the system accuracy will benefit from any improvements in generalization that arise from better selection of training examples. However, this setting has an important difference: we only ever evaluate $\hat{f}$ on $\mathcal{X}_{\text{inf}}$, and this set is under the control of the learner because any example that is selected for labelling is removed from the inference set. As a result, by choosing acquisition functions that select harder training examples for labelling, we make the test-time task of the learner easier, because the harder examples are pruned out of the inference set. We call this approach *inference set design*.

The data selection heuristics for inference set design are similar to standard active learning, but there are two important differences between the settings. First, if we evaluate performance under the standard active learning objective—generalization error measured on a held-out test set—then we can dramatically understate the performance improvement that result from inference set design; because of this, it is possible for active learning algorithms to dramatically outperform random search for inference set design, even when we only observe marginal improvements in generalization error. By actively selecting more challenging examples for acquisition and directing them to the observation set, the model can enhance its performance – it is allowed to actively design the inference set such that it can excel on it. This mechanism is illustrated in Figure 1.
Second, unlike active learning, inference set design requires an explicit stopping criterion in order to decide when you have collected enough samples to meet the accuracy threshold.

##### Acquisition functions

At each timestep $t$, the model $\hat{f}^{t}$ is trained on the observation set $\mathcal{X}_{\text{obs}}^{t}$, and an acquisition function $g$ is evaluated over the entire inference set to select the next batch of samples to acquire. To design easy inference sets, we want acquisition functions that select hard examples for experimentation. For this, we evaluate both least-confidence sampling and query-by-committee. Least-confidence sampling leverages the model’s class prediction probabilities as a confidence metric, selecting samples for which the model exhibits the lowest predicted probabilities:

$$ $\{x\}_{i=1}^{N_{b}}:=\operatorname*{arg\,max}_{x\in\mathcal{X}_{\text{inf}}} \Big{(}1-\max_{y}\,\,p(y|x)\Big{)}$ (3) $$

The query-by-committee approach trains an ensemble of $N_{\text{models}}$ independent predictors in parallel, selecting samples where the ensemble members exhibit the highest disagreement:

$$ $\{x\}_{i=1}^{N_{b}}:=\operatorname*{arg\,max}_{x\in\mathcal{X}_{\text{inf}}}\, \,\mathcal{H}\Big{(}\big{\{}\operatorname*{arg\,max}_{y}\,\,p_{j}(y|x)\big{\}} _{j=1}^{N_{\text{models}}}\Big{)}$ (4) $$

where $\mathcal{H}$ is the entropy over the voting distribution of the ensemble members. This helps identify regions of epistemic uncertainty where model disagrement may stem from over confidence.

### 2.2 Empirical Stopping Criterion

The key technical challenge of inference set design is deciding when to stop running new experiments. In a production system, the labels for the inference set will remain unavailable, so we need to estimate the system performance from the collected data. To avoid unnecessary experimentation, inference set design algorithms must employ an efficient stopping criterion.
We address this by maintaining a probabilistic lower bound on the system accuracy, and stopping once this bound exceeds the critical threshold, $\gamma$. A simple approach to maintaining such a bound is to leverage the feedback from each round of experimentation.

Because at every round we select examples that the model regards as challenging, the performance on each batch $\mathcal{X}_{b}^{t}$ we select at round $t$ will lower bound the performance on the inference set,

###### Lemma 1 .

Let $\hat{p}$ denote the prediction function, $\hat{f}$’s predicted probability that the outcome is 1. If the prediction function, $\hat{f}$, is approximately calibrated, such that for any $p_{1}>p_{2}$, $P[\hat{Y}=Y|\hat{p}=p_{1}]>P[\hat{Y}=Y|\hat{p}=p_{1}]$ for all $p_{1},p_{2}\in(0,1)$, then for a batch, $\mathcal{X}_{b}^{t}$, selected by least confidence, $E_{x\sim\mathcal{X}_{b}^{t}}[\mathbf{1}(Y=\hat{f}(X))]\leq E_{x\sim\mathcal{X}
_{\text{inf}}^{t}}[\mathbf{1}(Y=\hat{f}(X))]$

The proof is in Appendix D. Lemma 1 shows us that we can safely use performance on $\mathcal{X}_{b}$ as a proxy for performance $\mathcal{X}_{\text{inf}}^{t}$, because the former lower-bounds the latter. Because of this, we can treat performance on $\mathcal{X}_{\text{inf}}^{t}$ as though they were IID samples from $\mathcal{X}_{\text{inf}}^{t}$, and construct a (conservative) lower bound on the accuracy on $\mathcal{X}_{\text{inf}}$. Denote the batch accuracy, $\hat{\mu}_{t}=E_{x\sim\mathcal{X}_{b}^{t}}[\mathbf{1}(Y=\hat{f}(X))]$, and its lower bound as $\alpha_{t}$. Using standard concentration inequalities (Lattimore & Szepesvári, , page 135) we have that,

$$ $P(\mu<\alpha_{t})\leq\delta\quad\text{ if }\quad\alpha_{t}=\min\left\{a\in[0,1 ]:KL(\hat{\mu}_{t},a)\leq\frac{\log\left(\frac{1}{\delta}\right)}{b}\right\}$ (5) $$

where $\mu$ denotes the accuracy on $\mathcal{X}_{\text{inf}}^{t}$. We can compute the associated lower bound on system accuracy at each round by assuming we get an accuracy of $1$ on everything in $\mathcal{X}_{\text{obs}}^{t}$ and $\alpha_{t}$ on the remaining batches. Given this, our system stops at time $\tau$ when the following criterion hold,

$$ $\tau=\operatorname*{arg\,min}_{t\in 1,...,t}\frac{|\mathcal{X}_{\text{obs}}^{t }|+\alpha_{t}|\mathcal{X}_{\text{inf}}^{t}|}{|\mathcal{X}_{\text{target}}|}>\gamma$ (6) $$

which ensures achieving our desired accuracy threshold with probability at least $1-\delta$.

###### Lemma 1 .

## 3 Related Work

In this work, we apply active learning to the problem of hybrid screens for biological data acquisition. Warmuth et al. () were among the firsts to apply active learning to the field of drug discovery, with an acquisition function based on support vector machines for binding affinity predictors. AL has since then been explored as a tool for problems such as virtual screening (Fujiwara et al., ), cancer monitoring (Danziger et al., ), protein-protein interaction (Mohamed et al., ), compound classification (Lang et al., ) and chemical space exploration (Smith et al., ).

These approaches seek to produce a better model than training on random samples would, and are to be distinguished from bayesian optimization methods (Graff et al., ; Gorantla et al., ), which also fall under the umbrella of “active learning” but instead use the model to seek specific samples (e.g. with high binding affinity) and often focus on addressing the exploration-exploitation dilemma which has been widely studied by bandit algorithms (Svensson et al., ). For hybrid screens, we seek a model that can accurately predict the labels of all samples in the inference set, without preference over particular readouts. Our setting, focusing on a fixed set of data, is sometimes referred to as pool-based active learning (Wu, ; Zhan et al., ). A key difference is that most active learning work (including pool-based AL) tend to focus evaluations on a held-out test set (Atlas et al., ; Cohn et al., ; Gal et al., ; Margatina et al., ; Zhan et al., ; Luo et al., ; Li et al., ) or restrict the algorithm to very limited acquisition budgets (e.g. only $10\%$ of the data pool) (Wu, ), which generally does not leave enough margin for mechanisms such as inference set design to dominate.

A closely related problem is that of core set selection (Plutowski & White, ; Bachem et al., ; Guo et al., ), where the goal is to select the smallest set of samples from a pool of data such that a model trained on this subset performs on par with a model trained on the entire pool. The closest works to ours are that of Sener & Savarese () and Li & Rangarajan (), which make an explicit connection between active learning and core set selection and also seek to improve performance on the unselected examples. In our work however, we focus on uncovering the key mechanism in action, inference set design, and show empirical validations across a wide variety of academic and real-life datasets.

## 4 Experiments

In our experiments, we aim at highlighting the key role that inference set design plays into allowing AL agents to reach high levels of performance on the inference set (Section 4.1), validating the robustness of this mechanism on molecular datasets used for drug discovery (Section 4.2) and applying this method to a challenging real-world case of compound library screening on a large scale phenotypic assays (Section 4.3). We focus on comparing our active agent against a random agent, which is the de facto approach used in the industry and often found to be a surprisingly strong baseline even in active learning benchmarks (Yang & Loog, ), as well as heuristic-based orderings such as molecular size(^1^11The code for all experiments will be made public upon publication of this work. All datasets to reproduce our results are also publicly available, except one proprietary dataset for the results in Figure 10.).

### 4.1 Visualising Inference Set Design

To develop intuition for inference set design we first run experiments on the MNIST dataset. The whole MNIST training set is used as an inference set from which agents can acquire samples. The MNIST test set is split into 50% validation set used for early stopping and 50% test set used for measuring model performance on held-out data inaccessible by agents. In many real-world applications, data used for training ML models contains samples that are inherently difficult, only partially observed, or mislabelled. We investigate these three cases and simulate the challenge of partial observability by removing the lower part of the image (Figure 3b) and simulate labeling noise by shuffling labels for selected digits (see Figure 3c).

Figure: Figure 2: Percent of samples with shuffled labels acquired by each agent during active learning experiments.
Refer to caption: x2.png

In these experiments we study the performance of active and random agents. At every active learning step agents acquire a batch of 1000 samples from the inference set, add them to the training set, and retrain the model from scratch. The random agent selects uniformly sampled batches, while the active agent selects samples based on the model’s confidence (see Equation 3). During the active learning experiment we monitor accuracy on the inference set, test set, acquired batch, and the system performance as a whole (see Equation 1). To study the behavior of active agents, we also record what samples are acquired at every active learning step.

Figure: Figure 3: Performance of active and random agents on variations of the MNIST dataset. a) Original: MNIST dataset with naturally ocurring easy and hard examples. b) Partial Observability: the MNIST dataset where the bottom two thirds of the images have been cropped out. c) Noisy labeling function: MNIST dataset with shuffled labels for 6, 8, and 9 digits.
Refer to caption: x3.png

In all three studied settings in Figure 3, we observe the same phenomenon: the generalisation accuracy measured on the held-out test set quickly saturates for both agents, but the active agent is able to obtain significant improvements on the inference set which are also reflected in individual acquisition batches and on the system’s overall performance. The mechanism at play is the active design of the inference set by the AL agent. We illustrate this for the case of noisy labeling function in Figure 2. By selectively acquiring mislabeled images, the active agent makes predictions on remaining samples in the inference set much easier. Since the majority of samples acquired by an active agent in the first 20 steps are mislabeled, its accuracy on the acquired batch is significantly lower than a random agent’s (Figure 3c, $3$^rd row). However, once the process of removing mislabeled samples from the inference set is complete, the active agent’s accuracy on the acquired batch drastically increases and surpasses random agent’s accuracy.

To explore the tradeoff between task difficulty and efficiency gains, we run experiments with shuffled labels for increasingly larger subsets of digits (see Figure 4). The results show that with a higher number of mislabeled samples, the gap between active and random agents’ performance on the inference set increases. The active agent is able to achieve 100% accuracy on the inference set even with shuffled labels for 80% of the data but at a later stage of the experiment. However, a more corrupted target set also necessitates more acquisition batches to prune out the difficult examples from the inference set, incurring smaller experimental budget reductions for highly challenging settings.

We have shown that inference set design that inference set design leads to significantly improved system performance compared to random exploration, even with highly corrupted datasets. In the next section, we show that these observations hold for a wildly different type of data.

Figure: Figure 4: Performance of active and random agents on MNIST dataset with different proportions of shuffled labels.
Refer to caption: x4.png

### 4.2 Assessing Robustness of the Mechanism

Figure: Figure 5: Percent of “hard to predict” molecules acquired by each agent during active learning experiments, as assessed by an independent ensemble.
Refer to caption: x5.png

To demonstrate robustness of the inference set design mechanism, we evaluate our approach on different classification and regression tasks. We use two molecular datasets: Quantum Machine 9 (QM9) (Ruddigkeit et al., ; Ramakrishnan et al., ) and Molecules3D (Xu et al., ). QM9 contains $134k$ small organic molecules and their quantum chemical properties computed with Density Functional Theory (DFT). Molecules3D is a much larger dataset that contains structures and DFT-computed properties of approximately $4$ million molecules.

Figure: Figure 6: a) A sample of molecules from the QM9 dataset. b) Performance of active, heuristic, and random agents on HOMO-LUMO gap classification task for QM9 dataset.
Refer to caption: x6.png

In many practical applications exact geometries of screened molecules are unknown as they require computationally expensive DFT calculations. As a first data processing step, we use RDKit library (Landrum et al., ) to convert molecular structures into SMILES strings and compute their Extended Connectivity Fingerprints (ECFPs). The SMILES representation provides complete information about molecule’s composition and atomic connectivity, however, it removes all information about 3D atomic positions. Using SMILES representation is a common solution that simplifies generation of candidate molecules for screening but makes property prediction a more challenging task as many properties vary depending on specific 3D conformation of a molecule. We aim to predict total energy for Molecules3D dataset, and HOMO-LUMO gap for both QM9 and Molecules3D.

Datasets are cleaned by removing duplicated SMILES and fingerprints as well as single-atom structures. Since total energies computed by DFT have wide range of large values, we apply additional processing to normalize them (see Appendix C). The final QM9 and Molecule3D datasets contain $133,885$ and $3,453,538$ molecules respectively. Both datasets are split into inference, validation, and test sets with 80%, 5%, 15% fractions. The QM9 HOMO-LUMO gap values are discretized into 2 balanced classes using median as a boundary condition to explore agents’ performance on classification task.

For QM9 dataset, the active agent uses classification uncertainty to select next batch of samples (similar to MNIST experiment). For regression tasks on Molecules3D dataset it uses a query-by-committee approach computing the variance across the predictions of an ensemble. In addition to active and random agents, we evaluate performance of data selection in particular order. For QM9 experiments, we use original sorting of the dataset as acquisition rule, while Molecules3D dataset was sorted from smallest number of atoms in the molecule to largest.

Figure: Figure 7: Performance of active, heuristic, and random agents for HOMO-LUMO gap prediction (top) and energy prediction (bottom) on Molecules3D dataset.
Refer to caption: x7.png

The results on QM9 dataset demonstrate that the active agent achieves much higher accuracy on the inference set than the baselines. Furthermore, it reaches near 100% accuracy on both system and inference set after acquiring only 30% of data (see Figure 6). For both HOMO-LUMO gap and total energy regression tasks on Molecules3D dataset active agent achieves approximately 5 times lower MSE compared to random agent or in-order selection by molecule size (see Figure 7). For the QM9 experiments, the accuracy on each acquired batch is used for the stopping criterion.

Similar performance across agents on the test set again suggests that the active agent is able to make gains by pruning out difficult examples from the inference set rather than by generalizing better. To confirm this, we compile a list of “hard to predict” molecules in the QM9 dataset and track what the proportion which is acquired by each agent throughout the experiment. To identify these hard samples, we trained 5 models with different random seeds on a 5-fold cross-validation split of the inference set. We then selected molecules for which at least 4 out 5 models made a mistake. The resulting “hard to predict” subset contains $4,944$ molecules (approximately $5\%$ of the inference set). Figure 5 shows that the active agent acquire molecules from the “hard to predict” subset at a much higher rate than the baselines.

These results shows that inference set design is an effective approach to guide acquisition of molecular compounds. It significantly reduces the cost of acquiring a fixed set of molecules and ensures high system performance. In the next section, we apply this method to real-world biological data.

### 4.3 Inference Set Design in the Real World

Modern HTS platforms combined with genetic perturbation techniques facilitate large scale cell microscopy experiments that are designed to capture the effects of biological perturbations (Celik et al., ). In these experiments before taking microscopy images, cells undergo perturbation by either a CRISPR/Cas9-mediated gene knock-out or injection of a bioactive molecular compound at given concentration. The obtained cell microscopy images are then processed and passed through a neural network to obtain embeddings that correspond to a specific perturbation. With data from such HTS pipeline it is possible to build “Maps of Biology” that contain organized information about known and new biological relationships (see Figure 8b). Each point on the map corresponds to the cosine similarity between perturbation embeddings and reveals how strongly two perturbations are related. Expanding maps of biology will uncover new biological relationships that in turn will guide the discovery of leads for new medicines. However, even with ultra-HTS platforms, experimentally acquiring microscopy images of all possible cell perturbations is unfeasible. We aim to apply inference set design to this problem with the goal of reducing costs associated with building maps of biology.

Figure: Figure 8: a) A microscopy image of perturbed cells from the RxRx3 dataset. b) A sample of biological map from the RXRX3 dataset showing whether each pair of gene-based and compound-based perturbations is pheno-similar (in red) or not (in white).
Refer to caption: x9.png

For our experiments, we start by using the publicly available RXRX3 dataset (Fay et al., ) which contains the learned embeddings for $17,063$ CRISPR/Cas9-mediated gene knock-out perturbations, as well as $1,674$ FDA-approved compound perturbations at $8$ concentrations each. We train our models to predict the cosine similarity between the perturbation embeddings of any gene-compound pair, and frame this as a classification problem.

The dataset contains one embedding for each well. Each perturbation type (gene-guide pair or compound-concentration pair) has several replicates across wells, plates and experiments. Each plate also contains unperturbed control cells which are used to keep track of and eliminate a portion of the batch effects (Sypetkowski et al., ). These raw embeddings thus need to be aligned and aggregated. We align them by centering and scaling each perturbation embedding to the embeddings of the experiment-level unperturbed control wells. The embeddings are then aggregated through a multi-stage averaging procedure, across wells, plates, experiments and guides (for CRISPR perturbations), which yields an average embedding for each gene-perturbation and each compound-concentration perturbation. For each gene-compound pairs, we select the compound-concentration perturbation that yields the largest cosine-similarity with the target gene. The cosine-similarities are discretized by computing a gene-specific threshold using the upper semi-interquartile range (SIQR) with a step-size of $2.5$. Finally, we remove genes that have activity below 25%. The result is a $151\times 1,674$ matrix of labels encoding either pheno-similarity ($y=1$) or an absence of relationship ($y=0$) between a gene and a compound perturbation (see Figure 8b).

For this prediction task, we treat all gene embeddings as known and focus only on the much larger space of compound acquisition. To featurize compounds we use embeddings of MolGPS model (Sypetkowski et al., ). At every active learning step, agents acquire a batch of compounds from the inference set and uncover their relationships with all genes in the dataset (entire row of the matrix). The results on RxRx3 dataset show only minor improvements over random agent (see Appendix E). We believe this is due to high complexity of biological relationships and limited size of the inference set. Indeed, infering cellular similarity between gene and compound perturbations from only a few hundred examples is highly unlikely. To push this analysis further, we run the same experiment but with a much larger proprietary dataset similar to RxRx3 that contains cell perturbation embeddings for 102,855 compound and 5,580 gene perturbations, leading a similarity matrix more than $2,000$ bigger. With inference set design, the active agent is again able to outperform the baselines (see Figure 10). These experiments represent promising results for large scale applications of active learning to hybrid screens in drug discovery programs.

Figure: Figure 9: Performance on pheno-similarity classification task for proprietary dataset.
Refer to caption: x10.png

## 5 Conclusion

While active learning has undergone significant theoretical development and experimentation over the past 30 years, its widespread adoption in the industry remains a challenge (Reker, ). Active learning is often framed as a means to improve generalization performance on held-out test sets, which is a goal made difficult by real-world challenges such as noise in labeling functions, high diversity, and partial observability of the input space. However, we believe that a shift in perspective can help unlock the full potential of active learning. By focusing on optimizing performance on a target set of samples and allowing the model to decide which examples to label and which to predict, active learning can become a more powerful tool for real-world applications.

We applied this perspective to the problem of hybrid screens in drug discovery, where the goal is to acquire readouts for only a subset of compounds while making accurate predictions for the remaining ones. Our approach makes use of a confidence-based acquisition function and leverages the concept of inference set design, a strategy where the model selects the most challenging examples for labeling, leaving easier cases for prediction. Our empirical results, across image and chemical datasets, as well as a real world biological application, show that this approach leads to consistent and significant improvements.

#### Acknowledgments

We are extremely grateful to the entire team at Recursion and Valence Labs for participating in insightful discussions and providing feedback on this work. We also wish to specifically thank Oscar Mendez-Lucio and Peter McLean for their precious support with data processing and for engaging in fruitful exchanges about predictive models for phenomics.

## Appendix A Algorithm

Figure: Algorithm 1 Hybrid Screen using Inference Set Design

## Appendix B Hyperparameters and Implementation details

For all presented experiments in this work we use models with ResMLP architecture (Touvron et al., ). All experiments were repeated with 3 different random seeds. Hyperparameters for each experiments are summarized in Table 1.

**Table 1: Hyperparameters for experiments.**
| Hyperparemeter name | MNIST | QM9 | Molecules3D | RxRx3 | Proprietary data |
| --- | --- | --- | --- | --- | --- |
| Acquisition batch size | 1,000 | 50 | 10,000 | 10 | 1000 |
| Number of hidden layers | 2 | 3 | 2 | 2 | 2 |
| Hidden layer size | 512 | 512 | 512 | 512 | 1024 |
| Learning rate | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 |
| gradient norm clip | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| Dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 |
| Train epochs | 1,000 | 1,000 | 30 | 30 | 1000 |
| Train batch size | 1,024 | 1,024 | 32,768 | 1,024 | 1,024 |
| Early stop patience | 50 | 50 | 15 | 25 | 25 |
| Number of ensemble members | None | None | 5 | None | None |

## Appendix C Molecular dataset preprocessing

For total energies in Molecules3D dataset we use reference correction technique where atomic energies are calculated using linear model fitted to the counts of atoms in a molecule of each element present in the dataset. For reference correction a randomly selected sample of 100k molecules is used. The atomic energies are then subtracted from the total energies of all molecules in the dataset. The obtained referenced-corrected energies are normally distributed with mean around 0 eV. A small number of outliers with reference-corrected energy values above 10 standard deviations are removed from the dataset as well as 100k samples that were used for reference correction to avoid data leakage.

**Table 2: Atomic energies used for reference correction on Molecules3D dataset.**
| Atomic Number | Energy (eV) | Atomic Number | Energy (eV) |
| --- | --- | --- | --- |
| 1 | -26.765 | 14 | -7922.180 |
| 5 | -673.550 | 15 | -9313.610 |
| 6 | -1054.411 | 16 | -10810.329 |
| 7 | -1483.001 | 17 | -12474.680 |
| 8 | -2034.056 | 32 | -56538.647 |
| 9 | -2687.455 | 33 | -60828.588 |
| 12 | -5367.759 | 34 | -65296.349 |
| 13 | -6657.476 | 35 | -69980.303 |

## Appendix D Proof of Lemma 1

###### Proof.

Consider the second last batch, $t=T-1$, such $|\mathcal{X}_{\text{inf}}^{t}|=b$. By definition of the least confidence acquisition function, $\mathcal{X}_{b}^{t}$ is the set of $b$ examples in $\mathcal{X}_{\text{inf}}^{t-1}$, for which $1-\hat{p}$ is largest (i.e. the prefix of the set sorted in descending order by $\hat{p}$). If $\hat{f}$ is weakly calibrated, then,

$$ $\displaystyle E_{x\sim\mathcal{X}_{b}^{t}}[\mathbf{1}(Y=f(X))]$ $\displaystyle=\frac{1}{b}\sum_{i\in\mathcal{X}_{b}^{t}}E[\mathbf{1}(Y=f(X))]= \frac{1}{b}\sum_{i\in\in\mathcal{X}_{b}^{t}}P[\hat{Y}=Y|\hat{p}=f(X)|$ $\displaystyle\geq\frac{1}{b}\sum_{i\sim\mathcal{X}_{\text{inf}}^{t}}P[\hat{Y}= Y|\hat{p}=f(X)|=E_{x\in\mathcal{X}_{\text{inf}}^{t}}[\mathbf{1}(Y=f(X))]$ $$

where the inequality follows from weak calibration and least confidence’s selection. Thus the claim holds for the second last batch, $t=T-1$. But, we extend this to the claim holds for $t<T-1$, by simply iterating for any $t$ by induction.
∎

###### Proof.

## Appendix E Additional Results

Results on RxRx3 dataset demonstrate minor improvements compared to random selection. The accuracy on inference and test set remains low throughout the experiment regardless of the acquisition method. This is expected considering low data regime and complex task of predicting biological relationships. The result suggests that in the setting with low predictive power even when majority of the data is acquired, inference set design may provide limited savings and improvements to the system performance.

Figure: Figure 10: Performance of agents on pheno-similarity classification task for RxRx3 dataset.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1) Can you point out a few practical scenarios where ISD is relevant while Bayesian Optimisation not? (see Weaknesses)
2) How can the method handle measurement error?
3) Were you able to see any meaningful pattern in the hard to predict examples in the public dataset chemical problems? (HOMO-LUMO gap and energy prediction)
4) Pleas fix page limit, and the minor issues.
soundness: 4
strengths: The described problem is very close to industrial application, and as such quite relevant.  The paper is clearly written and quite easy to follow. The experiments are quite detailed.
summary: The paper describes the application of active learning for Inference Set Design, a problem where the goal is to label a predefined finite set of instances. To this end, a hybrid screen is proposed where experiment carried out on a subset of the instances (the observation set) and the label on the rest of instances (inference set) is approximated by the prediction of a model trained on the observation set. The paper shows that in this setup the inference set design aspects dominate the better model generalization aspects of active learning.
weaknesses: The machine learning novelty is limited, it describes classical active learning evaluated with a non traditional performance metric. (This metric is however quite well motivated). The argument by the authors, that the benefit of active learning should be interpreted more broadly if the goal is to label the dataset is valid. Using similar argument, however, most of the time the final goal is to maximize (target) or minimize (off-target, tox.) the outcome of the experiment (at least find multiple diverse extrema), the problem become a Bayesian optimization problem, with already existing literature and efficient methods. In the light of this, can you point out a few practical scenarios where ISD is still relevant?

The paper describes the goal as getting the labels cheaper, so if all measurement would be available Accuracy would be 100%. This assumption of no measurement error (zero aleatoric uncertainty) is not reasonable in practical biological assays. At first glance reformulating the problem with measurement noise is not trivial. For example, sometimes it may be beneficial to spend the budget on a repeat if the model flags it as “suspicious” (model based on other data consistently predict different value than the measured one) 

**On page limit**
The manuscript is over the page limit though not by much, as on page 5 a page break is left in. Please fix this.
also 392-393 line break should be removed

Minor:

Line 402 “We have shown that inference set design that inference set design”

L568-569 “With inference set design, the active agent is again able to outperform thebaselines (see Figure 10). ” <- this should be Figure 9 right?

Define $KL(\hat{\mu}_t, a)$ in eq 5

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: What are key differences to existing uncertainty based active learning methods? It appears to me, that the main difference is that the evaluation criterion is performance on the library and not generalization.

In what practical sense does the stopping criterion provide theoretical guarantees? The paper argues that there is a lower bound on target accuracy, assuming the model is weakly calibrated (i.e., the order of uncertainty is correct). However, how reliably does this assumption hold in practice?

For the committee approach, why do you focus on the voting entropy, rather than e.g. entropy of the mean predictive distribution?

I did not find a description in the main text of the ML models used in the experiments.

If I understand correctly, on Molecules3D you include a comparison with data ordered by molecule size, smallest first. Does this not go against the idea of training on difficult examples first? Would it not be more interesting to compare with data ordered by decreasing molecule size? 

Notation:

In eq. 3 Nb is not defined, and the notation is not clear. The left hand side has no index i inside the brackets, and the right hand side evaluates to a single value, not a set. Same goes for eq. 4.

Minor comments:

This sentence is unclear to me: "Because of this, we can treat performance on X_inf^t as though they were IID samples from X_inf^t, and construct a (conservative) lower bound on the accuracy on X_inf.

I am not sure why the method is called "inference set design" - it does not explicitly design the inference set, but is rather a fairly standard active learning scheme with a particular focus on optimizing system performance by including difficult examples in the observation set.
soundness: 3
strengths: The paper is generally well written and easy to follow.

The main idea is clear, useful and effective.

Good experiments to demonstrate the benefits of the approach.
summary: The paper presents an active learning method for hybrid screening, for the setting where there is a finite library and the goal is performance on this library (and not generalization). The idea is to use an active learning approach to include the difficult examples in the training set and stop when the performance is beyond some threshold. The paper discusses two reasonable acquisition functions for this purpose. In each round, a batch of difficult observations are labelled, and the performance on this batch is used to lower bound the performance on the remaining data in the inference set. The method is demonstrated on several relevant benchmark data sets.
weaknesses: The technical novelty is limited, as it is mostly an application of existing methods.

The paper would benefit from including comparisons with other heuristics that capture the "difficulty" of observations. For molecules, one could think of many ways to heuristically score their complexity, and use this to train on most complex examples first. I think this would be of practical interest to know how much the proposed method can improve over such baselines.

The paper appears to have formatting issues, with a mostly blank page and a full page of text exceeding the page limit.

---

confidence: 3
contribution: 2
initial_rating: 1
presentation: 2
questions: 1. What is $N_{\text{target}}$ on page 2? The notation appears without explanation, leaving its meaning unclear.
2. In Lemma 1, the inequality $P[\hat{Y} = Y \mid \hat{p} = p_1] > P[\hat{Y} = Y \mid \hat{p} = p_1]$ is used, but since both sides are identical, the inequality does not make logical sense.
3. In the proof of Lemma 1, the notation $i \in\in\mathcal{X}^{t}_{\text{inf}}$ is unclear. Additionally, the step claiming “the inequality follows from weak calibration and least confidence’s selection” does not seem justified, as it appears to have logical gaps and doesn't rigorously establish the claimed inequality. The overall proof structure needs a more detailed explanation to establish the claimed result.
4. On page 4, lines 212 to 213, this assertion seems to be logically circular, and the abbreviation 'IID' is used without definition. Although 'IID' is a common term, it should be defined in full upon its first appearance in the paper.
5. On page 6, the authors distinguish their approach from Bayesian methods but only compare it with a random agent. However, established Bayesian methods like Bayesian Active Learning by Disagreement (BALD) [1] can effectively select samples that are expected to reduce uncertainty across the dataset. The authors should explain why Bayesian active learning methods cannot be applied in this context and clarify their decision not to compare their approach with BALD or other active learning or reinforcement learning methods.
6. If the method's effectiveness is constrained by a 'low data regime and complex task' (as indicated by the RxRx3 dataset results), how can it be practically useful for real-world biological applications that commonly encounter these challenges?




[1] Houlsby, Neil, et al. "Bayesian active learning for classification and preference learning." arXiv preprint arXiv:1112.5745 (2011).
soundness: 2
strengths: The strength of this paper is derived from addressing an interesting real-world challenge in biological screening, where experimental costs are one of the most significant bottleneck. The proposed inference set design method shows somewhat promising potential for these applications by enabling a strategically targeted selection of samples based on model uncertainty. Another strength is the use of diverse real-world datasets, ranging from molecular property prediction (QM9 and Molecules3D) to cell microscopy experiments (RXRX3 dataset with 17,063 CRISPR knockouts and 1,674 FDA-approved compounds).
summary: This paper tackles the challenge of reducing experimental costs in high-throughput biological screening, with a focus on real-world applications such as drug discovery and cell microscopy. To achieve this, the authors formulate it as a sequential subset selection problem, which seeks to identify the smallest possible subset of compounds for experimental testing, whose outcomes can ensure a desired level of accuracy for the entire system. In other words, the authors propose a mechanism that aims to actively identify and experimentally test the most challenging compounds, leaving the relatively easier-to-predict compounds for the inference set. To strategically label only the most challenging compound samples, the authors propose to employ an uncertainty-based active learning approach that selects samples for which the model shows the lowest predicted probabilities or where ensemble models exhibit the highest disagreement.
weaknesses: The most significant weaknesses of this paper are both technical and presentational. First, this paper does not meet basic ICLR submission guidelines, as its length exceeds the 10-page limit, and this paper contains formatting issues, including a large blank space on page 5 and broken text formatting on lines 392-393 of page 8, which diminishes its professional presentation. Secondly, the experimental comparison is severely limited, with only a random agent used as a baseline method. Given the paper's relevance to active learning and sample selection, the absence of comparisons with reinforcement learning (RL) approaches or other active learning methods is a significant shortfall. Specifically, an RL agent with proper reward modeling could potentially offer competitive performance in sample subset selection. Lastly, the theoretical explanation of Lemma 1 lacks clarity.

---

confidence: 4
contribution: 2
initial_rating: 1
presentation: 1
questions: Typos:
- Is “weakly calibrated” (page 2) the same as “approximately calibrated” (Lemma 1)?
-  In Lemma 1, $P[\hat{Y} = Y \mid \hat{p} = p_1 ] > P [\hat{Y} = Y \mid \hat{p} = \underline{p_2} ]$ for $p_1 > p_2$
- In Equation (5), please mention that KL is the Kullback-Leibler divergence… also mention the name of the result (Corollary 10.4) instead of the page.

Questions:
- What does “treat performance on $X^t_\text{inf}$ as though they were IID samples from $X^t_\text{inf}$” (Line 213) mean? Independence of the probabilities given by the model across data samples from $X^t_\text{inf}$?
- **(Score-changing question)** Please provide a proof for Lemma 1 that solves my concerns listed above.
- **(Score-changing question)** Is there a closed-form expression or at least an upper bound on the stopping time $\tau$ in Equation (6) (as the whole selection process is deterministic as long as the prediction model is deterministic)? If so, is the subsequent budget of measurements (that is, $|X^t_\text{obs}|$ where t is the final selection step of the algorithm) realistic with respect to the total number of items (for instance, we might want a number of measured elements sublinear in N for a reasonable value of $\gamma$, where N is the total number of items in the data set)?
- **(Minor question)** At each time t, the model $\hat{f}^t$ (or $\hat{f}^t_1, … \hat{f}^t_{Nmodels}$ for the query-by-committee) has to be retrained on the full $X^t_{obs}$ and evaluated on $X^t_{inf}$, and then take the arg max on $X^t_{inf}$ (least confidence sampling and query-by-committee). How expensive is it in practice? The budget of measurements ($|X^t_{obs}|$) is important but perhaps we would like also to minimize the number of selection rounds (that is, $t$) as training/evaluating/sampling might also be expensive. Minimizing $t$ might however be contradictory with the objective of minimizing $|X^t_{obs}|$ though, as if we measure a slight over-approximation of the minimal $X^t_{obs}$ in a single selection step, we optimize for the former objective but not the latter.
- **(Minor suggestion)** Why not a more adaptive approach where all points such that the maximal proba across class labels is no higher than 0.5 (and then the acquisition batch size is no longer a constant)? 
- **(Score-changing questions)** (1) There is a shadowed orange part on the plots in Figure 1, are the curves the average across all considered points (System performance: all points in MNIST training set = $|X_\text{target}|$, inference set $|X_\text{target}|-|X^t_\text{obs}|$ -that changes across the x-axis- acquired batch $1,000$, and test set = 50% of full data set) or the average across random seeds (=3 according to the appendix)?

(2) What the “saving” mentioned in Figure 1? The difference between the number of actually measured points and measuring all samples at a given point in time?

(3) What’s the threshold $\gamma$ used in the experiments for the stopping criterion? Can you check that the theoretical guarantee of Lemma 1 is empirically satisfied (by applying some calibration procedure on the trained model, if needed)?

I understand that not all score-changing questions can be answered, especially in the short time for rebuttal. I would be willing to increase the score whenever one of these can be answered (the most important being Lemma 1 and the ones related to the plots) / if the major concerns listed above are addressed.

Summary of the suggested modifications:
- Provide a clear and consistent definition of $b$ (or $N_b$) early in the paper, ideally in a dedicated notation section.
- Ensure that this definition is used consistently throughout the paper, including in equations, lemmas, proofs, and pseudocode.
- Explicitly state earlier in the paper that a constant number of samples is added to the measured point set in each iteration.
- Clearly define all variables used in the proof of Lemma 1, including $\hat{Y}$, $Y$, $\hat{f}$, and $\hat{p}$, and explain their relationships.
- Revise the proof to ensure consistency with the statement in Lemma 1, particularly regarding the inequality sign.
- Provide a step-by-step explanation of the proof, including justifications for each step (e.g., the use of Bayes' theorem).
- Clarify the induction process, especially the step from T-1 to T-2.
- Explain or justify any assumptions made in the proof of Lemma 1, such as "$P(\hat{p}=\hat{f}(X))$"=1.
- Include empirical tests of Lemma 1 in the experiments, specifically verifying that the error is indeed less than $\gamma$ with probability $1-\delta$ at the stopping point.
- Provide access to the code used for the experiments, either through a public repository or as supplementary material.
- Include more details about the experimental plots.

------------------------
**UPDATE**: I raised my score to 8 after reading the updated manuscript and exchanging with the authors.
soundness: 1
strengths: - Originality: The submission tackles an interesting problem, where the goal is not to maximize model generalization but to ensure model correctness on a specific subset of data. This is a first step towards better theoretically-backed approaches in biology (compared for instance to what was done in the LINCS L1000 data set [1] for the regression of gene expression, where they only selected a fixed subset of genes to measure based on prior knowledge instead of difficulty in predicting gene expression).
       
- Quality: The experimental results look good, with appropriate baselines (random selection of new samples to measure, or adding a next batch to measure given the original order of samples). There is a good effort in applying the method to benchmark and real-life, biological data.
       
[1] Subramanian, Aravind, et al. "A next generation connectivity map: L1000 platform and the first 1,000,000 profiles." Cell 171.6 (2017): 1437-1452.
summary: The paper introduces and tries to tackle the problem of identifying a small subset of data on which to run costly measurements to train a machine learning model which infers the most probable measurements for the remainder of the data, with the theoretical guarantee that the model makes less of $\gamma$ percent of errors at inference time. The paper introduces a stopping criterion that would satisfy the correctness guarantee and applies their corresponding algorithm on several benchmark and real-life data sets.
weaknesses: - Clarity: **(Major concern)** The full paper is not very well-written: 

(1) The definition of $b$ (or $N_b$ in the pseudocode in appendix) is unclear and never defined. In Equation (5), it seems that $b=|X^t_\text{obs}|$ that is, the total number of points measured up at time $t$. 

In Lemma 1, $b=|X^t_b|$, which is the total number of new points added to the observed set $X^{t-1}_{obs}$ at time $t$. 

$N_b=|X^t_b|$ in the pseudocode in appendix. 

In the proof of Lemma 1 in appendix, $b=|X^t_{inf}|$, that is, the number of points that are left to infer at time $t$ (there is no reason to assume that $b=|X^t_{inf}|=|X^t_b|$...). 

Moreover, it should be mentioned earlier than the appendix or the experimental section that a **constant** number of samples is added to the measured point set. 

(2) Perhaps a good thing would be to define all variables ($X^t_\text{obs}$, $X^t_b$, etc.) in a single paragraph just after the introduction. Please don’t remove the time-dependent indices ($X_\text{obs}$, $\hat{f}$…) and the hats ($f$ instead of $\hat{f}$ in the proof of Lemma 1), as it makes everything very confusing.

(3) Why mention the query-by-committee sampling, as there is no theoretical guarantee associated with it?

(4) It seems that $\hat{f}$ has a range in $[0,1]$ (or $\mathbb{N}$) whereas $\hat{p}$ has a range in $[0,1]$. Then writing $\hat{p}=\hat{f}(X)$ in the proof of Lemma 1 is extremely confusing.

(5) **(Minor suggestion)** I would put the related works section earlier in the manuscript, for instance, when the differences between active learning and inference set design is discussed, as I think it would make more sense.

(6) **(Minor suggestion)** A one-liner of the full structure of the Algorithm should be present at the end of the section 2 with a link to the pseudo-code in the appendix, so that the algorithmic contribution is clearly mentioned.
       
- Quality/Significance: **(Major concern)** As it is, the proof of Lemma 1 is wrong. What is $\hat{Y}$ ? the difference with $Y$? Is one the prediction made by the model $\hat{f}$ and the other the true measured label (and then $\hat{Y}= \hat{f}(X)$)? However, $y$ seems to be the probability given by the model in Equations (3) and (4). The proof displays the opposite sign of what is stated in Lemma 1 in the main text. Please rewrite properly the proof, including the use of Bayes’ theorem (?) in the second equality, the problem of the definition of the variables $b$, $\hat{f}$, $\hat{Y}$, $\hat{p}$, why “$P(\hat{p}=\hat{f}(X))$”=1 (assuming that here $\hat{f}(X)$ is the probability returned by the model: is it related to the model being accurate on every measured point?), and how the induction works (at least) from T-1 to T-2.

The result shown in Lemma 1 is not empirically tested (that is, at the stopping point of the Algorithm, with a prespecified probability $1-\delta$, the error is less than $\gamma$). The code producing the experiments is not shared, as such, the research is not reproducible.
       
- Significance: **(Minor concern)** The choice of accuracy measure is perhaps not very appropriate for the data: for instance, we expect more drugs to be labeled negatively (for instance, in drug discovery) than positively. Area Under the Curve is said to be slightly more appropriate on imbalanced data [2].

[2] Ling, Charles X., Jin Huang, and Harry Zhang. "AUC: a better measure than accuracy in comparing learning algorithms." Advances in Artificial Intelligence: 16th Conference of the Canadian Society for Computational Studies of Intelligence, AI 2003, Halifax, Canada, June 11–13, 2003, Proceedings 16. Springer Berlin Heidelberg, 2003.

The proof of Lemma 1 being incorrect/badly-written is why I rated soundness to 1. The definitions of notation not being clear to the point of impairing the understanding of the paper is why I rated presentation to 1. The interesting question (mitigated by the perhaps incorrect Lemma 1) is why I rated contribution to 2. The main reason why I rated the paper 1 is because of the proof of Lemma 1.

Meta-review: The paper proposes a subset selection method for biological data labeling to achieve high accuracies with few examples. The reviewers are largely in agreement that this is an important problem, and the application area is novel with real world relevance. There were some concerns about novelty of the proposed method, its technical depth being limited and need for further empirical validation.

Decision: Accept (Poster)