Efficient Biological Data Acquisition through Inference Set Design
Authors: Ihor Neporozhnii
Julien Roy
Emmanuel Bengio
Jason Hartford
Venue: ICLR 2025 Poster (2025)
Keywords: Active Learning
Data Acquisition
ML for Drug Discovery
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose an active learning method for drug discovery, acquiring labels for some samples and training a model to predict the rest. Our models prune the most difficult examples from the target set to achieve high accuracy and reduce costs.

Abstract: In drug discovery, highly automated high-throughput laboratories are used to screen a large number of compounds in search of effective drugs. These experiments are expensive, so one might hope to reduce their cost by only experimenting on a subset of the compounds, and predicting the outcomes of the remaining experiments. In this work, we model this scenario as a sequential subset selection problem: we aim to select the smallest set of candidates in order to achieve some desired level of accuracy for the system as a whole. Our key observation is that, if there is heterogeneity in the difficulty of the prediction problem across the input space, selectively obtaining the labels for the hardest examples in the acquisition pool will leave only the relatively easy examples to remain in the inference set, leading to better overall system performance. We call this mechanism inference set design, and propose the use of a confidence-based active learning solution to prune out these challenging examples. Our algorithm includes an explicit stopping criterion that interrupts the acquisition loop when it is sufficiently confident that the system has reached the target performance. Our empirical studies on image and molecular datasets, as well as a real-world large-scale biological assay, show that active learning for inference set design leads to significant reduction in experimental cost while retaining high system performance.

Published as a conference paper at ICLR 2025
EFFICIENT BIOLOGICAL DATA ACQUISITION THROUGH INFERENCE SET DESIGN
Ihor Neporozhnii 1,2 Julien Roy.! Emmanuel Bengio' Jason Hartford 1.
Valence Labs Universitty of Toronto
UUniversity of Manchester
hhrr .etrr.neprrozhinn @mail utorento. ca
julien. wow..rroooaaleeelllllooo
ABSTRACT
In drug discovery, highly automated high-throughput laboratories are used to screen a large number of compounds in search of effective drugs. These experi- ments are expensive, so one might hope to reduce their cost by only experimenting on a subset of the compounds, and predicting the outcomes of the remaining ex- periments. In this work, we model this scenario problem: we aim to select the smallest set of candidates in order to achieve some desired level of accuracy for the system as whole. Our key observation is that, if there heterogeneity in the difficulty the prediction problem across the input space, selectively obtaining the labels for the hardest examples in the acquisition pool will leave only the relatively casy examples to remain in the inference set, leading to better overall system performance. We call this mechanism inference set design, and propose the use of a confidence-based active learning solution to prune these challenging examples. Our algorithm includes an explicit stop- ping criterion that interrupts the acquisition loop when it is sufficiently confident that the system has reached the target performance. Our empirical studies on im- age and molecular datasets, as well as a real-world aarge-scale biological assay. show that active learning for inference design leads to significant reduction in experimental cost while retaining high system performance.
sequential subset selection
INTRODUCTION
High-throughhut screening (HTS) laboratories have enabled scientists efficiently perform whole- genome CRISPR knockouts and screen large compound libraries to discover effective therapeutics (Mayr & Bojanic, 2009; Wildey al., 2017; Blay et al., 2020; Tom et al., 2024: Fay et al., 2023). However, conducting such experiments on every compound or gene in these vast design spaces remains resource-intensive. With typical sereening libraries holding on the order of 105 to 106 compounds (Hughes et al., 2011) and the number of possible small molecules estimated at 10 (Bohacek et al., 1996), the disparity between our screening capabilities and all which we could ex- plore is staggering. Reducing experimental costs without compromising the quality of the generated data would allow to greatly accelerate biological and pharmaceutical research.
At any stage of a screening procedure, to reduce experimental costs, we can train a machine learning model on the data that has already been collected, and then use that model to predict experimental outcomes for the remainder of the experiments (Naik et al., 2013; Reker & Schneider, 2015; Dara et al., 2022). This procedure generates a hybrid screen the target library, with some outcomes experimentally observed and others obtained from model predictions. However this approach entails interrelated questions: Which subset of the library should we use to maximize the accuracy of the predictions? How do we select this subset without access all experimental outcomes? How do ensure that we acquire a large enough collection to meet our accuracy goal?
This setting is similar to an active learning problem that we want select examples that maximize prediction accuracy, but instead of aiming to minimize generalization error on the entire data space, we focus solely on prediction on one particular set of experiments. The fact that this library is finite introduces an important difference: here the learner can influence the set of examples on which it
Denotes equal contribution

---

Published as a conference paper at ICLR 2025
is evaluated by strategically selecting the examples to be acquired. Since it is often the case that some experiments outcomes are inherently harder to predict than others, either due to their complex properties (Bengio et al., 2009), because of the partial observability of the system (Salch et al., 2021; Krenn et al., 2020) or due to noise in the labeling function (Frénay & Verleysen, 2013; Lukasik et al., 2020), the learner can select these harder examples into the training set early on avoid having to predict their outcomes at a later stage. Conversely, if a group of examples can be reliably predicted by the model, we can save experimental costs by not including them in the training set. We call this mechanism through which a learner can influence its own evaluation set: inference set design.
In this work, we propose an active learning basning-based solution to hybrid screening. Our approach uses the model's 's confidence to guide the selection of experiments and leverages the mechanism of in- ference set design to improve the system's 's performance on the target set. Our algorithm includes a practical stopping criterion that terminates the acquisition of additional labels once lower bound on target accuracy exceeded, and we show that this bound provides probabilistic guarantee on the performance of the algorithm as long as the model ís weakly calibrated. To validate our method, we conduct a series of empirical studies on image and molecular datasets, as well as a real-world case study in drug discovery. The results demonstrate that inference set design significantly reduces experimental costs while improving overall system performance. Importantly, this is true even when the generalization benefits of active learning-based acquisition functions are marginal compared to random search. This has important practical implications for active learning: if a problem is a hybrid screen-in the sense that one only needs good performance on a fixed, finite set of experiments- then evaluating generalization eπооr dramatically understates the benefits of active learning. By combining simple active learning acquisition functions with an appropriate stopping criterion, we show that it is possible to make large scale screening far more efficient.
2 METHODS
In this section, we present the problem of efficiently evaluating a large, finite set of experimental designs, such as screening a compound library. We formalize it as a subset selection problem, and show that it can be solved using an active learning strategy. Active learning is effective in this setting because it essentially selects its own inference set; we call this mechanism, inference set design. We propose stopping criterion to monitor the model's performance and trigger termination of the acquisition process and show that this algorithm allows to reach provably high levels of performance on target set of samples. A pseudo-code is available in Appendix B.
2.1 REDUCING EXPERIMENTAL COSTS VIA HYBRID SCREENS AND INFERENCE SET DESIGN
We are motivated by the problem of efficient data acquisition of compound libraries for drug discov- ery applications. Each "experimental readout" (or label) y requires us to run an experiment, which has some associated cost. We assume that we have a predefined larget set Karget = { { experimental designs for which we want to acquire a readout, where Naaget represents the number of samples we wish to obtain labels or predictions for. For example, in drug discovery, a common set of experimental designs, Kurget: would be a library of thousands or even millions of compounds, each required to be tested on several cell types and at different concentrations. For each E Kunget, the associated experimental readout, yi, would correspond to the drug's 's effect on the corresponding cell type. One way to reduce the acquisition costs of screening the entire target set is to train a predictive model p(x) on the observation set - the subset of the target set, Xats C Xuaget, for which we already have observed the labels y. We can then use this model's predictions in place of the real labels on the remaining samples. Because experiments are typically run in multiple rounds, at each round t, the target set Xarget can be partitioned into two mutually exclusive subsets:
Nume of
The observation set X which contains the union of all experimental designs that have already been tested and for which the readouts, Vis.. have been observed. We denote their pairing the observation dataset Di := y/ := {(x,, yi) } 1 At each step I, a constant number of N, samples are acquired and added the observation set, causing this set to grow over time.
The inference set X := Xuaget \ & which consists of the remaining experimental designs in the target set that have not yet been tested. This set shrinks over time as samples are selected from it for acquisition and transferred into the observation set.
2

---

Published as a conference paper at ICLR 2025
Observation Set (initilly emptyy
Inference Set (60% accuracy)
Set up
Initially, samples are unlabeled the Inference Set.
System accuracy:
The agent tasked select samples labeling
easy examples
difficult examples
Observation Set (labeled examples: 100% accuracy)
Inference Set (50% accuracy)
Random Agent Randomly selects examples to he labeled and transferred to the obsetvation set.
System accurary, 80%
Observation Set (labeled examples: 100% arcuracy)
Inference Set (100% accuracy)
Active Agent selects
difficult samples acquired and removed inference set.
System accurocye
Figure 1: Hybrid screen employing active learning as an inference set design strategy. The goal is to produce a "hybrid dataset", composed of the labels collected in the observation set and the predictions of the model made on the inference set. The system's performance is reported as the "system accuracy", measured as a combination of observed labels on the observation set and of predictions made the inference set. As samples are acquired and added to the observation set, the contribution of the accuracy of the prediction model gradually decreases. The aim of the active agent to select informative examples while pruning the most difficult ones in order to reach very high system accuracy without having to acquire all the samples.
Once acquisition is stopped, at timestep t = т, the output of the system is a hybrid dataset, i.e. the combination (X,)) (X,))) U (X' V of both the labeled pairs from the observation set and the predicted pairs from the inference set. We call this procedure a hybrid screen, as provides hybrid set of labels for the entire target set some measured, some predicted. The system's 's performance at each step is evaluated based on its accuracy Mggs across the entire target set, which consists of perfectly accurate labels from the observation set and the inferred labels ý for the inference set:
Psys
Σ
1(x; Xohn 1(:: yi
I., X
Karrgel
this formulation, acquiring all the labels would yield 100% accuracy, but at significant expense. We instead seek system that reaches an accuracy of at least y, while incurring minimal experimen- tal costs. This can be formulated as the following combinatorial optimization problem,
min
(obs such that
Mays
(2)
Kitt Marget
where |xataa denotes the number of samples in the observation set. A natural approach is to treat this as an active learning problem. Like standard active learning, we can sequentially select subsets of data for labeling and the system accuracy will benefit from any improvements in generalization that arise from better selection training examples. However, this setting has an important difference: only ever evaluate ş on Xuts and this set is under the control of the learner because any example that is selected for labeling is removed from the inference set. As a result, by choosing acquisition functions that select harder training examples first, we make the test-time task of the learner easier, as
Eqquation assumes classification problem, but also explore continuous outcomes y in Appendix E.I.
3

---

Published as a conference paper at ICLR 2025
the harder examples are effectively prumed out of the inference set. We call this approach inference set design - by selective label acquisition, the agent actively designs the composition of the inference set such that it can excel on it. This mechanism is illustrated in Figure 1.
The data selection heuristics for inference set design are similar to standard active learning, but there are two important differences between the settings. Firstly, if we evaluate performance under the standard active learning risk dramatically understating the performance improvement that result from inference design. Instead, evaluating on the target set of interest allows to capture dramatical improvements over a random sampler even when only marginal improvements in generalization error can be observed. Secondly, unlike active learning. inference set design requires an explicit stopping criterion in order to decide when you have collected enough samples to meet the accuracy threshold, which we address in the following section.
generalization error measured on a held-out test set-we
2.2 EMPIRICAL STOPPING CRITERION
The key technical challenge of inference set design is deciding when to stop running new exper- iments. In a deployment environment, the labels for the inference set will remain unavailable, so we need to estimate the system performance from the collected data only. To avoid unnecessary experimentation, inference set design algorithms must employ efficient stopping criterion. We address this by maintaining probabilistic lower bound on the system accuracy, and stopping once this bound exceeds the critical threshold, 7. A simple approach to maintaining such a bound is to leverage the feedback from each round of experimentation.
We consider a multi-class classification problem. Formally, let K= 21, K} be the of K labels and p(x) == (pi (z).... ................................ PK (2)) be the predicted probabilities for æ that = k for all k E K.. We denote y; the true label for a sample 2,. D; == arg max, Pre (I,) the predicted class by the model and é == max P& (ara) the predicted probability value for that class. We consider the use of the commonly employed least-confidence acquisition function:
Trelected =
arg max
(x))) for i= i......, No
(3)
where N₂ is the number of samples to be acquired, i.e. the acquisition batch-size. The samples a:; E X, are selected one by one by applying the least-confidence criterion in Equation 3 iteratively to the remaining samples in Xn that have not yet been transferred to Xg. Because at every round we select examples that the model regards as challenging. if the model maintains a correct calibration ordering, the performance a batch Xÿ that select round t should lower bound the performance on the entire inference set. We express this formally in the following lemma:
Lemma 1. Assuming that p's 's top- copprredittions are weakly calibrated on the inference set, i.e. that for any two samples iı and 22 in D'y 01 2 02 = Pr..i Y Y|61] 2 PY =YbE), then the expected accuracy on a batch DJ := { f selected by ceast-confidenceffro X is less or equal than the expected accuracy on the remaining inference set: PDę = Y) PDi IY YJ.
See Appendix A.I for more details and proof. Lemma shows that although the samples in X are not independent and identically distributed samples from X we can safely use the performance on Dj as conservative proxy for performance on Dí We thus leverage the measured performance on the acquired batch to estimate the performance on the remaining, unobserved inference set.
We can now use this conservative estimate design an appropriate stopping criterion. Let ju} := N. E,n, 1yy = úi)) denote the measured accuracy on the batch Dj at acquisition step I. A simple stopping criterion could consist in picking an accuracy threshold y for the desired system accuracy, and to stop acquiring new samples once system accuracy estimate A based on jú surpasses y:
2 + 11 1 Kuaretr
T
arg min play > tEI...,?
where denotes the stopping time. Optionally, hyperparameter to make it more robust to the stochasticity introduced by the finite sample size No- A more principled approach however is to directly account for the statistical uncertainty around At
could augment this criterion with patience
4

---

ensure achieving our Published as a conference paper at ICLR 2025
by using it derive bound ag on the true inference accuracy pau- Using standard concentration inequalities (see detailed derivation in Appendix A.2) we have that,
P( I an) S 8 with or = min { a E A8) KL(A a)
log (1) }
(5)
No
where -L(-II-) denotes the Kullback-Leibler divergence and ō is the failure probability of the bound. We can thus use the lower- value a place of p in Equation desired system accuracy threshold with probability at least 1 1 ò.
To summarize, in this section, we presented a framework for hybrid screens leveraging active learn- ing to perform inference set design by selectively acquiring labels for pard-to-predic examples, thereby simplifying the inference set for easier prediction, and using a conservative stopping crite- rion to ensure that the desired level of system accuracy is achieved at stopping time. The pseudo- code for our algorithm presented in Appendix B.
3 RELATED WORK
In this work, we apply Active Learning (AL) to the problem of hybrid screens for biological data acquisition. Warmuth al. (2003) were among the first to apply AL the field of drug discovery, with an acquisition function based on support vector machines for binding affinity predictors. AL has since then been explored as a tool for problems such virtual screening (Fujiwara al., 2008), cancer monitoring (Danziger et al., 2009), protein- protein-protein protein interaction (Mohamed et al., 2010), com- pound classification (Lang et al., 2016) and chemical space exploration (Smith al., 2018).
These approaches seek to produce better model than training on random samples would, and are to distinguished from bayesian optimization methods (Graff al., 2021; Gorantla et al., 2024), which also fall under the umbrella of "active learning" but instead use the model to seek specific samples (e.g. with high binding affinity) and often focus on addressing the exploration-exploiattio dilemma which has been widely studied by bandit algorithms (Svensson al., 2022). For hybrid screens, we seek a model that can accurately predict the labels of all samples in the inference set, without attempting maximize the value of the readouts. Our setting, focusing data, is sometimes referred to as pool- based active learning (Wu, 2019; Zhan et al., 2021). Another similar approach called transductive experimental design aims select samples that improve the predictions for a specific pre-defined test set (Y (Yu et al., 2006; Hübotter et al., 2024). A key difference that most active learning work (including pool-based AL and transductive experimental design) tends to focus evaluations on held-out test set (Atlas et al., 1989; Cohn et al., 1994; Gal et al., 2017; Margatina al., 2021; Zhan et al., 2021; Luo al., 2023; Li et al., 2024) or restrict the algorithm to very limited acquisition budgets (e.g. only 10% of the data pool) (Yu et al., 2006; Wu, 2019), which generally does not leave enough margin for mechanisms such as inference set design to dominate.
a fixed set of
A closely related problem is that of core set selection (Plutowski & White. 1993; Bachem et al., 2017: Guo et al., 2022), where the goal is to select the smallest set of samples from a pool of data such that a model trained on this subset performs on par with model trained on the entire pool. The closest works to ours are that of Scner & Savarese (2017) and Li & Rangarajan (2019), which make explicit connection between active learning and core set selection and also seek to improve performance on the unselected examples. In our work however, we focus on uncovering the key mechanism in action, inference set design, and show empirical validations across wide variety of academic and real-life datasets.
4 EXPERIMENTS
In our experiments we aim at highlighting the key role that inference set design plays in allow- ing AL agents to reach high levels of performance on the inference set (Section 4.1), validating the robustness of this mechanism molecular datasets used for drug discovery (Section 4.2), apply- ing this method to a challenging real-world case of compound library screening on a large seale
The code is available at https: : //g' thub. com/ i neporozhn's 1/1 fefeence_ssst dest inn.gn. datasets reproduce our results are publicly available, except one proprietary dataset for the results in Figure 8.
gn. All
5

---

Published as a conference paper at ICLR 2025
phenotypic assays (Section 4.3) and empirically validating theoretical assumption and stop- ping criterion (Section 4.4). Unless specified otherwise, curves show average metrics across seeds and the shaded areas denote the standard emrrr of the mean. Vertical lines denote the average stopping time, with min and max intervals (left to right). For all modalities, we use vectorized rep- resentations input and our model p is parameterized using feed-forward deep neural network with residual connections, Additional details regarding data pre-processing, network architectures and hyperparameters are presented in Appendices and D.
a)
Easy
Hard
b)
Original
Corrupted
Shuffled labels
3
4
6
4
3
9
8
x=4
System Performance
100
Accuracy.
LG 30.0% Random
Savtlgs: 71.7% Savingsi 43.7%
Sevtiggs
Savtigg: 53 33.3%
Bandom
Savings: 8.3%
Snvtings 6.7%
100
Inference Set
Accuracy.
ageat Randen apent
agent
ageat
Banden aguci
Randeen aquei
Acquired Batch
100
Accuraty
ogent Bandom agent
ogent Random agest
agent Randocn agenz
Test Set
Accuracy,
egentt
agent
Raniio agent
Rondem agent
ucquiinad,
acquired,
Parroaal acguired,
Figure 2: Performance of lcast-confidence (LC) and random acquisition functions on variations of MNIST (columns) across 3 seeds. Original: MNIST with its naturally occurring easy and hard examples. b) Partial Observability: MNIST where the bottom two thirds of the images have been cropped out. c) Noisy labeling function: MNIST with shuffled labels for images of 6's, 8's, and 9's. The system seeks target accuracy 7 of 98%.
4.1 VISUALIZING INFERENCE SET DESIGN
To develop intuition for inference set design we first run experiments on the MNIST dataset. The whole MNIST training MNIST test set is split 50-50 into a validation set used for early stopping and test set used for measuring model performance on held-out data inaccessible by agents. In many real-world applica- tions, data used for training MI. models contains samples that are inherently difficult, only partially observed, or mislabeled. We investigate these three cases. Specifically, we simulate the challenge of partial observability by removing the lower part of the image (Figure 2b) and simulate labeling noise by shuffling labels selected digits (see Figure 2c).
used the target set from which agents acquire samples. The

---

Published as a conference paper at ICLR 2025
Test Set
Inference Set
activw, 6-4 labels shufleed
0.7 lbels shufteed
Accuracy 180
labels shuftied
Accuracy (%)
0-0 bbels shuttled
random, 0-9 labels shuflldd labels shutteed
shullled
0.1 lbbes shuffted
Percent of data acquired (%)
33
Percent of data acquired (%)
100
0-5 labels shuffted
Figure 3: Performance of active and random agents MNIST subject to increasing number of
shuffled labels. As the task becomes harder, the accuracy gap increases, but the stopping time r is triggered later.
In these experiments we study the performance of an active agent using the least-confidence (LC) acquisition function of Equation and a random agent, selecting samples with uniform probability over the inference set. At every active learning step an agent acquires a batch of 1000 samples from the inference set, adds them to the training set, and retrains the model from scratch. Across these active learning steps, monitor the accuracy the inference set, the test set, the acquired batch, and the system accuracy
Percent of mislabeled samples acquired,
40
LC Cagent Random agent
whole (see Equation 1). We also record which samples have been acquired at every acqui- sition step. all three studied settings in Figure we ob- serve the same phenomenon: the generalisation accuracy measured on the held-out test set quickly saturates for both agents, but the active agent is able to obtain signif- icant improvements on the inference set, and these gains are also reflected in individual acquisition batches and the system's overall performance.
42%
83%
Percent data acquired, %
Figure 4: Percent of samples with shuffled labels acquired by each agent throughout the acquisition steps.
The mechanism at play here stems from the active design of the inference set by the agent. We illus- trate this for the case of a noisy labeling function in Figure By selectively acquiring the images it tends to mislabel, the active agent makes predicting the labels the remaining samples in the inference set much easier. Since the majority of samples acquired by the active agent in the first 20 steps represent these challenging examples, its accuracy the acquired batch is significantly lower than random agent's (Figure 2c, 3rd row). However, once the process of removing mislabeled sam- ples from the inference set is complete, the active agent's is accuracy the acquired batch drastically increases and surpasses the random agent. To further explore the tradeoff between task difficulty and efficiency gains, we run experiments with shuffled labels for increasingly larger subsets digits (see Figure 3). The results show that with higher number of such "noisy" samples, the gap between the active and random agents on the inference set increases. The active agent able to achieve 100% accuracy on the inference set even with shuffled labels for 80% of the data. However, a more cor- rupted target set also necessitates more acquisition batches to prune out difficult examples from the inference set, incurring smaller experimental budget reductions for highly challenging settings.
4.2 ASSESSING THE ROBUSTNESS OF THE MECHANISM
In the previous section, we have shown that inference set design leads to significantly improved system performance compared to random sampling, even with highly corrupted datasets. In this section, we validate that these observations hold for wildly different type of data molecular datasets. We use the Quantum Machine (QM9) (Ruddigkeit et al., 2012; Ramakrishnan et al., 2014). QM9 contains 134k small organic molecules and their quantum chemical properties com- puted with Density Functional Theory (DFT). For inputs, we convert the SMILES strings molecular representations into their Extended Connectivity Fingerprints (ECFPs). We predict the HOMO-

---

Published as a conference paper at ICLR 2025
b)
System Performance
Inference set
Accuracy %
20
90
20
Accuracy:
40
Sunning: 53,
BALD apert
LC Cagert
Divertly apent
BALD agent
Orginal oederig
Gering:
agant
Skeeam arcedin
20%
97%
100%
20%
40%
50%
2005
Percent of data aquired, 16 Acquired batch
Percent of data acquired, %
Accuracy,
40
Accuracy,
80
20
70
LC agent
Sueerit agent Orginal ordering
agent BALD agent
Oieers noent
sport
Onginal ordering
20%
40%
65%
59%
100%
0%
20%
40%
50%
50%
200%
Percent of data acquired, 36
Parcent data acquired, %
Figure 5: Results on QM9. a) sample of molecules from the QM9 dataset. b) Performance of active agents (LC and BALD), heuristic orderings, and random sampler on QM9 across 3 seeds.
LUMO gap framed as a classification task by discretizing the values into two balanced classes using median boundary condition to explore agents' performance classification task. Our main active agent uses the lcast-confidence acquisition function. We also explore the use the BALD acquisition function (Gal et al., 2017). In addition to active and random agents, for both datasets, also evaluate the performance of heuristic-based acquisition orderings (original ordering, smallest to largest molecules, ordered SA-score, etc.) well diversity sampling strategy (selecting molecules with the largest Tanimoto distance to the acquired set).
The results QM9 demonstrate that both LC and BALD agents achieve much higher accuracy on the inference set than the baselines. Furthermore, they reach near 100% accuracy on both system and inference set after acquiring only 30% of data (see Figure 5). Saturating performance across all agents on the test set suggests once again that
120
Percent of 'hard to predict'
molecules acquired, %
50-
the active agent is able to make gains by pruning out diffi- cult examples from the inference set rather than by gener- alizing better. To confirm this hypothesis, we compiled a list of "hard to predict" molecules by training five models with different random seeds on a 5-fold cross cross-validene validation
Diversity agant
BALD agent
20
SA-icore ordeting
Random agent
split of the inference set, and flagged the molecules for which at least 4 out models made a mistake. The re- sulting "hard to predict" subset contains 944 molecules (approximately 5% of the inference set). We then tracked the proportion of these samples that was acquired by each agent in the active learning experiments. Figure 6 shows that the active agents acquire molecules from the "hard predict" subset at much higher rate than the baselines.
Original ondering
25%
40%
80%
100%
Percent of data acquired,
Figure 6: Percent of "hard" examples acquired by each agent throughout the acquisition loop on QM9.
In addition to the classification results on QM9 presented here, we extend our method to a regression task the much larger Molecules3D dataset in Appendix E. and get to similar conclusion. Put together, these results on additional datasets spanning both classification and regression tasks and including 3 different active acquisition functions confirm and extend the observations made on MNIST in the previous section. They suggest that inference set design is an effective approach to guide acquisition of molecular compounds, by significantly reducing the cost of acquiring fixed set of molecules and ensuring high system performance. In the next section, we apply this method weal-world biological data.
8

---

Published as a conference paper at ICLR 2025
Farturbed Miscrosropy image
Compounds
Rococ3 Discretized Gene cone-Compound Relationships
tacak
fuoromahsoone
Dowlyline
Cydiheneaprh
mistemiine
De
zon
Futendizble
Epochilone
aroiro
atbazzcle
ncely--teucho ne
Accuzz mide
Metam zole
darona
Figure 7: Visualisation of the RxRx3 dataset. a) A microscopy image of perturbed cells. b) sample of biological map showing whether each pair of gene-based and compound-bans based perturbations is pheno-similar (in red) or not (in white).
System Performance
Inference set
Acquired batch
Test set
LC agret Savings
BALD apent
Accuracy.
Savings:
Random agent
Savings:
Ordered by
Giversky agent
SA-acore
06 205
4955
00%
E35
130%
35
205
40%
40%
80%
1000
005 Percent of dala acquired,
35
20%
40%
55%
10B5
06
285
45%
25
80%
100%
Percent dela acquired,
Percent of data acquired,
Percent of data acquired, %
Figure 8: Performance dataset. The least-confidence agent is shown to be the most effective on this task and reaches the targeted system accuracy of 98% at around 80% of the acquisition procedure.
pheno-similarity classification task
proprietary phenomics
4.3 INFERENCE SET DESIGN IN THE REAL WORLD
Modern HTS platforms combined with genetic perturbation techniques facilitate large scale cell microscopy experiments that are designed to capture the effects of biological perturbations (Celik et al., 2022). In these experiments before taking microscopy images, cells undergo perturbation by either CeLSPR/Cas--meditee gene knock-out or injection of bioactive molecular compound at given concentration. The obtained cell microscopy images are then processed and passed through a neural network to obtain embeddings that correspond to a specific perturbation. With data from such HTS pipeline it is possible build "Maps of Biology" that contain organized information about known and new biological relationships. Each point on the map corresponds to the cosine similarity related. Expanding
between perturbation embeddings and encodes whether maps of biology might uncover new biological relationships that turn could guide the discovery of leads for new medicines. However, even with ultra-HTS platforms, experimentally acquiring microscopy images of all possible cell perturbations is unfeasible. We aim to apply inference set design to this problem with the goal of reducing costs associated with building maps of biology.
perturbations
For our experiments, we start by using the publicly available RxRx3 dataset (Fay et al., 2023) which contains the learned embeddings for 17, 063 CRISPR/Cas9-mediatee gene knock-out perturbations, as well as 1, 674 FDA app-oopd compound perturbations at 8 concentrations each. We frame this as a classification problem and train our models to predict the discretized cosine similarity between the perturbation embeddings of any gene-compound pair. As preprocessing, for each gene-compound pairs, we select the compound-connentration perturbation that yields the largest cosine-simile similarit with the target gene. The cosine-similarities are discretized by computing a gene-specific threshold using the upper -imi-interquartile range (SIQR) with a step-size genes that have activity rate (across compounds) below 25%. The result is a 151 1, 674 matrix of labels encoding either sheni-similarity (y 1) or an absence of relationship (y = 0) between a gene and compound perturbation (see Figure 7b).
2.5. Finally, we remove the

---

Published as a conference paper at ICLR 2025
For this prediction task, we treat all gene embeddings known and focus only the much larger space of compound acquisition. To featurize compounds we use embeddings of the MolGPS model (Sypetkowski et al., 2024). At every active learning step, agents acquire batch of compounds from the inference set and uncover their relationships with all genes in the dataset (entire row of the matrix). The results the RxRx3 dataset show only minor improvements over random agent (see Appendix E). We believe this is due to high complexity of biological relationships and limited size of the inference set. Indeed, successfully learning infer cellular similarity between gene and compound perturbations from only few hundred examples is highly unlikely. To push this analysis further, we re-run the same experiment but on our proprietary phenomics dataset. It is similar to RxRx3 but much larger. It contains cell perturbation embeddings for 102,855 compound and 5,580 gene perturbations, leading to 2, 000 times larger relationship matrix. Using inference set design, the active agent is now able to outperform the baselines on this task, as shown in Figure 8. These experiments represent promising results for large scale applications of active learning to hybrid screens for early drug discovery.
4.4 EMPIRICAL. VALIDATION OF WRAK CALIBRATION AND STOPPING CRITERION
In this last experimental section, we validate the theoretical claims made in Section 2.2. The pro- posed stopping criterion assumes weak calibration i.c. that the true accuracy of the model does not need to be exactly equal to its confidence (perfect calibration), but that the correct ordering is con- served. Figure 12 in Appendix F.1 shows that this assumption is empirically verified in all of our experiments, supporting the claim that weak calibration can be achieved in practice. In Figure 13, in Appendix F.2, we also empirically validate that the proposed stopping criterion making use the bound presented in Equation 5 does indeed cause the agent to stop acquiring new labels at a system accuracy superior to the targeted threshold (see Equation 4).
5 DISCUSSION
While active learning has undergone significant theoretical development and experimentation over the past 30 years, its widespread adoption in the industry remains a challenge (Reker, 2019). Active learning is often framed as a means to improve generalization performance on held-out test sets, which is goal made difficult by real-world challenges such noisy labeling functions well as high diversity and partial observability of the input space. However, we believe that a shift in perspective can help unlock the untapped potential of active learning. By focusing on optimizing performance on a target set of samples and allowing the model to decide which examples to label and which to predict, active learning can become a powerful tool for real-world applications.
We applied this perspective to the problem of hybrid screens in drug discovery, where the goal to acquire readouts for only subset of compounds while making accurate predictions for the remaining ones. Our approach makes use of confidence beadddeenceebaeed acquisition function and leverages the concept of inference set design, a strategy where the model selects the most challenging examples for labeling, leaving easier cases for prediction. Our empirical results, across image and chemical datasets, as well as a real world biological application, show that this approach leads to consistent and significant improvements. Moreover, our results highlight that heuristic- used in real-world data acquisition efforts, are highly suboptimal. They can lead to drastically different results across tasks and datasets and often end up harming the performance by introducing unjustified biases in the acquisition ordering. Importantly, this bias then prevents the experimenter the
orderings, often
from using the performance on the acquisition batch as an indication of the performance remaining examples in the inference set. A confidence-based agent also introduces bias in sample selection, but assuming weak calibration, this bias is directional and still allows the acquisition batch to be used as lower bound for the performance on the remaining examples (see Lemma 1). It then compensates for the conservativeness of its stopping criterion by designing its own inference set, leading to overall earlier stopping time and greater budget reductions.
An important limitation of our work is the assumption that observed labels are deterministic and constitute perfect accuracy. Many real-world biological experiments however produce noisy obser- vations. Combining the inference set design framework with aleatoric uncertainty modeling to better address cases of weak supervision represents a promising direction for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We are grateful the entire team Recursion and Valence Labs for participating insightful discussions and providing feedback on this work. We also wish to specifically thank Austin Tripp for offering valuable input on this work as well as Oscar Mendez-Lucio and Peter McLean for their precious support with data processing and for engaging in fruitful exchanges about predictive models for phenomies.
REFERENCES
Les Atlas, David Cohn, and Richard Ladner. Training connectionist networks with queries and selective sampling. Advances in neural information processing systems, 2, 1989.
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv: htt:::70330000 00476 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning, pp. 41-48, 2009.
Vincent Blay, Bhairavi Tolani, Sunita Ho, and Michelle R. Arkin. High-throughput screen- ing: today's s biochemical and cell-based approaches. Drug Discovery Today, 25(10): 1821, 2020. ISSN 1359-6446. doi: http:///ccciieeeerrrre//////////
URL https:
/ //www. www.ssseieeeeeee..c....
Regine S Bohacek, Colin McMartin, and Wayne C Guida. The art and practice of muucture-based drug design: molecular modeling perspective. Medicinal research reviews, 16(1):3-50, 1996.
Safiye Celik. Jan-Christian Hütter, Sandra Melo Carlos, Nathan H Lazar, Rahul Mohan, Conor Tillinghast, Tommaso Biancalani, Marta Fay, Berton Earnshaw, and Imran S Haque. Biological cartography: Building and benchmarking representations of life. Biorxiv, pp. 2022-12, 2022.
Herman Chernoff. A measure of asymptotic efficiency for tests of hypothesis based the sum of observations. The Annals of Mathematical Statistics, pp. 493-507, 1952.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15:201-221 1994.
Samuel Danziger, Roberta Baronio, Lydia Ho, Linda Hall, Kirsty Salmon, Wesley Hatfield, Peter Kaiser, and Richard H Lathrop. Predicting positive p53 cancer rescue regions using most informative positive (mip) active learning. PLoS computational biology, 5())=e 1000498, 2009.
Suresh Dara, Swetha Dhamercherla, Surender Singh Jadav, CH Madhu Babu, and Mohamed Jawed Ahsan. Machine learning in drug discovery: review. Artificial intelligence review, 55(3):1947- 1999, 2022.
Marta M Fay, Oren Kraus, Mason Victors, Lakshmanan Arumugam, Kamal Vuggumudi, John Ur- banik, Kyle Hansen, Safye Celik, Nico Cernek, Ganesh Jagannathan, et al. Rxix3: Phenomics map of biology. Biorxiv, pp. 2023-02, 2023.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey, IEEE transactions on neural networks and learning systems, 25(5):845-869, 2013.
Yukiko Fujiwara, Yoshiko Yamashita, Tsutomu Osoda, Minoru Asogawa, Chiaki Fukushima, Masaaki Asao, Hideshi Shimadzu, Kazuya Nakao, and Ryo Shimizu. Virtual screening sys- tem for finding structurally diverse hits by active learning. Journal of chemical information and modeling, 48(4)99309940, 2008
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 11 1183-1192 192. PMLR, 2017.
Rohan Gorantla, Alzbeta Kubincova, Benjamin Suutari, Benjamin P Cossins, and Antonia SJS Mey. Benchmarking active learning protocols for figand-binding affinity prediction. Journal of Chem- ical Information and Modeling, 6955))195511993 2024.

---

Published as a conference paper at ICLR 2025
David E. Graff, Eugene I. Shakhnovich, and Connor W. Coley, Accelerating high-throughput virtual screening through molecular pool-based active learning. Chem. Sci., 12:7866-7884) 2021. doi: 10.1039/D0SC0680EE, URL http: : / /xx doi. .or. 1ttp///.1109/0SCC055550
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause. Trans- ductive active learning: Theory and applications. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Newral In- formation Processing Systems, volume 37, pp. 124686-124755, Curran Associates, Inc., 2024. URL https::/proocedings..tt neurips cc/ wttp:iipppppppppp..ppeee wwtps/peee..lleeeeppeeeeee/////e file/e (tl.a///6699fff177000001111e---0
pdf.
James P Hughes. Stephen Rees, S Barrett Kalindjian, and Karen Philpott. Principles of early drug discovery. British journal of pharmacology, 162(6))1239-1223 2011.
Mario Krenn, Florian Häse, KkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self- referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020.
Greg Landrum, Paolo Tosco, Brian Kelley, Ricardo Rodriguez, David Cosgrove, Riccardo Vianello, sriniker, Peter Gedeck. Gareth Jones, NadineSchneider, Eisuke Kawashima, Dan Nealschneider, Andrew Dalke, Matt Swain, Brian Cole, Samo Turk, Aleksandr Savelev, Alain Vaucher, Maciej Wojeikowski, Ichiru Take, Vincent Scalfani, Rachel Walker, Kazuya Ujihara, Daniel Probst, tadhurst cdd, guillaume godin, Axel Pahl, Juuso Lehtivarjo, François Bérenger, and strets123. rdkit/rdkit: 2024_03.6 (q1 2024) release, August 2024. URL BtLhttps:/ //t::////iiiie/// wrt./orgg111000 5281/ zenodo 13469990.
Tobias Lang, Florian Flachsenberg, Ulrike von Luxburg, and Matthias Rarey. Feasibility of active machine learning for multiclass compound classification. Journal of chemical information and modeling, 56(1):12-20, 2016.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Bowen Li and Srinivas Rangarajan. Designing compact training sets for data-driven molecular property prediction through optimal exploitation and exploration. Molecular Systems Design & Engineering, 1048:1048 2019.
Xiongquan Li. Xukang Wang, Xuhesheng Chen, Yao Lu, Hongpeng Fu, and Ying Cheng Wu. Un- labeled data selection for active learning in image classification. Scientific Reports, 14(1):424, 2024.
Michal Lukasik, Srinadh Bhojanapalli. Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 4448-6458. PMLR, 2020.
Yunan Luo, Yang Liu, and Jian Peng. Calibrated geometric deep learning improves kinase drug binding predictions. Nature Machine Intelligence, 5(12): (112))1390-1400)) 3901-40111, 2023.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. arXiv preprint 1tx.i:21099.37644. 2021.
Lorenz M Mayr and Dejan Bojanic. Novel trends in high-throughput screening. Current Opin-
ion Pharmacology. 95):580-588, 2009. ISSN 1471-4892. doi: http://doi.org/10..00 haoph.2009.08.000 URL https: /wwww c encedi rect com/ coom/science/art..c.. Anti-infecctives/New technologies.
pii/
Thahir P Mohamed, Jaime G Carbonell, and Madhavi K Ganapathiraju. Active learning for human
protein-protein interaction prediction. BMC bioinformatics, 11:1-9, 2010.
12

---

Published as a conference paper at ICLR 2025
Armaghan W Naik, Joshua D Kangas, Christopher J Langmead, and Robert F Murphy. Efficient modeling and active learning discovery of biological responses. PLoS One, 8(12):88396, 2013.
Emmanuel Noutahi, Cas Wognum, Hadrien Mary, Honoré Hounwanou, Kyle M. Kovary, Desmond Gilmour, thibaultvarin r, Jackson Burns, Julien St-Laurent, DomInvivo, Saurav Maheshkar, and rbyme momatx. 5281/2enoa 8373019.
0.9.4, September 2023. URL https: /ooi. org/ /0.
Mark Plutowski and Halbert White. Selecting concise training sets from clean data. IEEE Transac- lions on neural networks, 4(2):305-318, 1993.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quan- tum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1):140022, Aug 2014. ISSN 2052-4463.
doi: 10. (038/sdata.2014.30 URL. https:////// :/ctpp:://di... http//cc.oorrr https/////i.o/////////// wo..o////////0..0
sdata. 2014.22.
Daniel Reker. Practical considerations for active machine learning in drug discovery. Drug Discov- ery Today: Technologies, 32:73-79, 2019.
Daniel Reker and Gisbert Schneider. Active-learning strategies in computer-asisted drug discovery. Drug discovery today. 20(4):458 465, 2015.
Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, 52(11):2864-25 2875, 2012. doi: 10. 10.1021/ci300415d URL
https: /ttp///oiii.a /ww///////0/ /0t0////111111111/01101000111111 PMID: 23088335.
Kaziwa Saleh, Sándor Szénási, and Zoltán Vámossy. Occlusion handling in generic object detection:
review. In 2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pp. 000477-000484, IEEE, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 170.:00488 2017.
Justin S Smith, Ben Nebgen, Nicholas Lubbers, Olexandr Isayev, and Adrian E Roitberg, Less is more: Sampling chemical space with active learning. The Journal of chemical physics, 148(24), 2018.
Hampus Gummesson Svensson, Esben Jannik Bjerrum, Christian Tyrchan, Ola Engkvist, and Morteza Haghir Chehreghani. Autonomous drug design with multi-armed bandits. In 2022 IEEE International Conference on Big Data (Big Data), pp. 5592. IEEE, 2022.
Maciej Sypetkowski, Morteza Rezanejad, Saber Saberian, Oren Kraus, John Urbanik, James Tay- lor, Ben Mabey, Mason Victors, Jason Yosinski, Alborz Rezazadeh Sereshkeh, et al. RxIxl: A dataset for evaluating experimental batch correction methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4284 4293, 2023.
Maciej Sypetkowski, Frederik Wenkel, Farimah Poursafaci, Nia Dickson, Karush Suri, Philip Frad- kin, and Dominique Beaini. On the scalability of gnns for molecular graphs, 2024. URL https: waw.vvvvvv.. org/ wtt./ass////oooooooooooooooooaar /2/os/2404 2404 11568.
Gary Tom, Stefan Schmid, Sterling G. Baird, Yang Cao, Kourosh Darvish, Han Hao, Stanley Lo, Sergio Pablo-García. Ella M. Rajaonson, Marta Skreta, Naruki Yoshikawa, Samantha Corapi, Gun Deniz Akkoc, Felix Strteth-Kalthoff, Martin Seifrid, and Alán Aspuru-Guzik. Self-driving laboratories for chemistry and materials science. Chemical Reviews, 124(16):9633 9932)9993333399....-3 2024. doi: 10.102 10..000/acsccheeeee..oo.. www.eee...00005..... URL https:: /tts:///do..e http:///ooooogg//.1110 wtt.//.../.....//......o. a chemrev. 4c00055. PMID: 39137296.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training, 2021. URL
https: //tp///a.vvvv///////////ee org/
abss abs 22000 03404.
13

---

Published as a conference paper at ICLR 2025
Manfred K Warmuth, Jun Liao, Gunnar Rätsch, Michael Mathieson, Santosh Putta, and Christian Lemmen. Active learning with support vector machines in the drug discovery process. Journal of chemical information and computer sciences, 43(2):667-673, 2003.
Mary Jo Wildey, Anders Haunso, Matthew Tudor, Maria Webb, and Jonathan Connick. Chap- ter five high-throughput screening. In Robert A. Goodnow (ed.), Platform Techmologies in Drug Discovery and Validation, volume 50 of Annual Reports in Medicinal Chemistry, pp. 149-
195. Academic Press, 2017. doi: cctm://seeeereeearr..c.pst////// / //www //www. sciencedi rect. com www.//ccincaacccccccccccccccoccc
004. URL https:
Dongru Wu. Pool-based sequential active learning for regression. IEEE Transactions on Neural Networks and Learning Systems, 30(5):1348-133) 1359, 2019. doi: 10.11099TNNLS.218.... 28..286644..
Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs, 2021. URL https arxiv. org/ abs /2100. 01717.
Kai Yu, Jinbo Bi, and Volker Tresp. Active learning via transductive experimental design. In Pro- ceedings of the 23rd International Conference on Machine Learning, ICML '06, pp. 1081-1088, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10. 143844. 1139980. URL https: / doi.
1143980.
Xueying Zhan, Huan Liu, Qing Li, and Antoni B Chan. A comparative survey: Benchmarking for pool-based active learning. In IJCAI, pp. 4679-4686, 2021.
14

---

Published as a conference paper at ICLR 2025
A THEORETICAL SUPPLEMENTS
A. PROOF OF LEMMA I
For completeness, we start by re-stating Lemma 1 in more detail.
We consider a multi-class classification problem, with K. := {1, ,I, K} the set of K labels and p(x:) =- (f) (zr.....
px(x)) ......pppsfa the predicted probabilities for that y for all & E K. We denote Me the true label for a sample T;, Di = arg max, ()(();) the predicted class by the model and dy == maN P& (z:) the predicted probability value for that class. the context of sequential label acquisition, we denote Mf = {a the inference set containing N samples acquisition step t I, and D = { y: > the same set with the associated labels y,- Stepping forward, at time f, an acquisition batch of N₂ samples and their labels Dj == {(T; 11) are selected from the inference dataset of the previous step 1
Following the least-confidence acquisition function, at time-step t, the samples in D are ordered by confidence where 2i has the predicted class with the lowest confidence and z has the predicted class with the highest confidence:
022
S UN
GNa+1
ON
lirst Ni, samples
last - N2, samples
and the selected batch consists of the first N₂ samples. The remaining inference set is composed of the remaining N - N, samples, i.e. Dí == Dir Dj ((ri, v.) vc)
The practical utility of this ordering depends on the calibration of the model. Calibration reflects the correctness of the model's confidence compared to its true performance. A perfectly calibrated model P would output confidence levels that perfectly match the true probability of correct classifi- cation. We extend the notation of Guo et al. (2017) to be dataset-specific, and denote Pp[Y Y] the probability of P producing the correct class prediction for any sample from dataset D, which can equivalently be thought of as p's accuracy on that dataset:
Pp|Y = 81 == Em ( = 9:)]
Similarly, Pb(Y = 818 denotes the confidence-condition accuracy of the model for 0, a specific
confidence level?:
PolY = 8| = E (1 = 0:) T)
(
Perfect calibration on the inference set would thus imply that for any confidence level 0 E [o, 1], we have P = 8 = ö. Instead, in Lemma 1, we only assume weak calibration of the model on the predicted class, i.e. that the confidence ordering of the model's top predictions reflects the ordering of probabilities of correct classification. Formally, weak calibration on D implies that inf
for any pair of samples 24 and xy in our dataset inf we have:
6 < 02
PD 8l61 Pp--il- =|0, an
(9)
We aim to prove that, assuming weak calibration on the inference set, the expected accuracy on the least-confidence batch is bounded by the expected accuracy over the remaining inference set:
P IY
V1 S Poy I =Y
(10)
Proof. The expected accuracy from Equation 7 be rewritten in terms of its confidence- conditional form in Equation 8:
D Σ Poly
1
Pbl 1 D
-
(11)
From the weak calibration assumption in Equation
follows that the model's confidence- conditional accuracy is a non-decreasing function of its confidence. Combined with the acquisition
II practice such confidence levels would be binned together such that Pb \1 represents the accu- racy over all samples in D for which the model's confidence is in j0 c, ê ej. see (Guo et al., 2017).
15

---

Published as a conference paper at ICLR 2025
mechanism of Equation 6, obtain ordering of the probability that the model obtains the correct label along the entire inference set at step t 1:
P P I = 11111 S ... S P P = Yl I P y MON++l S ... P Y N
Nyfirst
N-Ns remaining terms
(12)
The empirical average of set of numbers that are inferior or equal to those of second set has to be inferior or equal to the empirical average of the second set:
No P D'r
PY-
-100 N No i=N++1
Σ
PY 1|0]]
(13)
No
By definition of D == from DJ and the right-hand side those from Di Rewriting each side using Equation 11, obtain:
inf Djj the left- hand side from Equation 13 thus captures all the terms
PDy 1 PDy Y Y=
(14)
This concludes the proof.
A.2 BOUND DERIVATION
Here we derive the bound presented in Equation 5. The Chernoff bound (Chernoff, 1952) for Bernouilli random variables provides exponential tail bound for the true mean μ of sequence of Bernouilli distributed random variables Z1, Zz,.... .2.....,,, Zn. It can be expressed using the Kullback- Leibler Divergence (KL) for Bernouilli distributions (Lattimore & Szepesvári, 2020, page 135, Cor- rollary 10.4):
P(u S a) S exp n KL(Alla)) a E [0, 1 j a
(15)
with KL(()la) == á log + (1 (b) log
(16)
quantifies the probability that the true mean μ is smaller or equal to some bound a given the observed sample mean and sample size For us, the value {0, 1} of variable 7, indicates whether a particular sample H; was correctly classified i.e. 7; = (v: = ûi). In the context of inference design and following the notation from Section 2. pi represents the observed accu- racy on the acquired batch at time-step f, with acquisition batch-size N₁. and sint represents the (unknown) accuracy of the model on the remaining inference set. Note that the Chernoff bound usually assumes that jij is an unbiased estimator of Mar In our case, Pf is a biased estimator due the active selection mechanism of the batch. However, in Lemma 1, we show that this estimator is actually conservative estimate of the true accuracy, and thus in turn contributes making the bound presented Equation 15 even more conservative, and preserves its validity.
To establish confidence level on that bound, we can lower-bound the right-hand side itself to the desired bound-failur probability ó, which after rearranging yields:
log(E) No
KL(AIla)= Z
(17)
At time i, we thus seek the maximum bound value a = or for phat such that the inequality on Equation 17 holds by finding the value of that satisfies the following condition:
a: = min { { [6] KL(A]la) <
log(t)
(18)
No
Since this is scalar optimization problem from closed form expressions, computing an can be done easily and efficiently using grid-search. With this choice for a, we obtain the desired probabilistic bound P( < ar) summarized in Equation
16

---

Published as a conference paper at ICLR 2025
B ALGORITHM
Algorithm Hybrid Screen using Inference Set Design
1: Input: Acquisition batch-size No, threshold 7, margin
2: Initialize Step t=0, observation set X + 0, inference X Rungets predictor p.
3: repeat
4: Train the predictor on (X, 3/) (if not empty)
5: Obtain the predictions on the inference set Pi - {f(a;) Vots E Xmr)
6: Run acquisition function to obtain scores S g (P(na)
7:
Select a batch of N₂ inputs with the highest scores S to form X.b.
8:
Remove the acquired batch from the inference set X +- Anf Rb
xt-1
9:
Obtain the true labels 2% for the acquisition batch
10:
Append the acquired batch to the observation set (X, + (K, (
11:
Compute on (X') y from Equation 5
12: until
Roos + a Nin
or X = 0
Kurrett
13: Return hybrid screen readouts:
(1 U (X..)
C ADDITIONAL DETAILS ON DATASETS AND PREPROCESSING
C.I MOLECULAR DATASET PREPROCESSING
In many practical applications, exact geometries screened molecules are unknown as they require computationally expensive DFT calculations. As first data processing step, we use the RDKit (Landrum et al., 2024) and Molfeat (Noutahi et al., 2023) libraries to convert molecular structures into SMILES strings and compute their Extended Connectivity Fingerprints (ECFPs). Both molec- ular datasets are cleaned by removing duplicated SMILES and fingerprints structures. For total energies the Molecules3D dataset we use a reference correction technique where atomic energies are calculated using linear model fitted to the counts of atoms of each type present molecule (the obtained atomic energies are presented Table 1). For reference correction, randomly selected sample of 100k molecules is used. The atomic energies then subtracted from the total energies all molecules in the dataset. The obtained referenced-correcte energies normally distributed with mean around 0 eV.. A small number of outliers with reference- corrected energy values above 10 standard deviations are removed from the dataset as well as the 100k samples that were used for reference correction avoid data leakage.
well as single-atom
The final QM9 and Molecule3D datasets contain 133, 885 and 3, 453, 538 molecules respectively. Both datasets split into inference, validation, and test with 80%, 5%, 15% fractions. The QM9 HOMO-LUMO gap values are discretized into 2 balanced classes using median as a boundary condition explore agents' performance on a classification task.
Atomic Number
Energy (eV) -26.765
Atomic Number
Energy (eV) -7922.180
14
-673.550
-9313.610
6
-1054.411
16
-10810.322
-1483.001
-12474.680
8
-2034.056
32
-56538.66 6.47
-2687.455
33
-60828.58
12
-5367.759
34
-65296.349
13
-6657.476
35
-69980,303
Table 1: Atomic energies used for reference correction on Molecules3D dataset.
17

---

Published as a conference paper at ICLR 2025
C.2 RxRx3 DATASET PREPROCESSING
The RxRx3 dataset contains one embedding for each well. Each perturbation type (gene-guide pair componn--coneentaatoon pair) has several replicates across wells, plates and experiments. Each plate also contains unperturbed control cells which are used to keep track of and eliminate portion of the batch effects (Sypetkowski al., 2023). These raw embeddings thus need to be aligned and aggregated. We align them by centering and scaling each perturbation embedding to the embeddings of the experiment-level unperturbed control wells. The embeddings are then aggre- gated through a multi-stage averaging procedure, across wells, plates, experiments and guides (for CRISPR perturbations), which yields an average embedding for each cene-perturbation and each compound-conceetraaiio perturbation. We then use the obtained embeddings compute the cosine similarities between gene and compound perturbations in the RxRx3 dataset.
D HYPERPARAMETERS AND IMPLEMENTATION DETAILS
For all experiments presented in this work we use MLP models with residual connections (Touvron et al., 2021). All experiments were repeated with 3 different random seeds and the hyperparameters are summarized in Table 2.
Hyperparemeter name
MNIST
QM9
Molecules3I
RxRx3
Proprietary
data
Acquisition batch size
1,000
250
10,000
10
1000
Number of hidden layers
2
3
2
2
2
Hidden layer size
512
512
512
512
1024
Learning rate
0.001
0.001
0.001
0.001
0.001
gradient norm clip
1.0
1.0
1.0
1.0
1.0
Dropout
0.1
0.1
0.1
0.1
0.1
Train epochs
1,000
1,000
30
30
1000
Train batch size
1,024
1,024
32,768
1,024
1,024
Early stop patience
50
50
15
25
25
Number of ensemble members
5
Table 2: Hyperparameters for experiments.
18

---

Published as a conference paper at ICLR 2025
E ADDITIONAL RESULTS
E.1 REGRESSION TASK ON MOLECULES3D
To evaluate the inference set design paradigm on a regression task we use the Molecules3D dataset (Xu et., 2021). Molecules3D contains structures and DFT-computed properties of nearly million molecules. In our experiments we aim to predict the HOMO-LUMO gap and total energy. For inputs, we convert the SMILES strings molecular representations into their Extended Connectivity Fingerprints (ECFPs) (see Appendix C. for all details).
For this regression task, we use query-by-committer (QBC) active learning approach that computes variance across the predictions of an ensemble. To determine the stopping time we use a criterion with two parameters: MSE threshold tMSE on the acquired batch and patience p. The stopping time is reached if the acquired batch MSE is lower than tMS for steps. Like for our QM9 experiments, in addition active and random agents, we also evaluate the performance of heuristic- based acquisition orderings (molecules ordered by size, sorted by SA-score, etc.). QBC achieves an approximately five times lower MSE compared to the random agent or heuristic-based orderings (see Figure 9). This shows that inference set design approach is not limited to classification tasks and can be applied to regression problems.
System Performance
Acquired batch
Test
QBC
Savings:
Bandom agent
Sevings:
SA-scone agent
Ordered
33
16
large
Drdered
small
20%
49%
4B%
X0%
JEHM
sM
1000
ca 2PM are 50%
100%
Percent of data acquired, 4, 56
Percent of data acquired, 56
Percent of data acquired,
Percent of data acquired, %
System Performance
Inference set
Acquired batch
Test set
DBC
Savings:
MSE. ove
107
Random agent
Sawings:
St-score agent
Ordered targo Ordered
Lange
63% 90% Percent of data acquired, %
100
Des Percent of data acquired. 16
25% 40% Percent of data acquired. %
Percent of data acquired, %
Figure 9: Performance of QBC, random, and heuristic agents for HOMO-LUMO gap prediction (top) and energy prediction (bottom) on Molecules3D dataset. The dashed vertical shows average stopping time across random seeds, surrounded by mix to max intervals (left to right). The patience parameter = 10 was used in all experiments with the threshold MSE tMSE 0.leV2 for HOMO- LUMO gap and tMsE 100eV² for Energy predictions. QBC agent satisfied stopping condition after acquiring zs 80% of the data.
Although, in many applications predicting properties of larger molecules presents more challeng- ing task, our experiments on the Molecules3D dataset demonstrate that acquiring molecules ordered from large to small may harm the predictions on inference set and overall system performance (see Figure 9). One of the reasons is the distribution of chemical elements across molecules in the dataset. When acquiring molecules from small to large, all unique chemical elements of the Molecules3ID dataset are present in the training set after acquiring just the first 1,000 samples. However, when acquiring molecules from large small, some chemical elements remain only in the inference set until the very end of the experiment which is especially detrimental to the total energy predictions (see Figure 10). This result demonstrates that using heuristic rules such ordering molecules by size for data acquisition does not guarantee optimal acquisition or generalization of heuristic rules new data and instead risks harming the performance due the introduction of such biases.
19

---

Published as a conference paper at ICLR 2025
unique chemical elements acquired
14
10
Molecules ordered small to large
Molecules ordered large to small
2
10"
102
102
N molecules acquired
Figure 10: Number of unique chemical elements in the training set when acquiring samples ordered by molecule size. Acquiring molecules from large small leaves several chemical elements out of the training set until the end of the dataset.
E.2 CLASSIFICATION TASK ON RxRx3
Our results on the RxRx3 dataset demonstrate minor improvements from the active agents (LC and BALD) compared to random selection. The accuracy the inference and test sets remains low throughout the experiment regardless of the acquisition method. This is unsurprising considering the extreme difficulty of predicting biological relationships in low data regime. The result suggests that in the setting with low predictive power, even when the majority of the data is acquired, the ability of inference design provide significant budget reductions is limited.
System Performance
Inference set
100
SA-score
Accuracy,
80
Accuracy,
20
60
60
LC sgent
SAvscore agent Divensiy agent
BALD agent
Rondom agont
0%
20%
40%
60%
80%
100%
0%
2015
45%
63%
80%
100%
Percent of data acquired, %
Percent of data acquired, %
Acquired batch
Test set
130
agent
Sieecare apert
100
agent
Sivacore agent Diversity egent
BALD agent
Diversity agent
BALD agent
90
Random agent
50
Random agent
Accuracy,
Accuracy,
70
70
50
50
0%
40%
60%
100%
20%
40%
80%
100%
Percent data acquired,
Percent
data acquired,
Figure 11: Performance agents on pheno-similarity classification task on RxRx3.
20

---

Published as a conference paper at ICLR 2025
F ADDITIONAL ANALYSIS
F.1 EMPIRICAL VALIDATION OF WEAK CALIBRATION ASSUMPTION
In Lemma 1, we show that when using least-confidence acquisition function, at any time-step t, the measured accuracy in on the acquisition batch is a lower-bound on the unobserved accuracy on the remaining inference set Mar assuming that the model ý is weakly calibrated the inference set. A weakly calibrated model is such that increased confidence translates to a higher likelihood of correct prediction (higher accuracy). In Figure 12, we can see that at several points throughout the active learning loop, the least-confidence based agent is not perfectly calibrated. Indeed, there is substantial gap between its confidence levels and the true accuracy (seen when comparing the col- ored bars to the identity function shown in gray). However, the model is generally weakly calibrated (the colored bars are always increasing). These observations support the empirical validity of our assumption for Lemma 1.
Model Calibration on Inference Set
Outnue
Accuracy
612 Confidence
309
Die Confidence
333
AM
ki
o Confidence
219 Confidence
Confidence
Confidence
Go
Accuracy
Cropped
231 Outeut
Confidence
Confidence
Oulpet
Accuracy
Shufted
Confidence
Confidence
Confidence
Confidence
Confidence
Confidence
Darpot
Dutpel
Duter
Dase
Accuracy
209
650 Confidence
Confidence
Confidence
Confidence
Confidence
Confidence
Phenomics
Accuracy
Taee
S-I.
239
600
1-4
USe 5.A
USA U/6
AJ9
UCU Confidence
Confidence
Confidence
Confidence
Confidence
Confidence
Figure 12: Model calibration analysis for a run of the least confidence agent at different active learning steps (columns) for all of our classification experiments (rows). The colored bars represent the accuracy of predictions binned w.r.t their confidence level, and the gray bars show the identity function illustrating what perfect calibration would look like. The models satisfy the condition of being weakly calibrated since the accuracy of the model's predictions increases monotonically with their confidence. The confidence distribution shifting to the right as increases indicates the growing confidence of the model in correctly predicting the labels of the remaining examples in the inference set.
21

---

Published as a conference paper at ICLR 2025
F.2 EMPIRICAL VALIDATION OF STOPPING CRITERION
In Section 2.2, we present a stopping criterion based on the fact that the accuracy measured on the acquisition batch can be used as a proxy for the accuracy on the remaining, non-acquired samples (inference set). The criterion is simple, once the estimated system accuracy isys at time (see Equation 4) is above a user-defined targeted threshold ", the acquisition stopped. This criterion makes use of the bound of Equation 5, and both random sampler and a least-confidence based agent can use this bound in principled way. The random agent can validly use it because the accuracy of its acquisition batches, uniformly drawn from the inference set, are unbiased estimates of the true accuracy on the entire inference set, which is typically required for such bounds. The least-confidence agent can use it validly because we show in Lemma that, assuming that the model weakly calibrated (which we empirically validate in Appendix F.1), because of its confidence- based acquisition function, the accuracy of its acquisition batches represent a lower-bound on the true inference set accuracy, making the bound of Equation 5 even more conservative.
To empirically validate this bound (Equation 5), confidence (LC) and random agents across the classification datasets used in the experiments section. The results show that for the LC agent, all trials ended at stopping time t system accuracy Ksys was above the threshold 7, which is in accordance with the fact that the LC agent uses lower-bound estimate in place of unbiased estimate for the inference set accuracy, resulting in a looser bound for Equation 5 and an actual callure-probabilee random agent, we observed 1% of the trials end with a system accuracy below the threshold in one of the experiments, 6% in another, and 0% the three remaining datasets. 'hese results are in accordance with the theory presented in Section 2.2 and Appendix A.2. On QM9, out of 100 experiments resulting in slight bound failure is statistically in accordance with the theoretical bound failure probability of 8 5%. For the other datasets, the bound was looser, which is also possibility, and showcases that the observed gap between å and the true bound-failure probability can be mroblem-dependent.
large number of trials for both the least-
which the true
lower than 8. For the
LC agent
Random agent
Threshold y
1.00
System Accuracy at stopping time Mas'
0.99
0.98
0.96
0.95
MNIST
Cropped MNIST
Shuffied MNIST
QM9
Phenomics
Figure 13: System accuracy stopping time observed across 100 seeds for the MNIST, Cropped MNIST, Shuffled MNIST and QM9 experiments, and 50 seeds for the Phenomics experiments. In all experiments, we
bound-failure probability of 8 0.05. For the LC agent, all trials lead to a stopping time t = т at which the true system accuracy f higher than the threshold y. For the Random Agent, we observed the same thing for Cropped MNIST, Shuffled MNIST and Phenomics. We also observed 1% of bound failure for (regular) MNIST, and 6% for QM9.
22


Reviews:
[{"review_id": "HzjRWYidub", "reviewer": "Reviewer_Et7D", "summary": "The paper describes the application of active learning for Inference Set Design, a problem where the goal is to label a predefined finite set of instances. To this end, a hybrid screen is proposed where experiment carried out on a subset of the instances (the observation set) and the label on the rest of instances (inference set) is approximated by the prediction of a model trained on the observation set. The paper shows that in this setup the inference set design aspects dominate the better model generalization aspects of active learning.", "questions": "1) Can you point out a few practical scenarios where ISD is relevant while Bayesian Optimisation not? (see Weaknesses)\n2) How can the method handle measurement error?\n3) Were you able to see any meaningful pattern in the hard to predict examples in the public dataset chemical problems? (HOMO-LUMO gap and energy prediction)\n4) Pleas fix page limit, and the minor issues.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The described problem is very close to industrial application, and as such quite relevant.  The paper is clearly written and quite easy to follow. The experiments are quite detailed.", "weaknesses": "The machine learning novelty is limited, it describes classical active learning evaluated with a non traditional performance metric. (This metric is however quite well motivated). The argument by the authors, that the benefit of active learning should be interpreted more broadly if the goal is to label the dataset is valid. Using similar argument, however, most of the time the final goal is to maximize (target) or minimize (off-target, tox.) the outcome of the experiment (at least find multiple diverse extrema), the problem become a Bayesian optimization problem, with already existing literature and efficient methods. In the light of this, can you point out a few practical scenarios where ISD is still relevant?\n\nThe paper describes the goal as getting the labels cheaper, so if all measurement would be available Accuracy would be 100%. This assumption of no measurement error (zero aleatoric uncertainty) is not reasonable in practical biological assays. At first glance reformulating the problem with measurement noise is not trivial. For example, sometimes it may be beneficial to spend the budget on a repeat if the model flags it as “suspicious” (model based on other data consistently predict different value than the measured one) \n\n**On page limit**\nThe manuscript is over the page limit though not by much, as on page 5 a page break is left in. Please fix this.\nalso 392-393 line break should be removed\n\nMinor:\n\nLine 402 “We have shown that inference set design that inference set design”\n\nL568-569 “With inference set design, the active agent is again able to outperform thebaselines (see Figure 10). ” <- this should be Figure 9 right?\n\nDefine $KL(\\hat{\\mu}_t, a)$ in eq 5", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkDdqceygi", "reviewer": "Reviewer_i8MT", "summary": "The paper presents an active learning method for hybrid screening, for the setting where there is a finite library and the goal is performance on this library (and not generalization). The idea is to use an active learning approach to include the difficult examples in the training set and stop when the performance is beyond some threshold. The paper discusses two reasonable acquisition functions for this purpose. In each round, a batch of difficult observations are labelled, and the performance on this batch is used to lower bound the performance on the remaining data in the inference set. The method is demonstrated on several relevant benchmark data sets.", "questions": "What are key differences to existing uncertainty based active learning methods? It appears to me, that the main difference is that the evaluation criterion is performance on the library and not generalization.\n\nIn what practical sense does the stopping criterion provide theoretical guarantees? The paper argues that there is a lower bound on target accuracy, assuming the model is weakly calibrated (i.e., the order of uncertainty is correct). However, how reliably does this assumption hold in practice?\n\nFor the committee approach, why do you focus on the voting entropy, rather than e.g. entropy of the mean predictive distribution?\n\nI did not find a description in the main text of the ML models used in the experiments.\n\nIf I understand correctly, on Molecules3D you include a comparison with data ordered by molecule size, smallest first. Does this not go against the idea of training on difficult examples first? Would it not be more interesting to compare with data ordered by decreasing molecule size? \n\nNotation:\n\nIn eq. 3 Nb is not defined, and the notation is not clear. The left hand side has no index i inside the brackets, and the right hand side evaluates to a single value, not a set. Same goes for eq. 4.\n\nMinor comments:\n\nThis sentence is unclear to me: \"Because of this, we can treat performance on X_inf^t as though they were IID samples from X_inf^t, and construct a (conservative) lower bound on the accuracy on X_inf.\n\nI am not sure why the method is called \"inference set design\" - it does not explicitly design the inference set, but is rather a fairly standard active learning scheme with a particular focus on optimizing system performance by including difficult examples in the observation set.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well written and easy to follow.\n\nThe main idea is clear, useful and effective.\n\nGood experiments to demonstrate the benefits of the approach.", "weaknesses": "The technical novelty is limited, as it is mostly an application of existing methods.\n\nThe paper would benefit from including comparisons with other heuristics that capture the \"difficulty\" of observations. For molecules, one could think of many ways to heuristically score their complexity, and use this to train on most complex examples first. I think this would be of practical interest to know how much the proposed method can improve over such baselines.\n\nThe paper appears to have formatting issues, with a mostly blank page and a full page of text exceeding the page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nv8BC4vf0T", "reviewer": "Reviewer_VGSj", "summary": "This paper tackles the challenge of reducing experimental costs in high-throughput biological screening, with a focus on real-world applications such as drug discovery and cell microscopy. To achieve this, the authors formulate it as a sequential subset selection problem, which seeks to identify the smallest possible subset of compounds for experimental testing, whose outcomes can ensure a desired level of accuracy for the entire system. In other words, the authors propose a mechanism that aims to actively identify and experimentally test the most challenging compounds, leaving the relatively easier-to-predict compounds for the inference set. To strategically label only the most challenging compound samples, the authors propose to employ an uncertainty-based active learning approach that selects samples for which the model shows the lowest predicted probabilities or where ensemble models exhibit the highest disagreement.", "questions": "1. What is $N_{\\text{target}}$ on page 2? The notation appears without explanation, leaving its meaning unclear.\n2. In Lemma 1, the inequality $P[\\hat{Y} = Y \\mid \\hat{p} = p_1] > P[\\hat{Y} = Y \\mid \\hat{p} = p_1]$ is used, but since both sides are identical, the inequality does not make logical sense.\n3. In the proof of Lemma 1, the notation $i \\in\\in\\mathcal{X}^{t}_{\\text{inf}}$ is unclear. Additionally, the step claiming “the inequality follows from weak calibration and least confidence’s selection” does not seem justified, as it appears to have logical gaps and doesn't rigorously establish the claimed inequality. The overall proof structure needs a more detailed explanation to establish the claimed result.\n4. On page 4, lines 212 to 213, this assertion seems to be logically circular, and the abbreviation 'IID' is used without definition. Although 'IID' is a common term, it should be defined in full upon its first appearance in the paper.\n5. On page 6, the authors distinguish their approach from Bayesian methods but only compare it with a random agent. However, established Bayesian methods like Bayesian Active Learning by Disagreement (BALD) [1] can effectively select samples that are expected to reduce uncertainty across the dataset. The authors should explain why Bayesian active learning methods cannot be applied in this context and clarify their decision not to compare their approach with BALD or other active learning or reinforcement learning methods.\n6. If the method's effectiveness is constrained by a 'low data regime and complex task' (as indicated by the RxRx3 dataset results), how can it be practically useful for real-world biological applications that commonly encounter these challenges?\n\n\n\n\n[1] Houlsby, Neil, et al. \"Bayesian active learning for classification and preference learning.\" arXiv preprint arXiv:1112.5745 (2011).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of this paper is derived from addressing an interesting real-world challenge in biological screening, where experimental costs are one of the most significant bottleneck. The proposed inference set design method shows somewhat promising potential for these applications by enabling a strategically targeted selection of samples based on model uncertainty. Another strength is the use of diverse real-world datasets, ranging from molecular property prediction (QM9 and Molecules3D) to cell microscopy experiments (RXRX3 dataset with 17,063 CRISPR knockouts and 1,674 FDA-approved compounds).", "weaknesses": "The most significant weaknesses of this paper are both technical and presentational. First, this paper does not meet basic ICLR submission guidelines, as its length exceeds the 10-page limit, and this paper contains formatting issues, including a large blank space on page 5 and broken text formatting on lines 392-393 of page 8, which diminishes its professional presentation. Secondly, the experimental comparison is severely limited, with only a random agent used as a baseline method. Given the paper's relevance to active learning and sample selection, the absence of comparisons with reinforcement learning (RL) approaches or other active learning methods is a significant shortfall. Specifically, an RL agent with proper reward modeling could potentially offer competitive performance in sample subset selection. Lastly, the theoretical explanation of Lemma 1 lacks clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tDNbWFQcx", "reviewer": "Reviewer_Nxf3", "summary": "The paper introduces and tries to tackle the problem of identifying a small subset of data on which to run costly measurements to train a machine learning model which infers the most probable measurements for the remainder of the data, with the theoretical guarantee that the model makes less of $\\gamma$ percent of errors at inference time. The paper introduces a stopping criterion that would satisfy the correctness guarantee and applies their corresponding algorithm on several benchmark and real-life data sets.", "questions": "Typos:\n- Is “weakly calibrated” (page 2) the same as “approximately calibrated” (Lemma 1)?\n-  In Lemma 1, $P[\\hat{Y} = Y \\mid \\hat{p} = p_1 ] > P [\\hat{Y} = Y \\mid \\hat{p} = \\underline{p_2} ]$ for $p_1 > p_2$\n- In Equation (5), please mention that KL is the Kullback-Leibler divergence… also mention the name of the result (Corollary 10.4) instead of the page.\n\nQuestions:\n- What does “treat performance on $X^t_\\text{inf}$ as though they were IID samples from $X^t_\\text{inf}$” (Line 213) mean? Independence of the probabilities given by the model across data samples from $X^t_\\text{inf}$?\n- **(Score-changing question)** Please provide a proof for Lemma 1 that solves my concerns listed above.\n- **(Score-changing question)** Is there a closed-form expression or at least an upper bound on the stopping time $\\tau$ in Equation (6) (as the whole selection process is deterministic as long as the prediction model is deterministic)? If so, is the subsequent budget of measurements (that is, $|X^t_\\text{obs}|$ where t is the final selection step of the algorithm) realistic with respect to the total number of items (for instance, we might want a number of measured elements sublinear in N for a reasonable value of $\\gamma$, where N is the total number of items in the data set)?\n- **(Minor question)** At each time t, the model $\\hat{f}^t$ (or $\\hat{f}^t_1, … \\hat{f}^t_{Nmodels}$ for the query-by-committee) has to be retrained on the full $X^t_{obs}$ and evaluated on $X^t_{inf}$, and then take the arg max on $X^t_{inf}$ (least confidence sampling and query-by-committee). How expensive is it in practice? The budget of measurements ($|X^t_{obs}|$) is important but perhaps we would like also to minimize the number of selection rounds (that is, $t$) as training/evaluating/sampling might also be expensive. Minimizing $t$ might however be contradictory with the objective of minimizing $|X^t_{obs}|$ though, as if we measure a slight over-approximation of the minimal $X^t_{obs}$ in a single selection step, we optimize for the former objective but not the latter.\n- **(Minor suggestion)** Why not a more adaptive approach where all points such that the maximal proba across class labels is no higher than 0.5 (and then the acquisition batch size is no longer a constant)? \n- **(Score-changing questions)** (1) There is a shadowed orange part on the plots in Figure 1, are the curves the average across all considered points (System performance: all points in MNIST training set = $|X_\\text{target}|$, inference set $|X_\\text{target}|-|X^t_\\text{obs}|$ -that changes across the x-axis- acquired batch $1,000$, and test set = 50% of full data set) or the average across random seeds (=3 according to the appendix)?\n\n(2) What the “saving” mentioned in Figure 1? The difference between the number of actually measured points and measuring all samples at a given point in time?\n\n(3) What’s the threshold $\\gamma$ used in the experiments for the stopping criterion? Can you check that the theoretical guarantee of Lemma 1 is empirically satisfied (by applying some calibration procedure on the trained model, if needed)?\n\nI understand that not all score-changing questions can be answered, especially in the short time for rebuttal. I would be willing to increase the score whenever one of these can be answered (the most important being Lemma 1 and the ones related to the plots) / if the major concerns listed above are addressed.\n\nSummary of the suggested modifications:\n- Provide a clear and consistent definition of $b$ (or $N_b$) early in the paper, ideally in a dedicated notation section.\n- Ensure that this definition is used consistently throughout the paper, including in equations, lemmas, proofs, and pseudocode.\n- Explicitly state earlier in the paper that a constant number of samples is added to the measured point set in each iteration.\n- Clearly define all variables used in the proof of Lemma 1, including $\\hat{Y}$, $Y$, $\\hat{f}$, and $\\hat{p}$, and explain their relationships.\n- Revise the proof to ensure consistency with the statement in Lemma 1, particularly regarding the inequality sign.\n- Provide a step-by-step explanation of the proof, including justifications for each step (e.g., the use of Bayes' theorem).\n- Clarify the induction process, especially the step from T-1 to T-2.\n- Explain or justify any assumptions made in the proof of Lemma 1, such as \"$P(\\hat{p}=\\hat{f}(X))$\"=1.\n- Include empirical tests of Lemma 1 in the experiments, specifically verifying that the error is indeed less than $\\gamma$ with probability $1-\\delta$ at the stopping point.\n- Provide access to the code used for the experiments, either through a public repository or as supplementary material.\n- Include more details about the experimental plots.\n\n------------------------\n**UPDATE**: I raised my score to 8 after reading the updated manuscript and exchanging with the authors.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- Originality: The submission tackles an interesting problem, where the goal is not to maximize model generalization but to ensure model correctness on a specific subset of data. This is a first step towards better theoretically-backed approaches in biology (compared for instance to what was done in the LINCS L1000 data set [1] for the regression of gene expression, where they only selected a fixed subset of genes to measure based on prior knowledge instead of difficulty in predicting gene expression).\n       \n- Quality: The experimental results look good, with appropriate baselines (random selection of new samples to measure, or adding a next batch to measure given the original order of samples). There is a good effort in applying the method to benchmark and real-life, biological data.\n       \n[1] Subramanian, Aravind, et al. \"A next generation connectivity map: L1000 platform and the first 1,000,000 profiles.\" Cell 171.6 (2017): 1437-1452.", "weaknesses": "- Clarity: **(Major concern)** The full paper is not very well-written: \n\n(1) The definition of $b$ (or $N_b$ in the pseudocode in appendix) is unclear and never defined. In Equation (5), it seems that $b=|X^t_\\text{obs}|$ that is, the total number of points measured up at time $t$. \n\nIn Lemma 1, $b=|X^t_b|$, which is the total number of new points added to the observed set $X^{t-1}_{obs}$ at time $t$. \n\n$N_b=|X^t_b|$ in the pseudocode in appendix. \n\nIn the proof of Lemma 1 in appendix, $b=|X^t_{inf}|$, that is, the number of points that are left to infer at time $t$ (there is no reason to assume that $b=|X^t_{inf}|=|X^t_b|$...). \n\nMoreover, it should be mentioned earlier than the appendix or the experimental section that a **constant** number of samples is added to the measured point set. \n\n(2) Perhaps a good thing would be to define all variables ($X^t_\\text{obs}$, $X^t_b$, etc.) in a single paragraph just after the introduction. Please don’t remove the time-dependent indices ($X_\\text{obs}$, $\\hat{f}$…) and the hats ($f$ instead of $\\hat{f}$ in the proof of Lemma 1), as it makes everything very confusing.\n\n(3) Why mention the query-by-committee sampling, as there is no theoretical guarantee associated with it?\n\n(4) It seems that $\\hat{f}$ has a range in $[0,1]$ (or $\\mathbb{N}$) whereas $\\hat{p}$ has a range in $[0,1]$. Then writing $\\hat{p}=\\hat{f}(X)$ in the proof of Lemma 1 is extremely confusing.\n\n(5) **(Minor suggestion)** I would put the related works section earlier in the manuscript, for instance, when the differences between active learning and inference set design is discussed, as I think it would make more sense.\n\n(6) **(Minor suggestion)** A one-liner of the full structure of the Algorithm should be present at the end of the section 2 with a link to the pseudo-code in the appendix, so that the algorithmic contribution is clearly mentioned.\n       \n- Quality/Significance: **(Major concern)** As it is, the proof of Lemma 1 is wrong. What is $\\hat{Y}$ ? the difference with $Y$? Is one the prediction made by the model $\\hat{f}$ and the other the true measured label (and then $\\hat{Y}= \\hat{f}(X)$)? However, $y$ seems to be the probability given by the model in Equations (3) and (4). The proof displays the opposite sign of what is stated in Lemma 1 in the main text. Please rewrite properly the proof, including the use of Bayes’ theorem (?) in the second equality, the problem of the definition of the variables $b$, $\\hat{f}$, $\\hat{Y}$, $\\hat{p}$, why “$P(\\hat{p}=\\hat{f}(X))$”=1 (assuming that here $\\hat{f}(X)$ is the probability returned by the model: is it related to the model being accurate on every measured point?), and how the induction works (at least) from T-1 to T-2.\n\nThe result shown in Lemma 1 is not empirically tested (that is, at the stopping point of the Algorithm, with a prespecified probability $1-\\delta$, the error is less than $\\gamma$). The code producing the experiments is not shared, as such, the research is not reproducible.\n       \n- Significance: **(Minor concern)** The choice of accuracy measure is perhaps not very appropriate for the data: for instance, we expect more drugs to be labeled negatively (for instance, in drug discovery) than positively. Area Under the Curve is said to be slightly more appropriate on imbalanced data [2].\n\n[2] Ling, Charles X., Jin Huang, and Harry Zhang. \"AUC: a better measure than accuracy in comparing learning algorithms.\" Advances in Artificial Intelligence: 16th Conference of the Canadian Society for Computational Studies of Intelligence, AI 2003, Halifax, Canada, June 11–13, 2003, Proceedings 16. Springer Berlin Heidelberg, 2003.\n\nThe proof of Lemma 1 being incorrect/badly-written is why I rated soundness to 1. The definitions of notation not being clear to the point of impairing the understanding of the paper is why I rated presentation to 1. The interesting question (mitigated by the perhaps incorrect Lemma 1) is why I rated contribution to 2. The main reason why I rated the paper 1 is because of the proof of Lemma 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)