Reliable Algorithm Selection for Machine Learning-Guided Design
Authors: Clara Fannjiang
Ji Won Park
Venue: ICML 2025 poster (2025)
Keywords: machine learning-guided design
biological sequence design
uncertainty quantification
generative models
model-based optimization
prediction-powered inference
model selection
Primary area: applications->health_medicine
TL;DR: We propose a method for selecting algorithms for successful machine learning-guided design, with theoretical guarantees on the distributions of designs produced by the selected algorithms.

Abstract: Algorithms for machine learning-guided design, or *design algorithms*, use machine learning-based predictions to propose novel objects with desired property values. Given a new design task—for example, to design novel proteins with high binding affinity to a therapeutic target—one must choose a design algorithm and specify any hyperparameters and predictive and/or generative models involved. How can these decisions be made such that the resulting designs are successful? This paper proposes a method for *design algorithm selection*, which aims to select design algorithms that will produce a distribution of design labels satisfying a user-specified success criterion—for example, that at least ten percent of designs’ labels exceed a threshold. It does so by combining designs’ predicted property values with held-out labeled data to reliably forecast characteristics of the label distributions produced by different design algorithms, building upon techniques from prediction-powered inference (Angelopoulos et al., 2023). The method is guaranteed with high probability to return design algorithms that yield successful label distributions (or the null set if none exist), if the density ratios between the design and labeled data distributions are known. We demonstrate the method’s effectiveness in simulated protein and RNA design tasks, in settings with either known or estimated density ratios.

Reliable Algorithm Selection for Machine Learning-Guided Design
Clara Fannjiang'
Ji Won Park
Abstract
Algorithms for machine learning- guided design, or design algorithms, use machine learning- bausing-based predictions to propose novel objects with desired property values. Given a new design task-for ex- ample, design novel proteins with high binding affinity to a therapeutic target-one must choose design algorithm and specify any hyperparam- eters and predictive and/or generative models in- volved. How can these decisions be made such that the resulting designs are successful? This pa- per proposes method for design algorithm selec- fion, which aims select design algorithms that will produce a distribution of design labels satisfy- ing user-specified success criterion-ffe exam- ple, that at least ten percent of designs' labels ex- ceed a threshold. It does so by combining designs' predicted property values with held-out labeled data to reliably forecast characteristics of the label distributions produced by different design algo- rithms, building upon techniques from prediction- powered inference (Angelopoulos et al., 2023). The method is guaranteed with high probability to return design algorithms that yield successful label distributions (or the null set none exist), if the density ratios between the design and labeled data distributions known. We demonstrate the method's effectiveness in simulated protein and RNA design tasks, in settings with either known or estimated density ratios.
o0...m GG 09...9v
AO
o0..co TA
label (unknown)
AG...AT TT..A
OT...IO
TO GA.. GA..
desired average design label
prediction
Figure I. Design algorithm selection. Different design algorithm configurations (shown robots) produce different distributions of designs (c.g., biological sequences). Distributions of designs' predicted property values (x-axis) and labels (y-axis) also differ (blobs). Before the costly step of acquiring design labels, design algorithm selection aims choose design algorithm configurations that will satisfy a success criterion- example, that the average design label surpasses threshold (horizontal black line). This challenging because designs' predictions can be misleading; true labels can be low even if predictions are high (e.g.. rightmost blob).
that efficiently catalyze reactions of interest (Greenhalgh et al., 2021), photoreceptors with unprecedented light sen- sitivity for optogenetics (Bedbrook et al., 2019), and anti- bodies with enhanced binding affinity to therapeutic targets (Gruver et al., 2023), among many other applications. The methods used in such efforts, which we call design algo- rithuss, are varied. Some entail sampling from a generative model that upweights objects with promising predictions (Brookes al., 2019; Biswas et al., 2021); others start with initial candidates and iteratively introduce modifications that yield more desirable predictions (Bryant et al., 2021; Thomas et al., 2025). To propose designs, one must choose among these design algorithms. Many have consequential hyperparameter(s) that must be set, such as those that nav- igate a trade-off between departing from training data to achieve unprecedented predicted values and staying close enough that those predictions can be trusted (Biswas et al., 2021; Gruver et al., 2023; Zhu et al., 2024). One must also specify the predictive model that the design algorithm con- sults, as well as any generative model it may involve. These decisions collectively yield a design algorithm configura- tion, or configuration for short- for example, the design algorithm AdaLead (Sinai et al., 2020) using the hyperpa-
Design Algorithm Selection
Machine learning-guided design aims to propose novel ob- jects, or designs, that exhibit desired values property of interest by consulting machine learning-basee predictions of the property in place of costly and time-consuming measure- ments. The approach has been used to design novel enzymes
Preseient Design, Genentech. Correspondence to: Clara Fan- njiang <wong- fannjiang clara
clara @ @wwneggeeeeoo. com.>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
rameter settings r- 0z 0.02, k 0.05 and ridge regression predictive model. The choice of the configuration dictates the resulting designs, and, consequently, whether the design effort succeeds or fails. Specifically, we say that a config aoniiuuationn is successful if it produces a distribution of design labels that satisfies a user-specified success cri- terion-for example, that at least ten percent of the design labels exceed some threshold. In this paper, we propose a solution to the problem of design algorithm selection:
between plausible upper and lower bounds. In other cases, such as when variety of design algorithms are appropriate for the task, we may want to consider multiple options for the design algorithm, its hyperparameterrss) and the predic- tive model it consults. The menu could then comprise all combinations of such options. Indeed, the menu can include or exclude any configuration we desire; we can include op- tions for any degree of freedom whose effect on the designs we want to consider, while holding fixed those that can be reliably set by domain expertise.
How can one select design algorithm configuration that is guaranteed to be successful?
Our contributions are as follows. We introduce the problem of design algorithm selection, which formalizes a conse- quential decision faced by practitioners of machine learning- guided design and connects it to the goal of producing a distribution of design labels that satisfies a criterion. We propose method for design algorithm selection, which combines designs' predicted property values with held-out labeled data in principled way to reliably assess whether candidate design algorithm configurations will be success- ful. We provide theoretical guarantees for the configurations selected by the method, if the ratios between the design and labeled data densities are known. Finally, we demonstrate the method's effectiveness in simulated protein and RNA design tasks, including settings where these density ratios are unknown and must be estimated.
To anticipate whether a configuration will be successful, one can examine predicted property values for designs that it produces. However, these predictions can be particularly error-prone, as design algorithms often produce distribu- tions of designs shifted away from the training data in order to achieve unprecedented predicted values (Fannjiang & Listgarten, 2024). We propose a method for design algo- rithm selection that combines predictions for designs with labeled data held out from training, in a way that corrects for these errors and enables theoretical guarantees on the se- lected configurations. First, we formalize design algorithm selection as multiple hypothesis testing problem. We consider a finite set of candidate design algorithm configu- rations, called the memu, which we describe shortly. Each configuration on the menu is affiliated with a hypothesis test of whether it satisfies the success criterion. The method then computes a statistically valid p-value for this hypothe- sis test, by combining held-out labeled data with predicted property values for designs generated by the configuration. Loosely speaking, it uses the labeled data to characterize how prediction error biases p-value based on predictions alone and removes this bias, building upon techniques from coddittion-oweere inference (Angelopoulos et al., 2023). Finally, the p-values are assessed with a multiple testing cor- rection to choose set of design algorithm configurations. The method is guaranteed with high probability to select successful configurations (or return the null set if none are identified), if the ratios between the design and labeled data densities are known for all configurations on the menu. If these density ratios are unknown, we show empirically that the method still frequently selects successful configurations using estimated density ratios.
2 Problem Formalization
This section formalizes the design algorithm selection prob- lem. The next section proposes our solution.
The goal of a design task is to find novel objects in some domain, X, whose labels in some space y satisfy a desired criterion. For example, one may seek protein sequences of length L, x E X' = A" where A is the set of amino acids, whose real-valued catalytic activities for a reaction of interest, y E y = R. surpass some threshold. Note that it often neither necessary nor feasible for every proposed design to satisfy the criterion- it is sufficiently useful that some of them do so (Wheelock et al., 2022). This observa- tion guides our formalization of the success criterion, which we describe shortly. We first describe the other components of our framework.
Design algorithmss are methods that output novel objects whose labels are believed satisfy the desired criterion. We focus those that consult predictive model, though our framework is agnostic as to how they do so; they may also use other sources of information, such as unlabeled data or biomolccular structures. An example of a design algorithm for the aforementioned enzyme design task is to sample from a generative model fit to sequences that are evolutionarily related known enzyme, and return the samples with desirable predictions (Thomas al., 2025).
The contents of the menu will depend on the task at hand. If a bespoke design algorithm has been developed specifically for the task, we may be interested in setting a key hyperpa- rameter, such as real-valued hyperparameters that dictate how close to stay to the training data or other trusted points (Schubert et al., 2018: Linder et al., 2020; Biswas et al., 2021; Gruver al., 2023; Fram et al., 2024; Tagasovska al., 2024). The menu would then be finite set of candi- date values for that hyperparameter, such grid of values
2

---

Reliable Algorithm Selection for Machine Learning-Guided Design
A design algorithm configuration or configuration is spec- ification of all the hyperparameter settings and models needed to deploy design algorithm. Given design task, a practitioner constructs a menu, A: a set of candidate con- figurations to be considered. For each configuration on the menu, E A, we get predicted property values for V de- signs produced by the configuration: { (xt)) where fx is the predictive model used by configuration X.
level metric. Examples include the mean design label when 9 is the identity, as well as the fraction of designs whose labels exceed some value 7 E Y when g(y) = 1yy 2 7]- For example, a practitioner can request that at least ten percent of the designs' labels exceed that of a wild type, IWWT, using the success criterion By~Prja [ 2 /wr]] 2 0.1. We call a configuration successful if it yields a design label distribution that achieves the success criterion.
We assume access to a set of i..dd labeled data: I; Pubb- y: ~ Pr|x=z, i = 1...., nt, where Pub is the labeled distribution and Py|x= is the conditional distribution of the label random variable, Y. given the point x. We assume that this conditional distribution is fixed for every point in X, as the case when the label is dictated by the laws of nature. This data must be independent from the training data for the predictive models used on the menu, but it need not be from the training data distribution. Whenever unclear from context, we will say held-out or additional labeled data to disambiguate this data from the training data.
Our goal is to select subset of configurations from the menu such that, with guaranteed probability, every selected configuration is successful (or to return the empty set, if no successful configuration exists on the menu). That is, for any user-specified error rate a E [0, 1], we aim to construct subset À S such that
P(Ox 2 AA Å) 2 1 a
(2)
If the density ratios between the design and labeled distri- butions are known for all configurations on the menu, our proposed method, which we detail next, guarantees Eq. 2.
2.1 The Success Criterion
3 Design Algorithm Selection by Prediction-Powertt Multiple Testing
Given the above components, the goal of design algorithm selection is to select subset of contigurations, A C A, that satisfy the success criterion, which we now formalize.
Our method tackles design algorithm selection by approach- ing as multiple hypothesis testing problem (Alg. 1). The goal is to select a subset of configurations that are all success- ful, such that the error rate- the probability of incorrectly including one or more unsuccessful configurations- at most a. To accomplish this, for each configuration on the menu, consider the null hypothesis that it is unsuccessful: Hx 0₂ == Ey~Py: [Y) < T. We compute a p- p-value, Px. for testing against this null hypothesis, as we describe shortly. Finally, we use the Bonferroni correction to se- lect all configurations whose p-values are sufficiently small. This subset of configurations satisfies Eq. 2, our goal for design algorithm selection.
The designs produced by any design algorithm configura- tion, À E A, are sampled from some design distribution over X, denoted Px;A- This distribution may be specified explicitly (Brookes et al., 2019; Zhu et al., 2024), or only implicitly, such as when the algorithm iteratively introduces mutations to training sequences based on the resulting pre- dicted property values (Sinai et al., 2020). The design distri- bution in turn induces the design label distribution over V, denoted PY: which is the distribution of design labels and can be sampled from as follows: x Px;A. y ~ X Note that the labeled data and design data are related by covariate shift (Shimodaira, 2000): the distributions over X', Piah and AX: differ, but the conditional distribution of the label for any point, Py|x== for any x, is fixed.
The key subproblem is obtaining statistically valid p-values for testing against the null hypotheses H,, X E A. These hypotheses concern the design label distributions, yet we do not have labels for the designs, only predictions. To extract information from these predictions without being misled by prediction error, we turn to crndicctio--owweee inference (Angelopoulos et al., 2023), a framework that combines predictions with held-out labeled data to conduct valid statistical inference. Specifically, we use prediction- powered inference techniques adapted for covariate shift, due to the covariate-shift relationship between the design data and labeled data. We defer thorough treatment to 1, but conceptually, the labeled data, weighted by the density ratios between the design and labeled distributions, is used to characterize how prediction error distorts estima- tion of the population- metric, 0x, based on predictions alone. This error characterization is then combined with the
As previously noted, the aspiration for most design endeav- in practice is not that every single design satisfies criterion, but that enough of them do (Wheelock al., 2022). Accordingly, our framework defines success in terms of the design label distribution, PY: rather than the label of any specific design. The practitioner can specify any success criterion that requires the expected value of some function of the design labels to surpass some threshold:
0x := Ey lo
(L)
for some g: y R and E R.! We call 8g the population-
See 1 for generalization other success criteria; here focus on this special case for its broad applicahility.
3

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
predictions for the designs to compute p-values that have either asymptotic (Alg. 2) finite-sample (Alg. 3) validity. Using the latter, the output of Algorithm I satisfies Eq. 2.
Selecting zero or multiple configurations Note that an error rate of zero can be trivially achieved by returning the empty set-that is, by not selecting any configuration. This is legitimate outcome if there are no successful configura- tions on the menu, but otherwise, we will show empirically that our method also exhibits a high selection rate, or rate of returning nonempty sets. On the other hand, if our method selects multiple configurations then one can safely use any (or any mixture) of them with the same high-probability guarantee of success. In particular, one can further narrow down A using additional criteria- example, picking the selected configuration that produces the most diverse de- signs, in order to hedge against unknown future desiderata.
Algorithm 1 Design algorithm selection by multiple hy- pothesis testing
Inputs: N designs generated by each design algorithm configuration on the menu, {x;² Yi Aλ E A; predictive
models used by each configuration, {fA labeled data, {(z),)) B;)) error rate, a E [0, 1].
Output: Selected configurations, Å C A.
1: for À E A do
2: Predictions for designs, U/¹² + (x i [N].
3: Predictions for labeled data, ūj + fa(m)).j E [n].
Density ratios between design and labeled distributions Computing the p-values requires the density ratio between the design and labeled distributions, Px; (x.)) (I,), for every configuration and every labeled instance, Important settings in biological sequence design where the design den- sity can be evaluated are when designs are sampled from autoregressive generative models (Shin et al., 2021) or when the design distribution is a product of independent categori- cal distributions per sequence site (Weinstein et al., 2022; Zhu et al., 2024). Labeled sequence data is often gener- ated by adding random substitutions to wild-type sequences (Biswas et al., 2021; Bryant et al., 2021) or by recombin- ing segments of several "parent" sequences (Romero et al., 2013; Bedbrook et al., 2019), in which case Plab is also explicitly known. Valid p-values can also be computed if both Рxd2 and Plab are only known up to normalizing fac- tors, such as when sequences are generated by Potts models (Russ et al., 2020; Fram et al., 2024) ($B.2). In other set- tings, however, these density ratios need to be approximated. In experiments with unknown density ratios, we use multi- nomial logistic regression- based density ratio estimation (Srivastava et al., 2023) and show that our method still em- pirically outperforms others in selecting successful configu- rations. Although Theorems 3.1 or A. no longer apply in this setting, they inform us what guarantees we can recover, to the extent that the density ratios are approximated well.
4: PA f GETPVALUE (b IN I= (5 1)))
5: end for
6: Å + (AAA:
Algorithm 2 Prediction-powered p-value for testing Hx
0^ == Ey <
Predictions for designs, (aAj labeled data and their predictions, {(rg Vý. Új) }y)
Inputs:
Output: p-value, P.
1: A +i )
2: w/ +- DENSITYRATIO(Z,)) j=
3: Å + 1 EE== wz(g(v,) - g(,;))
4: 0 < j ++4 A
5: pmd
CE=1(((0)) 2
n Eド= (w5lg(u)) -G( ))²
6: 02
7: P - 1 - T (0--))
P N +
n1
Theorem 3.1. For any error rale E [0, 11, function 9: : + I. and threshold T E R, Algorithm I using Algorithmm 3 as the p-value subroutine returns subset of configurations from the menu, Å C A, that satisfies
P(O 2., V
When have we gone "too far?" For each configuration, the further apart the design and labeled distributions are, the higher the variance of the density ratios that are used to weight the labeled data. This reduces the effective sample size of the labeled data in characterizing prediction error, which leads to higher uncertainty about the value of the population-level metric, 02. Consequently, even if the con- figuration is successful, it may not be selected if the design and labeled distributions
where 0₂ := EY-PYY (Y and the probability is over random draws of the labeled data and the designs for all configurations on the menu.
Using asymptotically valid p-values (Alg. 2) yields an asymptotic version of the guarantee (Thm. A. In ex- periments with known density ratios between the design and labeled distributions, we always achieved error rates under even with asymptotically valid p-values. Since these are faster to compute, as they leverage closed-form representa- tions of the asymptotic null distributions, we recommend using these in practice.
too far apart. Indeed, this is desirable behavior: the method essentially identifies where over X' we lack sufficient statistical evidence of achieving the success criterion. It returns the empty set if the predic- tions and labeled data do not collectively provide adequate evidence that any configuration on the menu is successful.
4

---

Reliable Algorithm Selection for Machine Learning-Guided Design
Another factor how many configurations are selected is the multiple testing correction, it determines what counts as adequate evidence of success. A conceptual strength of framing design algorithm selection as multiple testing problem is that any multiple testing procedure that controls faie error rate (FWER) can be used in Line 6 of Al- gorithm 1. Our instantiation uses the Bonferroni correction, as it does not require any assumptions about how different configurations are related, and it yielded reasonably high selection rates with menus of few hundred configurations in experiments. However, one can also substitute FWER- controlling procedures that account for hierarchical (Bretz et al., 2009) or correlation structure (Dudoit et al., 2003) in the menu, which could yield less conservative multiplicity corrections and therefore higher selection rates.
multiple testing also select designs from a candidate pool, with guaranteed upper bounds on the false discovery rate, or expected proportion of selected designs whose label falls below a desired threshold. The setting of these methods differs from ours in that they assume access to a pool of candidates that are exchangeable-forr example, candidates drawn i.i.d. from some distribution. However, depending on the application, may not always be straightforward to narrow down large design space X' to suitable pool of candidates to begin with, without selecting a design algo- rithm configuration with which to generate designs. The goal of our work is therefore to choose from set of can- didate design algorithm configurations, each of which will produce a different distribution of designs when deployed. Our approach aims to select configurations that will achieve success criterion, which we formalize as a user-specified population- metric, 0x, surpassing some threshold.
Related Work
Bayesian optimization (BO) (Shahriari et al., 2016) is a well- studied paradigm for iteratively selecting designs, acquiring their labels, and updating a predictive model in order to optimize a property of interest. In each round, using the lan- guage of our framework, BO prescribes a design algorithm: it chooses the design (or batch of designs (Desautels et al., 2014)) that globally maximizes some acquisition function that quantifies desirability based on the model's predictions, and which typically incorporates the model's predictive un- certainty, such as the expected improvement of a design's 's label over the best one found thus far. A typical goal of BO algorithms is to converge to the global optimum as the rounds progress, under regularity conditions on the property of interest (Srinivas et al., 2010; Bull, 2011; Berkenkamp et al., 2019). In contrast, the goal of design algorithm se- lection is to achieve criteria on the distribution of design labels to be imminently proposed. Such guarantees can help justify designs to stakeholders when acquiring labels for even one round is resource- -intureveeee and the priority is to find designs that achieve specific improvements within one or a few rounds, rather than to eventually find the best pos- sible design as the rounds progress indefinitely. However, nothing precludes batch BO algorithms from being used as design algorithms in our framework: configurations of these algorithms with different hyperparameter settings or acquisition functions, for example, can be on the menu.
Design algorithm selection belongs to a body of work on managing predictive uncertainty in machine Sarning-guided design. Design algorithms can be constrained to stay close to the training data or trusted reference points, such that predictions remain trustworthy (Brookes et al., 2019; Lin- der et al., 2020; Biswas al., 2021; Gruver et al., 2023; Tagasovska et al., 2024), and out-of-distribution detection methods can flag individual designs whose predictions are unreliable (Damani et al., 2023). A variety of methods have been used to quantify predictive uncertainty for individ- ual designs in biomolecular design tasks (Greenman et al., 2025), including those based on ensembling (Scalia al., 2020; Gruver et al., 2021), Gaussian processes (Hie al., 2020; Tran et al., 2020), and evidential learning (Soleimany et al., 2021). In particular, conformal prediction techniques produce prediction sets for designs that also have frequentist- style guarantees: the sets contain the design labels with guar- anteed probability, where the probability over drawing designs from the design distribution (Fannjiang et al.. 2022: Prinster et al., 2023). However, it is unclear how to invoke such statements to make decisions about which designs to use, as there is no guarantee regarding the prediction sets for any specific designs of interest-for example, those whose prediction sets satisfy a desired condition.
Moreover, quantifying uncertainty for individual designs is perhaps unnecessary for the goal of many design cam- paigns in practice. Success often does not necessitate that every design performs desired, but only that sufficiently many do so, regardless of which specific ones (Wheelock et al., 2022). For example, Wang et al. (2022) develop a method that selects designs from a pool of individual can- didates, such that the selected subset contains a desired expected number whose label surpasses a threshold. Con- formal selection methods (Jin& & Candès, 2022; 2023; Bai & Jin, 2024) combine ideas from conformal prediction and
Similar to our work, Wheelock et al. (2022) also aim to characterize the design label distribution. To do so, however, their approach focuses on how to construct forecast, or model of the conditional distribution of the label, for indi- vidual designs. An equally weighted mixture of designs' forecasts then serves as a model of the design label dis- tribution. A rich body of work also exists on calibrating forecasts, such that various aspects of these conditional dis- tributions are statistically consistent with held-out labeled data (Gneiting et al., 2007; Kuleshov et al., 2018 8; Song et al.,
5

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
2019). Our approach differs in that it directly estimates the population-level metric, 0x, that determines whether con- figuration is successful, rather than modeling the labels of individual designs. That is, our method handles prediction error in a way that is specifically tailored for the endpoint of selecting which design algorithm configuration to deploy.
ods differ in how they construct the forecast for designed sequence; we now describe the first, called GMMForecasts.
We follow Wheelock et al. (2022), who model the condi- tional distribution of the label as a mixture of two Gaussians with sequence-specific parameters, which capture beliefs over the label if the sequence is "functional" or "nonfunc- tional." After training this model, we infer forecasts for every design produced by configuration X, with differ- ent values of hyperparameter 9 E [0, 1] that controls, roughly speaking, how much the forecasts deviate from the training data (see $C.2 for details). The equally weighted mixture of these forecasts, PCMMM serves as a model of the design label distribution for configuration A. We select À if Pr pernnlg(( 2 T.
At a technical level, our work uses prediction- poweeedd in- ference techniques (Angelopoulos al., 2023) adapted for covariate shift, in order to conduct statistically valid hy- pothesis tests of whether configurations are successful. The multiple hypothesis testing approach is similar to that of Angelopoulos et al. (2021), who use multiple testing to set hyperparameters for predictive models to achieve desired risk values with high-probability guarantees.
Calibrated forecasts method For this method, called WilibbaateForeccattt the forecast for each design is ini- tially modeled as a Gaussian with mean and variance set to the predictive mean and variance, respectively, given by the predictive model. We then use the labeled data to calibrate these forecasts post hoc using isotonic regression (Kuleshov et al., 2018) (see $C.3 for details). We select configuration A if x [Y)] 2 т, where Paal is the equally weighted mixture of the calibrated forecasts for the designs from A.
Experiments
We first demonstrate that our method selects successful configurations with high probability, as guaranteed by the- ory, when the design and labeled densities are known. Next, we show that it still selects successful configura- tions more effectively than alternative methods when these densities are estimated. Code for these experiments is at https://github..c httpaab/oo//oaaaa/ydssiigggg
O
Two metrics are of interest: error rate and selection rate. Er- ror rate is the empirical frequency at which a method selects a configuration that fails the success criterion (Eq. 1), over multiple trials of sampling designs from each configuration as well as held-out labeled data (for methods that require it). Selection rate is the empirical frequency over those same trials at which method selects anything at all. A good method achieves a low eтror rate while maintaining high selection rate, which may be challenging for ambitious success criteria.
Conformal prediction method We adapted conformal pre- diction techniques to conduct design algorithm selection (see $C.4 for details). This method has similar theoretical guarantees our method, but is prohibitively conservative: it never selected anything in any of our experiments. We therefore exclude these results for clarity of visualization.
5.1 Algorithm Selection for Designing Protein GB1
The design task for the first set of experiments was to de- sign novel protein sequences that have high binding affinity to an immunoglobulin, by specifying the amino acids at four particular sites of a protein called GBI. These ex- periments simulate library design, an important practical setting in which both the design and labeled densities have closed-form expressions. Specifically, the most time- and cost-effective protocols today for synthesizing protein se- quences in the wet lab can be described mathematically as sampling the amino acid at each site independently from a site-specific categorical distribution, whose parameters we can set; the density for any sequence is then the product of the probabilities of the amino acid at each site. We follow the design algorithm developed by Zhu, Brookes, & Busia et al. (2024): after training a predictive model of binding affinity (see $D for details), f. we set the parameters of the site site--pecific categorical distributions such that sequences with high predictions have high likelihood, as follows.
Note that the prediction-only and GeMForecasts methods do not need held-out labeled data. For fair comparison, we ran these methods with predictive models trained on the total amount of labeled data used by our method (10k instances, whereas our method trained on 5k and held out 5k).
Prediction-only method This baseline uses only the pre- dictions for configuration's designs assess whether it is successful. Specifically, it follows the same multiple testing framework as our method, but computes the p-values using the designs' predictions as if they were labels (Alg. 6).
forecasts method Both this and the next method construct a forecast, model of the condi- tional distribution of the label, for every design generated by a configuration. They then use the equally weighted mix- ture of the designs' forecasts as a model of the design label distribution for a configuration, and select the configuration this model satisfies the success criterion. These two meth-
Let Q denote the class of distributions that are products of four independent categorical distributions over twenty
6

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
amino acids. The authors use stochastic gradient descent to approximately solve QA arg minge Q (p II q). where P (z) exp(f(x)/A) and À > 0 is temperature hyperparameter that needs to be set carefully. Note that the training distribution, described shortly, was similar to a uniform distribution, which corresponds to À = co. À is low, designs sampled from QA tend to have predictions that are high but untrustworthy since qx is far from the training distribution, while the opposite is true for high À (Fig. 2). The goal is therefore to select À such that QA is successful.
labeled sequences. We then solved for ςx as described above, for all À E A. For each of 7 10 trials, sampled N 1M designs from each ax and ran both methods to select temperatures.
For our method and Calibr iebrratedie orecasts, which use held-out labeled data, we trained the predictive model on 5k labeled sequences and solved for qx À A. For each of T 500 trials, we sampled n 5k additional labeled sequences, which were used to run both methods along with N = IM designs from each ga-
We can compute the true value of 0^ for all À E A, since we have labels for all sequences in X' (Wu et al., 2016). The error rate for each method was then computed as E KElt e Å s.t. 0g 7//T. We used a 0.1 a representative value for the desired error rate.
0.7
training distribution
0.
probabill
20
site
0.
0.6
temperature (A)
from divergence
ESRAAPTG
CHONIED WM
0.5
amino acid
0.4
Results good design algorithm selection method achieves low error rate and a high selection rate for variety of success criteria (settings of and in the criterion Ey [Y) 2 r). For success criteria concerning the mean design label (i.e., g is the identity), the prediction- only and CalibratedForeee
10
0.3
0.2
0.5
0.0
0.5
1.0
1.5
methods had error rates of 100% for most values of T considered (Fig. 3a). Partic- ularly for the former, the mean design label achieved by selected temperatures could be considerably lower than т (Fig. 3b).
mean prediction for designs
Figure Library design for protein GB1. Mean prediction and KL divergence from the training distribution of the 101 design algorithm configurations on the menu. Each dot corresponds to the design distribution, Qx. for a specific value of the temperature hyperparameter, A. Two red-outlined squares correspond to design distributions for the lowest temperature (A 0.2) and the highest temperature (A = 0.7), whose parameters values are shown in the inset (top and bottom heatmaps, respectively).
Our method had error rates below the desired level of a for all values of 7 considered. The selection rate was 100% for a broad range of r, though it gradually declined for 7 > 1, reflecting increasing conservativeness for more stringent success criteria (Fig. 3a). We also ran our method with estimated density ratios for comparison, with very similar results (see $D.1 and Fig. 6).
Labeled data To simulate training and held-out labeled data, we sampled sequences from a common baseline called the NNK library, which is close to uniform categorical dis- tributions at every site, but slightly modified reduce the probability of stop codons. For the labels, used a data set that contains experimentally measured binding affinities for every sequence in X (Wu et al., 2016)-thee is, all 20 variants of protein GB1 at 4 specific sites. Labels were log-ratios relative to wild-type GBI, whose label was O.
For GMMForecasts, smaller values of the hyperparameter E Jo, 1] yield forecasts that are, roughly speaking, more similar to the training data (see SC.2 for details). Using E {0, 0.5) was prohibitively conservative and never se- lected anything on any trial. Using the maximum value, g 1, did yield high selection and low error rates for 0.5. However, for greater values of т, the method ceased selecting anything, incorrectly indicating that no configuration can achieve these success criteria (Fig, 3a). All methods had similar qualitative performance for suc- cess criteria concerning the exceedance over 1, though GMMForecasts was slightly less conservative (Fig. 7).
Menu and success criteria The menu contained 101 values of À between 0.2 and 0.7. We used the following success criteria: that the mean design label surpasses т, for E [0, 1.5] (for reference, the mean training label was -4.8), and that the exceedance over 1 (i.e., the fraction of design labels that exceed 1, using gYY)= = 1[ 2 1) surpasses T, for 7 E [o, 1] (for reference, the training labels' exceedance over was 0.006).
Recall that both the prediction-only and aMMForecast methods do not require held-out labeled data, and therefore used predictive models trained on all 10k labeled sequences. Our method outperformed them in spite of using only half the amount of training data, demonstrating the benefit in this setting of reserving labeled data for quantifying and managing the consequences of prediction error.
Selection experiments For the prediction-only method and GMMorrcassts. which do not need held-out labeled data, we trained the binding affinity predictive model on 10k
7

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
(a)
achieved mean design label
(b)
1.0
selection rate error rate
achieved desired
prediction-only
0.5
1.2
GMMForecasts, q
GMMForecasts, q 0.5
0.0
0.9
GMMForecasts, q
1.0
0.6
CalibratedForecasts method
0.5
0.3
0.0
0.0
0.0
0.3
0.6
0.9
1.2 1.5
0.0
0.3
0.6
0.9
1.2
1.5
desired mean design label (T)
desired mean design label (T)
Figure 3. Design algorithm selection for designing protein GBL. (a) Error rate (top; lower is better) and selection rate (bottom; higher better) for all methods, for range of values of r, the desired mean design label. For reference, the mean label of the training data was -4.8. GMMForecasts with hyperparameter y [0, 0.53 (dark and medium purple lines) never selected anything, resulting in and selection rates of zero for all т. (b) For each method, the median (solid line) and 20 to 80 percentiles (shaded regions) of the lowest mean design label achieved by selected configurations, over trials for which the method did return the empty set. Dots mark where cach median trajectory ends (i.c., the value of r beyond which method ceases to select any configuration, and the lowest mean design label of configurations selected that r). Results on or above dashed diagonal line indicate that selected configurations are successful.
(a)
achieved mean design label
(b)
1.0
selection rate error rate
a = 0.1
0.44
achieved desired
prediction-only
0.5
GMMForecasts,
0.40
GMMForecasts, q 0.5
0.0
GMMForecasts, 1
1.0
0.36
CalibratedForecasts method
0.5
0.32
0.0
0.28
0.28
0.32
0.36
0.40
0.44
0.28
0.32
0.36
0.40 0.44
desired mean design label (T)
desired mean design label (T)
Figure 4. Design algorithm selection for designing RNA binders. (a) Error rate (top: lower is better) and selection rate (bottom; higher better) for all methods, for range of values r, the desired mean design lahel. For reference, the mean label the training data was 0.28. (b) For each method, the median (solid line) and 20th to 80th percentiles (shaded regions) of the lowest mean design label achieved by selected configurations, over trials for which the method did not return the empty set. Dots mark where each median trajectory ends (i.e., the value of beyond which method ceases select any configuration, and the lowest mean design label of configurations selected for that r). Results on or above the dashed diagonal line indicate that selected configurations are successful.
5.2 Algorithm Selection for Designing RNA Binders
Menu and success criteria The menu contained the fol- lowing design algorithms and respective hyperparameter settings (see $E. and Figs. 8, for details): AdaLead (Sinai et al., 2020), with its threshold hyperparameter, K, set to values in [0.2, 0.01]; the algorithm used by Biswas al. (2021), which approximately runs MCMC sampling from p(x: x (x)(f) with the temperature hyperparame- ter, T, set to values in [0.005, 0.02]; Conditioning by Adap- tive Sampling (Brookes et al., 2019) and Design by Adaptive Sampling (Brookes Listgarten, 2018), with their quantile hyperparameter, Q. set to values in 10.1, 0.9]; and Proximal Exploration (Ren al., 2022) with default hyperparameters, the authors did not highlight any critical hyperparameters. Each of these design algorithms and respective hyperparam-
These next experiments show the utility of our method in a setting that involves a variety of design algorithms and predictive models on the menu, and that requires density ratio estimation. The task to design length-50 RNA se- quences that bind well RNA target, where the label is the ViennaFold binding energy (Lorenz al., 2011).
Labeled data To simulate training and held-out labeled data, we generated random mutants of a "seed" sequence, with 0.08 probability of mutation at each site. Each mutant, x, was assigned a noisy label, y BINDINGENERGY( +
6, where Ao = 0.02).
8

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
eter settings was run with three different predictive models, resulting in menu of 78 configurations: ridge regression model, with the regularization hyperparameter set by leave- one-out cross-validation, an ensemble of fully connected neural networks, and an ensemble of convolutional neural networks. The success criterion was that the mean design label surpasses T. for T E [0.28, 0.5] (for comparison, the mean training label was 0.28). We also ran our method with an expanded menu of 249 configurations involving an addi- tional hyperparameter and architectures, to assess how the multiple testing correction affects selection rates for larger menus (see $E.1 and Fig. 8).
stopped selecting any configurations for greater r (Fig. 4a, bottom). Using q E (0.5, 1} maintained 100% selection rates for broader ranges of at the cost of much higher error rates, though like our method, the errors' consequences were not severe, especially for = 0.5: the mean design labels of unsuccessful configurations were not far below T (Fig. 4b).
Overall, compared to any alternative method except GMMF orecasts with q = 0, our method had lower error rates over the range of for which the altemnttie had non-zero selection rates (Fig. 4a). Our method also main- tained non-zero selection rates for a broader range of 7 than oMMaoresssts with any (Fig. 4a, bottom). These results- which hold even with estimated density ratios, and holding out labeled data-iillsstrate the benefits of quantifying the consequences of prediction error on downstream quantities or decisions of interest, over focusing the uncertainties of individual predictions.
Density ratio estimation Since the configurations do not have closed-form design densities, we used multinomial logistic regression-based density ratio estimation (MDRE) (Srivastava et al., 2023), which trains a classifier between designs and labeled sequences to estimate the density ratios between their distributions (see $E.2 for details).
6 Discussion
Selection experiments For the prediction-only method and which do not require held-out labeled data, we first trained the three predictive models on 10k labeled sequences. For each of T = 10 trials, we sampled N = 50k designs from each configuration and ran both methods to select configurations.
We introduced an algorithm selection method for machine design, which selects design algorithm con- figurations that will be successful with high probability- that is, produce distribution of design labels that satisfies user-specified population- success criterion. It does so by using held-out labeled data to characterize and then undo how prediction error biases the assessment of whether a configuration is successful. Though the present work fo- cuses on success in a single "round" of design, it can also provide a principled decision-making framework for multi- round design endeavors in which a top priority is that the designs at each round achieve certain criteria-for example, to justify resources for acquiring their labels.
For our method and CalibratedForecase which use held-out labeled data, we trained the three predictive models on 5k labeled sequences. These sequences, as well as N = 50k designs generated from each configuration, were used to fit the MDRE model used by our method. For each of T = 200 trials, we sampled n = 5k additional labeled sequences, which were used to run both methods along with the N designs from each configuration.
For each configuration. X, we took the average of 500k design labels to serve as fx. The error rate for each method was then computed as Σ yEE e Á s.t. 0x < T//T, and we used
As with other uncertainty quantification methods that achieve frequentist-style guarantees under covariate shift (Tibshirani al., 2019; Fannjiang et al., 2022: Prinster et al., 2023; Jin & Candès, 2023), the method uses the density ra- tios between the design and labeled distributions. Advances in density ratio estimation techniques will strengthen the method's performance in settings where these density ratios are not known- in particular, techniques that are tailored for importance-weighhe ing direction is the incorporation of multiple testing pro- cedures that respect structure among configurations on the menu, which may enable less conservative selection than the Bonferroni correction. Looking forward, we encourage continued work on how to address predictive uncertainty in machine learning-guided design with respect to how it directly impacts endpoints or decisions of interest, rather than with general -urppse notions of uncertainty (Greenman et al., 2025).
0.1 as the desired error rate.
Results The prediction-ons only and Cal CalibratedForech methods had 100% error rates for much of the range of T considered. Our method had much lower error rates, though greater than for 7 > 0.32 due to density ratio estimation error (Fig. 4a, top). Furthermore, when our method selected configurations that were unsuccessful, their mean design labels were still close to r (Fig. 4b; the shaded green region does not extend far below the dashed diagonal line). Our method assesses configurations more accurately at the cost of selected subsets, Å, that are higher variance (Fig. 4b; the shaded green region is wider than other shaded regions), due to the reduced effective sample size of the weighted labeled data.
mean estimation. Another promis-
GMMPorecasts with = 0 had low error rates, even zero, for T < 0.37, but was the most conservative method: it
9

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
Acknowledgements
Salakhutdinov, R. (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 773-782. PMLR, 09-15 Jun 2019.
Our gratitude goes to Anastasios N. Angelopoulos, Stephen Bates, Richard Bonneau, Kyunghyun Cho, Andreas Loukas, Ewa Nowara, Stephen Ra, Samuel Stanton, Nataša Tagasovska, and Tijana Zmnic for helpful discussions and feedback on this work.
Brookes, D. H. and Listgarten, Design by adaptive sam- pling- In NeurIPS Workshop on Machine Learning for Molecules and Materials, 2018.
Impact Statement
Bryant, H., Bashir, A., Sinai, S., Jain, N. K., Ogden, P. J.,
This work enables practitioners machine learning-guide -uided design select successful algorithms more reliably than existing methods. There are many potential societal conse- quences of machine learning-guide guided design, none which we feel must be specifically highlighted here.
Riley, P. F., Church, G. M., Colwell, J., and Kelsic, E. D. Deep diversification of an AAV capsid protein by machine learning. Nat. Biotechnol., February 2021.
Bull, D. Convergence rates of efficient global optimiza- tion algorithms. Journal of Machine Learning Research, 12088):889--2904 2011.
References
Damani, F., Brookes, D. H., Sternlieb, T., Webster, C., Ma- lina, S., Jajoo, R., Lin, K., and Simai, S. Beyond the training set: an intuitive method for detecting distribu- tion shift in model-basee optimization. arXiv [cs.LG], November 2023.
Angelopoulos, A. N., Bates, S., Candès, J., Jordan, M. I., and Lei, L. Learn then test: Calibrating predictive algo- rithms achieve risk control. arXiv [es.LG), October 2021.
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, Prediction-powered inference. Science, 382 (6671):669-677 674, November 2023.
Desautels, T., Krause, A., and Burdick, J. W. Paralleliz- ing exploration-explootattoo tradeoffs in gaussian process bandit optimization. J. Mach. Learn, Res., 15(1)38873- 3923, 2014.
Bai, T. and Jin, Y. Optimized conformal selection: Powerful selective inference after conformity score optimization. arXiv |stat.MEJ, November 2024.
Dudoit, S., Shaffer, P., and Boldrick, J. C. Multiple hypothesis testing microarray experiments. Stat. Sci., 18(1):71-103, February 2003.
Bedbrook, C. N., Yang, K. K., Robinson, E., Mackey, E. D., Gradinaru, V., and Arnold, F. H. Machine learning- guided channelhhodopsin engineering enables minimally invasive optogenetics. Nat. Methods, 16(11):117--1844 November 2019.
Fannjiang. C. and Listgarten. J. Is novelty predictable? Cold Spring Harb. Perspect. Biol., 16(2):a04145) February 2024.
Fannjiang. C., Bates, S., Angelopoulos, A. N., Listgarten, J., and Jordan, M. I. Conformal prediction under feedback covariate shift for biomolecular design. Proc. Natl. Acad. Sei. U. A., 119(43): October 2022.
Berkenkamp, F., Schoellig, A. P., and Krause, A. No-regret bayesian optimization with unknown hyperparameters. J. Mach. Learn. Res., (1101190000335(50)1111 January 2019.
Fram, B., Su, Y., Truebridge, I., Riesselman, A. J., Ingraham, J. B., Passera, A., Napier, E., Thadani, N. N., Lim, S., Roberts, K., Kaur, G., Stiffler, M. A., Marks, D. S., Bahl, C.D. Khan, A. R., Sander, C., and Gauthier, N. P. Simul- taneous enhancement of multiple functional properties using evolution evolution-informed protein design. Nat. Commun., (5(1):514) June 2024.
Bickel, S., Bruckner, M., and Scheffer, T. Discriminative learning under covariate shift. Mach. Learn. Res., 10: 2137-2155, 2009.
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., and Church, G. M. Low-N protein engineering with data- efficient deep learning. Nat Methods, 18(4):389-366, April 2021.
Gneiting, T., Balabdaoui, F., and Raftery, A. E. Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Series B Stat. Methodol, 69(2):243 268, April 2007.
Bretz, F., Maurer, W., Brannath, W., and Posch, M. A graphical approach to sequentially rejective multiple test procedures. Stat. Med., 28(4):586-604 February 2009.
Greenhalgh, J. C., Fahlberg, S. A., Pfleger, B. F., and Romero, P. A. Machine learning- guided acyl-ACP re- ductase engineering for improved in vivo fatty alcohol production. Nat. Commun., 12(1):5825, October 2021.
Brookes, D., Park. H., and Listgarten, J. Conditioning by adaptive sampling for robust design. In Chaudhuri, K. and
10

---

Reliable Algorithm Selection for Machine Cuuiddng-Guided Design
Greenman, K. P., Amini, A. P., and Yang. K. K. Bench- marking uncertainty quantification for protein engineer- ing. PLoS Comput. Biol., 21(1))e1012639 January 2025,
Prinster, D., Saria, S., and Liu, A. JAWS-x: Addressing efficiency bottlenecks of conformal prediction under stan- dard and feedback covariate shift. In Krause, A., Brun- skill, E., Cho, K., Engelhardt, B.. Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Confer- ence on Machine Learming, volume 202 of Proceedings of Machine Learning Research, pp. 28167-281 2899. PMLR, 23-29 Jul 2023.
Gruver, N., Stanton, S., Kirichenko, P., Finzi, M., Maf- fettone, P., Myers, V., Delaney, E., Greenside, P., and Wilson, A. G. Effective Surrogate Models for Protein Design with Bayesian Optimization. In ICML Workshop on Computational Biology, 2021.
Ren, Z.,, Li, J., Ding, F., Zhou, Y., Ma, J., and Peng, J. Proximal exploration for model model-guided guided protein sequence design. In Chaudhuri, K., Jegelka, S., Song, L.., Szepes- vari, C., Niu, G., and Sabato, (eds.), Proceedings of the 39th International Conference on Machine Learning, vol- ume 162 of Proceedings of Machine Learning Research, pp. 18520- 18520-18536, PMLR, 17-23 Jul 2022.
Gruver, N., Stanton, S., Frey, N., Rudner, T. G. J., Hotzel, I. Lafrance- aafrancceVaanasee J., Rajpal, A., Cho, K., and Wilson, A. G. Protein design with guided discrete diffusion. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M.. and Levine, S. (eds.), Advances in Neural Informa- fion Processing Systems, volume 36, pp. 12489-12517. Curran Associates, Inc., 2023.
Rhodes, B., Xu, K., and Gutmann, M. U. Telescoping density-ratio estimation. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances Neural Information Processing Systems, volume 33, pp. 1905-4918 Curran Associates, Inc., 2020.
Gutmann, M. U. and Hyvärinen, A. Noise-contrastive esti- mation of unnormalized statistical models, with applica- tions to natural image statistics. J. Mach. Learn. Res., 13 (11):307331,, 2012.
Romero, P. A., Krause, A., and Arnold, F. H. Navigating the protein fitness landscape with gaussian processes. Proc. Natl. Acad. Sci. U. S. A., 1103))E193-201, January 2013.
Hie, B., Bryson, B. D., and Berger, B. Leveraging uncer- tainty in machine learning accelerates biological discov- ery and design. Cell Syst, (1(5):4614477.ck November 2020.
Russ, W. P., Figliuzzi, M., Stocker, C., Barrat-Charlaix, P., Socolich, M., Kast, P., Hilvert, D., Monasson, R., Cocco, S., Weigt, M., and Ranganathan, R. An evolution-based model for designing chorismate mutase enzymes. Science, 369(650)):440-445 July 2020.
Jin, Y. and Candès, E. Selection by prediction with con- formal p-values. J. Mach. Learn. Res., 24(244):244:1- 244:41, October 2022.
Jin, Y. and Candès, E. Model-free selective inference under covariate shift via weighted conformal p-values. arXiv [stat.ME], July 2023.
Scalia, G., Grambow, C. Pernici, B., Li, Y.-P., and Green, W. H. Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. J. Chem. Inf. Model., 60(6))2697-271) June 2020.
Kuleshov, V., Fenner, N., and Ermon, Accurate uncer- tainties for deep learning using calibrated regression. In Dy. J. and Krause, A. (eds.), Proceedings of the 35th In- fernational Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2796- 2804. PMLR, 10-15 Jul 2018.
Schubert, B., Schärfe, C., Dönnes, P., Hopf, T., Marks, D., and Kohlbacher, O. Population-specific design of de- immunized protein biotherapeutics. PLoS Comput. Biol., 14(3) 1005983, March 2018.
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. Taking the human out of the loop: A review of bayesian optimization. Proc. IEEE, 104(1): 148-175, January 2016.
Linder, J., Bogard, N., Rosenberg, B., and Seelig, G. A generative neural network for maximizing fitness and diversity of synthetic DNA and protein sequences. Cell Syst, (22::49222.1666 July 2020.
Shimodaira, H. Improving predictive inference under co- variate shift by weighting the log-likelihood function. J. Stat. Plan. Inference, 90(2):227-244, October 2000.
Lorenz, R., Bernhart, S. H., Höner Zu Siederdissen, C., Tafer, H., Flamm, C., Stadler, P. F., and Hofacker, I. L. ViennaRNA package 2.0. Algorithms Mol. Biol., 6:26, November 2011.
Shin, J.-.., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., and Marks, D. S. Protein design and variant prediction using autoregressive generative models. Nat. Commun., 12(1): 2403, April 2021.
Owen, A. B. Monte Carlo Theory, Methods, and Examples. https: /eta:///atowwr /wwtwweettteteee/ee//// su doma ins/in///////mmm.mmmmmee 2013.

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
Sinai, S., Wang, R., Whatley, A., Slocum, S., Locane, E., and Kelsic, R. D. AdaLeead: A simple and robust adap- tive greedy search algorithm for sequence design. arXiv [cs.LG], October 2020.
Wang, L.,, Joachims, T., and Rodriguez, M. G. Improv- ing screening processes via calibrated subset selection. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22702-22726. PMLR, 2022.
Soleimany. A. P., Amini, A., Goldman, S., Rus, D., Bhatia, S. N., and Coley, C. W. Evidential deep learning for guided molecular property prediction and discovery, ACS Cent Sci, 7(8):1356- 101113666-666,,-,- 1367, August 2021.
Waudby-Smin Smith,, I. and Ramdas, A. Estimating means of bounded random variables by betting. J. R. Stat. Soc. Series B Stat. Methodol., February 2023.
Song. H., Diethe, T., Kull, M., and Flach, Distribution calibration for regression. In Chaudhuri, K. and Salakhut- dinov, R. (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pp. 5897-5906. PMLR, 09 15 Jun 2019.
Weinstein, E. N., Amin, A. N., Grathwohl, W. S., Kassler, D., Disset, J., and Marks, D. Optimal design of stochastic dna synthesis protocols based on generative sequence models. In Camps- Valls, G., Ruiz, J. R., and Valera, I. (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 7450- 7482. PMLR, 28-30 Mar 2022.
Srinivas, N., Krause, A., Kakade, S., and Seeger. M. Gaus- sian process optimization the bandit setting: no regret and experimental design. In Proceedings of the 27th In- ternttional Conference International Conference Machine Learning, pp. 1015-1022. Omnipress, 2010.
Wheelock, L. B., Malina, S., Gerold, J., and Sinai, Fore- casting labels under distribution-shh for machine-guided sequence design. In Knowles, D. A., Mostafavi, S., and Lee, S.-I. (eds.), Proceedings of the 17th Machine Learn- ing in Computational Biology Meeting, volume 200 of Proceedings of Machine Learning Research, pp. 166-180. PMLR, 2022.
Srívastava, A., Han, S., Xu, K., Rhodes, B., and Gutmann, M. U. Estimating the density ratio between distributions with high discrepancy using multinomial logistic regres- sion. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Wu, N. C., Dai, L.,, Olson, C. A., Lloyd-Smith, J. O., and Sun, R. Adaptation in protein fitness landscapes is facili- tated by indirect paths. eLife, 5, July 2016.
Tagasovska, N., Gligorijević, V., Cho, K., and Loukas, A. Implicitly guided design with propen: Match your data to follow the gradient. In Globerson, A., Mackey, L., Bel- grave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Sys- tems, volume 37, pp. 35973-36001. Curran Associates, Inc., 2024.
Zhu, D., Brookes, D. H., Busia, A., Carneiro, A., Fannjiang, C., Popova, G., Shin, D., Donohue, K. C., Lin, L. F., Miller, M., Williams, E. R., Chang, E. F., Nowakowski, J.J. Listgarten, J., and Schaffer, D. V. Optimal trade-off control in machine learning-based library design, with application to adeno assocciasd virus (AAV) for gene therapy. Sci Ady, re(4)3a8)37766 January 2024.
Thomas, N., Belanger, D., Xu, C., Lee, H., Hirano, K., Iwai, K., Polic, V., Nyberg, K. D., Hoff, K. G., Frenz, L., Emrich, C. A., Kim, J. W., Chavarha, M., Ramanan, A., Agresti, J. J., and Colwell, L. J. Engineering highly active nuclease enzymes with machine learning and high- throughput screening. Cell Syst., 16(3):101236 March 2025.
Tibshirani, R. J., Foygel Barber, R., Candes, E., and Ramdas, A. Conformal prediction under covariate shift. In Wal- lach, H.. Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019.
Tran, K., Neiswanger, W., Yoon, J., Zhang, Q., Xing, E., and Ulissi, Z. W. Methods for comparing uncertainty quantifications for material property predictions. Mach. Learn. Sei. Technol, 1(2:0025006, June 2020.
12

---

Reliable Algorithm Selection for Machine Learning-Guided Design
A Additional Algorithms and Proofs
Algorithm 3 Prediction poweeedd p-value for testing HA 0x := Byaay.alll (9()) (finite sample-valid)
Inputs: Predictions for designs, (0, L. labeled data and their predictions, {(zz, My 8y)1y=1; small grid spacing, Output: p-value, P.
j j- .,....7 2: for E .0, 5, 1] do
1: wj +- DENSITYRATIOSS
3: La t PPMEANL. (út )= ((y5 ūj. w5) w;));=1
4: if La > T then
5: P+a
6: break
7: end if
8: end for
Algorithm 4 PPMEANLB: Prediction-powente confidence lower bound on 0사 == Ly [o(Y)] (finite sample-valid)
Inputs: Significance level, e [0, 1]; predictions for designs, {ūr J== labels, predictions, and density ratios for labeled data, {1V, J., w range, [L, U]. of gYY); bound, D, on the density ratios. Output: Confidence lower bound, L.
1: Plower MEANLB(O011 a, (o(ú))) [L. UI)
2: Ajower +- MEANLB(O.. a, {wy (g(8;) ((0 U), L)))
3: L Plower + Å lowwr
Algorithm 5 MEANLB: Confidence lower bound on a mean (finite sample-valid; Waudby-Smith & Ramdas (2023))
Inputs: Significance level, a E [0, 1]; data, { range of random variable, [L, U].
Output: Confidence lower bound,
1: 2, + (2; (U - L) .....,n
2: A + (0,8,... 1- 1- 8,1}
3: Mat (m) + 1, E A
4: for 1,,,.,,, do
5: Â 0.5+ zi . 0., d + 6+1
2 (2/a( no?-1
1+1
6: for m E A do
7:
+ (1 min
فني [z m)) M,
8:
if M (m) 2 1/a then
9:
A+A {m]
10: end if
11: end for
12:
end for
t min A. (U L
+L
Theorem A.1. For any error rate a E [0, 1), function g y + R. and threshold E RR Algorithm I using Algorithm 2 as the p-value subroutine returns a subset of configurations from the menu, A C A, that satisfies
lim inf P(Ox TT.
(3)
where 0g : Ey [g()), n and N are the amounts of labeled data and designs from each configuration, respectively. n N + for some r E (0, 1), and the probability is over random draws of the labeled data and the designs from all configurations on the menu.
13

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
A.1 Proofs of Theorems 3.1 and A.1
The proofs of Theorems 3.1 and A. I rely on establishing the validity of the p-values computed by Algorithms 3 and respectively. Both algorithms the framework of prediction-powent inference (Angelopoulos al., 2023), which constructs confidence intervals and p-values for estimand of interest-hert the population level metric, Py, (Y()
by combining predictions for Y with an estimand-specite characterization of prediction error called the rectifier. For our estimand, these two components emerge from the following simple decomposition of the population-level metric:
By [Y())] = Ex-Px (((((( + Ex,y~Px Prix (Y)) g(f())),,
-
(4)
where f any predictive model. The first summand the mean prediction for designs, and the second is the mean prediction bias, which serves the rectifier. More generally, prediction-powent inference encompasses any estimand that can be expressed as the minimizer of expected convex loss (see Appendix B.1), in which case the rectifier can be derived from similar decomposition of the expected loss gradient.
A1N Let { ]==1 denote N designs from configuration X, and let {(Tj, w;) denote the n labeled instances. We can use the predictions for the designs to estimate the first summand in Eq. 4 as
N
Σ ,((()))),
(5)
and use the labeled instances to estimate the second, the rectifier. Specifically, we leverage the covariate shift relationship between the design and labeled data to rewrite the rectifier as
(x:A(X) (x) Plan (an(X)yYIXN (Y X) (o(x) ((1r))(((x)))
Ex,y~Pxa
Pu
Px:a
(g(Y) - o(f(x)))
=Ex
(X)
This final expression can be estimated using the labeled data, where each instance is weighted by the density ratio between the design and label distributions:
(x;)(1z) (g(w)) g(f(z,))
(6)
(ra)(r))
Adding together the two estimates in Eqs. 5 and 6 yields prediction-powered estimate of the population- metric,
i :=
from which a prediction-powerte p-value can be computed. Specifically, Algorithm follows standard protocol for constructing asymptotically valid p-values, by first deriving the asymptotic null distribution of the prediction- -ooeeeedd-- estimate, and then evaluating its survival function produce a p-value. Alternatively, Algorithm 3 inverts finite sample-valid, prediction-powered confidence lower bounds on the population-level metric to obtain finite sample-valid p-values.
We now briefly describe why the prediction-poweee variance-and yields correspondingly more powerful p-values-thi ignoring the N predictions and using only the n weighted labeled instances to estimate the population-level metric. Specifically, the covariate shift relationship between the design and labeled data allows us to rewrite the population-level metric as
estimate of the population-lever metric, Eq. 7, generally has lower
PX (Y (X)
By>>Pralo(Y) = Ex,y (yx(g(())) = Exxy-PasPPis
Plabl X X)PXIX (Y) X) 9())
Pxx((X) (X) 9(M)
(8)
and correspondingly construct a "labeled-only" estimate,
(x))(rz)
Plab (z,)) g(U)).
(9)
14

---

Reliable Algorithm Selection for Machine Learning- Design
We now compare the asymptotic variances of the prediction-powerts estimate, Eq. 7, and the labeled-only estimate, Eq. 9. Critically, assume that N > n, it is typically far cheaper generate designs in silico than it is to acquire labeled data. The central limit theorem gives the asymptotic variance of the prediction-powered estimate as
Varlg(f(X*)) + I Var ((xx(X) (X) (g(Y)- -(()))(((x)))
For N > n, this expression dominated by the second term, the variance of the weighted prediction error scaled by 1 1/n (indeed, we can make the first term arbitrarily small by computationally generating more designs). The asymptotic variance of the labeled-only estimate is the variance of the weighted labels scaled by 1/n.
(xxa(X) Piat ((())
Var
So long as the variance of the weighted prediction error is smaller than that of the weighted explain some of the variance of labels-the asymptotic variance of the prediction-powent estimate will be smaller than that of the labeled-only estimate. That is, the prediction-powerse estimate uses information from the predictions to increase its effective sample size compared to the labeled-only estimate.
is, the predictions
Proof of Theorem A.1 (asymptotically valid design algorithm selection)
Proof. We first establish the asymptotic validity of the p-value, Px computed by Algorithm 2. Specifically, we show that under null hypothesis, Hy 0x T, PA is the survival function the asymptotic distribution of the prediction-powere estimate 0, and therefore satisfies
lim sup P(px S u) S u, Vu E 10, 10,
(10)
n.N Nox
where n/N + r for some r E 10.
We first derive the asymptotic distribution of = ú + Å, where i = (1/N) Eim g(o is the average prediction for N designs, and Á == (1/n) Σ₁= (y(g(,))- ((ū,)) where wy = (TE))//)aPPP(CCC is the importance-weighted average prediction bias for the labeled instances. The central limit theorem implies that
VN E(A]) 2 (0, ppec):
Vn(A - B(Å]) $ NO0 o2))
where ofiedd Varx~ [x([())) and oūr == Varx Pyx (g()) ,([(X))) Applying the continuous mapping theorem to the sum of fe and Á yields that
N
A A Bla+ + ÂJ) = VN(A - E(A) + V -(A)
$ N(0,, Jpro + -
where recall that n/N- r. Since
fl Á = Ex (0= + Ex.y
-(V)))(((x))) ((/00)))))))
= 0x,
equivalently have
VN(A + -00)) S NO +
1 -
We can now evaluate the survival function this distribution under the null hypothesis, HA 8g r. obtain a p-value. Specifically.
0
Px
15

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
where 6² is any consistent estimate of o + to satisfies Eq. 10. We use the estimate 6² 6 o pred N N (o ui)² and o2r == consistent estimates of dpred and oem: respectively.
preed n o where
r
[(ū;)] }} which is consistent as pred and o
Having established the validity of px, À E A, can control the family-wise error rate (FWER) with Bonferroni correction:
lim sup FWER lim sup P R,N-+00
U
X
Σ
lim sup P
(
-
NEA: PAKT
BAST
DEA: 0x
n.N
Σ
lim sumpupP P
Px
S [A]
a
n.N
AEA: Cx<r
|| A: 0x
<
<TH
a
A
$ |||
where the first line uses union bound, the second follows from the definition of in Algorithm 1, and the third due to the validity of each Px- This gives lim inf, No00 P(Ox 2 T, AA
A) = lim sup,,N-ox FWER 2 1 a.
Proof of Theorem 3.1 (finite sample-valid design algorithm selection)
Proof. We first show that the p-value computed by Algorithm 3, px, has sinile validity, meaning that under the
null hypothesis, Hx 0x < r, we have P(px I u) I u, Yu E [0. 1]. First, the confidence lower bound, La, computed by PPMEANLB (Alg. 4) is valid: it satisfies P0 2 La) 2 a. This follows from the fact that MEANLB produces valid confidence lower bounds, fluoerr and Âlwwer for Ex-Pxxx (x))) and Ex Prjx (Y - v(f())) respectively (Theorem from Waudby-Smith & Ramdas (2023)). Adding together these bounds therefore yields valid confidence lower bound, L., for 4x. Algorithm then constructs a a p-value by inverting La:
Px = inf E [0, L.a 2 r}.
(11)
This p-value is valid because under the null hypothesis, HA : 0₂ < r. for all u E [0, 1] we have
P(ex S u) < P(O, < L..) = 1 - F(Ox 2 L.,)) S - (1 - u) u,
where the first inequality follows from the definition of px in Eq. 11 and the fact that 0^ r, and the second inequality comes from the validity of L..
Having established the validity of px, À E A, the family-wise error rate (FWER), or the probability that one or more unsuccessful configurations is selected, can be controlled with the Bonferroni correction:
FWER P
U AEA <
Σ
P A()EA)
YEA: BAKT
AEA: BAKT
Σ IP Px < a
A
AEA: CAAT
H T A 0x <T]
< A
a,
A
where the first line uses union bound, the second follows from the definition of in Algorithm 1, and the third is due to
the validity of each px. We then have P(OA 2 T, AA e Á) = 1 - -FWER 2 1 1- aa
16

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
B
Extensions
B.1 More General Success Criteria
The main text considers success criteria of the following form: 0x Ey (g()) for g : 3 - R, R. More generally, we can use instiction-poweree inference techniques to compute valid p-values whenever the population-level metric, xx can be expressed as the minimizer of the expectation of some convex loss (Angclopoulos et al., 2023):
xx == arg min Ex Prix (xx,)e
where &g is convex in 0. When (x Y) = (o(Y) 0)² for some 9 y - IR, we recover the special case in the main text. We could not conceive of practical settings requiring this general characterization of 0x, but it may be useful for future work.
B.2 Design and Labeled Densities Known up to Normalizing Constants
We can compute asymptotically valid p-values if we have unnormalized forms of the design and labeled densities, such as when sequences are generated from energy-based models (Biswas et al., 2021), Potts models (Russ et al., 2020; Fram et al., 2024), or other Markov random fields. Specifically, assume we can evaluate x = a (xx:x and Plab (z) b. Pla (x) for unknown constants a, E IR, where the superscript indicates that the densities are unnormalized. To leverage these in place of the exact densities in Algorithm 2, consider the corresponding sealed density ratios on the labeled data, wy = px:x (3tg)/ (zn)) ,,....n. The self aeffmnormaleed importance- estimator of prediction bias,
Ej=1 W (9(2)) g(0,))
A
اラ w
a consistent estimator of the rectifier in Eq. 4, Pxx.Y~Px.-PPip (Y) (((())). Since Å" is ratio of means, the delta method can be used to derive its asymptotic variance (Owen, 2013), which be estimated as
1 1 E (wyg) ((9(1)) g( Â")²
71
[ト wy)
1 n
We can then compute an asymptotically valid p-value using Algorithm 2, but replacing Å with Å in Line 3 and 2 with
62
Line
A
C Other Methods
C.1 Prediction-Only Method
Algorithm 6 Prediction-only p-value
Inputs: Predictions for designs, (0) i desired threshold, r E R.
Output: p-value, P.
N Σn ( 2: pred + N D (o(?)) 2))
1: 0
3: P + 1 - Tb (0 /) V /
The prediction-only method runs multiple testing (Alg. 1) with p-values computed using only the predictions for the designs (Alg. 6), treating them as if they were labels. These p-values are asymptotically valid for testing whether Ex~Px [x))] 2 r r-that is, whether the expected function of predictions for designs, but not necessarily their labels, surpasses a threshold.
C.2 Gaussian-Mixtnn Mirtt-M--ottteeeeeee Forecasts Method
The GMMPorecasts method follows Wheelock et al. (2022), who model the forecast for a designed sequence as a mixture of two Gaussians (representing beliefs over the label if the sequence "nonfunctional" and "functional") with sequence- specific
17

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
mixture proportion, means, and variances. To construct these forecasts, their approach first assumes access a predictive mean and variance for each designed sequence, described below. It then uses the training data fit mapping from these initial predictive means and variances to Gaussian mixture model (GMM) parameters. It also seeks to address covariate shift between the design and training data by using a sequence's 's edit distance from a reference training sequence (set to wild-type GBI in the GB1 experiments, and the seed sequence in the RNA binder experiments) as an additional feature in fitting this mapping, Forecasts- is, GMM parameters- are then inferred for each designed sequence.
The forecasts also involve key hyperparameter, q E [0, 1]. For each designed sequence, after GMM parameters are inferred, the mean of the "functional" Gaussian adjusted by taking a convex combination of it and the original predictive mean, where q weights the latter. Using different values of q reflects how much one trusts the predictions; high values result in forecasts where the "functional" Gaussian mean is determined largely by the original predictive mean. We ran the method with g E {0, 0.5, 1} to span the range of possible values.
Predictive mean and variance for sequence When using predictive models that were ensembles (i.c., the fully connected ensemble in the GBI experiments, and the fully connected and convolutional ensembles in the RNA binder experiments), the predictive mean and variance for sequence was set to the empirical mean and variance, respectively, of the predictions for that sequence. When using the ridge regression model in the RNA binder experiments, the model weights were fit on 90% of the training data, and the predictive mean for a sequence was set to the model's prediction for it. The (homogeneous) predictive variance was set to the model's 's mean squared error over the remaining 10% of the training data.
C.3 Calibrated Forecasts Method
The Cal ibratssoorecssts method is based on the idea that forecasts, or models of the conditional distributions of the label, should be statistically consistent with held-out labeled data. Specifically, we want the forecasts to be calibrated as defined by Kuleshov et al. (2018), whose definition is related to the notion of probabilistic calibration (Gneiting et al., 2007):
Px, S Fx (p)) p. vp E [0,
(12)
where Fx denotes the CDF of the forecast for a sequence X E X', and the probability is over the distribution of labeled data. That is, for any E [0, 1]. the label falls under the p-quantile given by a calibrated forecast with frequency p-
For a given sequence, let an initial forecast be a Gaussian with mean and variance set to a predictive mean and variance, as described above for CDFs, such that the transformed CDF is closer achieving calibration (Eq. 12). Specifically, we use isotonic regression, following Kuleshov et al. (2018). We then construct forecast for every designed sequence by first forming the initial forecast, and then transforming the corresponding CDF with the fitted isotonic regression function.
We then use the held-out labeled data learn transformation of these initial forecast
C.4 Conformal Prediction Method
Algorithm 7 Conformal prediction-based method for design algorithm selection
Inputs: Designs generated with each configuration, (x: ) for all X A; predictive models used by each configuration, (JaJAe held-out labeled data, {(z}, W;)}"= desired threshold, E R; error rate, a E [0. 1].
Output: Subset of selected configurations, Å A.
1: for x e A do
11 2: Predictions for designs, Ui. +- i=(,.),i==,.... N
3: Predictions for labeled data, 0ū + ((f)),j==,,....
4: Density ratios for designs, U; + DENSITYRATION
N 5: Density ratios for labeled data, W. < DENSITYRATIO I....,n
6: IA + SPL.TTCONFORMALL.. ((),, ot), ((V), UJ my }) a/(|A/ -N), I,...,N N
7: TA (1/N) 1
8: end for
9: Å - (AEA:
We adapt conformal prediction techniques to conduct design algorithm selection (Algs. 7, 8). For simplicity, we describe the method with g as the identity function; for other functions, one can replace all references to labels with gY)).
18

---

Reliable Algorithm Selection for Machine Learning-Guided Design
Algorithm 8 SPLITCONFORMALLLB split conformal lower bound for design label
Inputs: a design's prediction and density ratio, (i*, u); labels, predictions, and density ratios for held-out labeled data, {15 9j. ";)] a E [0, 1].
Output: Lower bound, L E I..
0j Σ5 wy II...,m
2: u F - witu
3: f- (1 quanntiee of the distribution comprising the mixture of point masses Zi= Uj -ō + 800
4: I 4-1 5
For a given configuration, A, we can construct a valid lower bound for the empirical average of design labels, (1/N) Σ y by averaging Bonferroni-corrected conformal lower bounds for each design. Concretely, we can use split conformal method (Tibshirani et al., 2019) construct lower bounds,
4, for the labels of N designs, with confidence of 1 a/N -a/N, where the probability is random draws of the held-out labeled data. The average of these bounds, L = (1/N) Σ 1 4, then satisfies (((1N)); y 2 L) 2 a. This is because the event {yt 2 44, Vi E [N]} occurs with probability least 1 a due the Bonferroni correction, and on this event, we have (1/N) E U 2 (1/N) (i/ 4.
each (Alg. 8). These lower bounds each satisfy P(u," 2 1,)
To conduct design algorithm selection, we introduce an additional Bonferroni correction for the size of the menu, |A]: for each configuration, construct split conformal lower bound with a confidence of 1 - a/([A] NN) for each design, then take their average lower bound, Lx- The event {(1/N) Σ ya 2 LA, AA E A} occurs with probability at least 1 a, which in turn implies that for Å == {) E A LA 2 T), we have
N Σ i=1
y T VAEA
21-a.
N
Note that the lower bounds Lx, À E A, are immensely conservative: because conformal prediction techniques are meant to characterize individual labels, they cannot naturally account for how prediction errors over many designs can "cancel out" in estimation of the mean design label. We rely instead on Bonferroni corrections to guarantee the extremely stringent criterion that every individual design's ' lower bound correct, which in practice meant that LA was always negative infinity in our experiments. Conformal prediction is fundamentally not the right tool when one is interested in how prediction error affects distributions of labels, rather than the labels of individual instances.
D Protein GB1 Experiment Details
Labels were log enrichments relative to wild-type GB1, such that values greater (less) than 0 indicate greater (less) binding affinity than the wild type. Following Zhu, Brookes, & Busia el al. (2024), the predictive model, ʃ, was an ensemble of fully connected neural networks, trained using a weighted maximum likelihood method that accounted for the estimated variance of each sequence's 's log enrichment label. After training on 5k labeled sequences, the model's predictions for all X' yielded an RMSE of 1.02, Pearson correlation coefficient of 0.79, and Spearman correlation coefficient of 0.68 (Fig. 5).
D.1 Estimated Density Ratios
We ran our method with estimated density ratios, for success criteria concerning the mean design label. Specifically, we separately estimated the labeled distribution and the design distribution corresponding each configuration. For the former, we performed maximum--ikellhoon estimation with Laplace smoothing (with pseudocounts of 1) to estimate the site-specific categorical distributions using the held-out labeled sequences; for the latter, we did the same using the designed sequences. For a given sequence, we then took the ratio of its densities under these two estimated distributions as the estimated density ratio. The results from our method using these estimated density ratios were very similar to the original results using the known density ratios (Fig. 6).
19

---

Reliable Algorithm Selection for Machine Learning- Design
2.5
0.0
label
-2.5
-5.0
-7.5
-7.5
-5.0
-2.5
0.0
2.5
prediction
Figure 5. Binding affinity predictions for all protein GB1 variants. Predicted and measured binding affinity for all variants of protein GB1 l at four sites of interest. Lahels are from Wu et al. (2016) and predictions are from an ensemble of fully connected models trained on 5k labeled sequences from NNK library. Both log enrichments relative wild-type GB1, whose label is
(a)
(b)
selection rate error rate
1.0
achieved mean design label
1.5
0.1
.... achieved desired
0.5
1.2
estimated DRs
0.0
0.9
original results (known DRs)
1.0
0.6
0.5
0.3
0.0
0.0
0.6
0.9
1.2
1.5
0.0
0.3
0.6
0.9
1.2
desired mean design (t)
desired mean design label (z)
Figure 6. Design algorithm selection for designing protein GB1 (estimated density ratios). (a) Error rate (top; lower is better) and selection rate (bottom; higher is better) for our original results (with known density ratios between the designed and labeled distributions; green) and our method with estimated density ratios (purple). (b) The median (solid line) and 20th to 80th percentiles (shaded regions) of the lowest mean design label achieved by selected configurations, over trials for which the method did not return the empty set. Dots mark where each median trajectory ends (i.e., the value of beyond which the method ceased select any configuration. and the lowest design label of configurations selected for that r). Results on or above the dashed diagonal line indicate that selected configurations are successful.
20

---

Reliable Algorithm Selection for Machine Searning-Guided Design
(a)
(b)
Selection rate error rate
1.0
1.0
achieved exceedance
prediction-only
0.5
0.8
GMMForecasts, q
GMMForecasts,
0.0
0.6
GMMForecasts, CalibratedForecasts our method
1.0
0.4
0.5
0.2
achieved desired
0.0
0.0
0.0
0.2
0.4
0.6
0.8 1.0
0.0
0.2
0.4 0.6
0.8 1.0
desired exceedance (T)
desired exceedance (T)
Figure 7. Design algorithm selection for designing protein GB1 (exceedance-basedd success criteria). (a) Error rate (top: lower better) and selection rate (bottom; higher better) for all methods, for range of values of the desired exceedance over (that is, the fraction of designs whose label exceeds 1). For reference, the training labels' exceedance over was 0.006. GMMForecasts with hyperparameter q [0, 0.5, 13 (dark purple, medium purple, and pink lines) had error of for all values of (b) Median (solid line) and thth to 80th percentiles (shaded regions), of the lowest mean design label achieved by selected configurations, across trials for which each method did not return the empty set. Dots mark where each median trajectory ends (i.c., the value 7 beyond which method ceases to select any configuration, and the lowest mean design label of configurations selected for that 7). Results on or above the dashed diagonal line indicate that selected configurations are successful.
D.2 Exceedance-Basse Success Criteria
We ran the same experiments described in $5.1 and shown in Figure 3, except with success criteria involving the exceedance
over 1: ByyPy [L(Y 2 1)] 2 т. for r [0, 1]. For context, label value of was greater than 99.4% of the training labels and represents binding affinity of about 2.7 times that of wild-type GB1, The results were qualitatively similar those in the main text, except that cMMorressss with {0,0.5 0.5) was slightly less conservative and yielded high selection rates for limited range of (Fig. 7).
E RNA Binder Experiment Details
To facilitate interpretability of the label values, following Sinai et (2020) we normalized the ViennaFold binding energy (Lorenz et al., 2) by that of the complement of the RNA target sequence, which be seen as an estimate of the energy of the true optimal binding sequence. Consequently, label value of 1 means binding energy equal to that of the complement sequence.
E.1 Menu of Design Algorithm Configurations
See Fig. 8 for diagram of the menu structure.
Predictive models The three predictive models were ridge regression model, where the ridge regularization hyperparameter was set by leave-one-out cross-validation; an ensemble of three fully connected neural networks, each with two 100-unit hidden layers; and ensemble of three convolutional neural networks, each with three convolutional layers with 32 filters, followed by two 100-unit hidden layers. Each model in both ensembles was trained for five epochs using Adam with learning rate of 10-3
Design algorithm hyperparameter settings For AdaLead (Sinai et al., 2020), the values of the threshold hyperparameter
on the menu were (0.2, 0.15, 0.1, 0.05, 0.01). For Biswas, the algorithm used by Biswas et al. (2021), the values of the temperature hyperparameter on the menu were 7' E {0.02, 0.015, 0.01, 0.005]. For Conditioning by Adaptive Sampling 0.9}.
(Brookes et al., 2019), or CbAS, the values of the quantile hyperparameter on the menu were Q E (0.1,0.2, For Design by Adaptive Sampling (Brookes & Listgarten, 2018) (DbAS) with either the fully connected or convolutional models, the values of the quantile hyperparameter on the menu were Q E 0.2,, 0.9}, and with ridge regression,
21

---

Reliable Algorithm Selection for Machine Learning-Guided Design
design algorithm
hyperparameters
predictive model
AdaLead
X
threshold
X
recombination rate
ridge regression
Biswas
X
temperature
generative model
X
fully connected
CbAS
X
quantile
X
VAE latent
X
VAE # hidden
ensemble
ensemble
DbAS
X
quantile
X
VAE latent
X
VAE # hidden
PEX
Figure Menu structure for designing RNA binders. menu results main text (size 78) nested configuration space involving categorical options for the predictive model and design algorithm, and grids of real values for one hyperparameter for each design algorithm. The expanded menu for results Figure (size 249) adds an additional real-valued hyperparameter for Ada Lead, and ialeger-valued options for the generative model architecture for CbAS and DbAS (dash-outlined cells).
16
training data
adalead-cnn
14
edit distance from seed
adalead-fc
adalead-ridge
biswas-cnn
10
biswas-fc
biswas-ridge
cbas-cnn
cbas-fc
cbas-ridge
2
200
dbas-cnn
dbas-fc
0
dbas-ridge
0.15
0.20
0.25
0.30
0.35
0.40
0.45
pex-cnn
training label or design prediction (see caption)
pex-fc
pex-ridge
Figure 9. Design algorithm configurations for designing RNA binders. Gray dots give the label (ı-axis) and edit distance from a seed sequence (y-axis) for the 5k training sequences used by the predictive models. Colored markers give the average design prediction (r-axis) and average edit distance from the seed (y-axis) for each configuration on the menu. Multiple markers of the same type correspond to configurations with the design algorithm and predictive model, but different hyperparameter values. For example, the five blue triangles correspond to AdaLead using the convolutional ensemble and threshold hyperparameter values {0.2, 0.15, 0.1, 0.05, 0.01}.
22

---

Reliable Algorithm Selection for Machine Luarddng-Guided Design
(a)
(b)
1.0
achieved mean design label
0.48
selection rate error rate
0.1
achieved desired
0.5
0.44
expanded menu (size 249)
0.0
0.40
original menu (size 78)
1.0
0.36
0.5
0.32
0.0
0.28
0.28
0.32
0.36
0.40
0.44
0.48
0.28
0.32
0.36
0.40
0.44
0.48
desired mean design label (r)
desired mean design label (x)
Figure 10. Design algorithm selection for designing RNA binders (expanded menu). (a) Error rate (top; lower better) and selection rate (bottom; higher is better) for original results (with menu of size 78, green) and method with the expanded menu of size 249 diagrammed in Figure (purple). (b) For both menus, the median (solid line) and 20th to 80 percentiles (shaded regions) of the lowest mean design label achieved by selected configurations, over trials for which the method did not return the empty Dots mark where each median trajectory ends (i.e., the value of 7 beyond which the method ceased to select any configuration, and the lowest mean design label of configurations selected for that 7). Results above the dashed diagonal line indicate that selected configurations successful.
Q E 0.21, Both CbAS and DbAS involve generative model, which was a variational autoencoder (VAE) with 10 latent dimensions, and fully connected models with 20-unit hidden layers for both the encoder and decoder. Both algorithms were run for twenty iterations; each iteration retrained the VAE for five epochs using Adam with learning rate 10-3 30-3 PEX (Ren et al., 2022) was run with default values of all byperparameters. The resulting menu of design algorithm configurations varied greatly in their mean design prediction, as well as their distance from the training sequences (Fig. 9).
To assess how the multiplicity correction might impact selection rates for larger menus, we also our method with an expanded menu of 279 configurations. In addition to the hyperparameter settings listed above, this menu included different values of the AdaLead recombination rate hyperparameter, r {0.1, 0.2, 0.5}. and numbers of hidden units (f5, 10}) and latent dimensions ((20, 100}) in the VAE used by both CbAS and DbAS (Fig. 8). Interestingly, our method exhibited similar error rates but higher selection rates with this larger menu, including non-zero selection rates for a broader range of r than with the original menu of size 79 (Fig. 10a), due to the expanded menu containing more configurations that produce higher mean design labels.
F.2 Density Ratio Estimation
To estimate the density ratio function, Px.xx))/paa()), for every configuration on the menu, E A, we used multinomial logistic regression-bass based density ratio estimation (MDRE) (Srivastava et al., 2023). MDRE builds upon a formal connection between density ratio estimation (DRF) and classification (Bickel al., 2009; Gutmann & Hyvärinen, 2012): for the (correctly specified) binary classifier that minimizes the population cross-entropy risk in distinguishing between samples from two distributions, its logit for any input x is equivalent to the density ratio for z, Px.x (2)) Pran (x)) (Gutmann & Hyvärinen, 2012). In practice, however, we can only hope to find a classifier that minimizes the empirical risk between finite samples from the two distributions. This distinction between the population and empirical risks hinders DRE far more than does classification for classification's sake: obtaining the density ratio requires getting the exact value of the population-optimal classifier's logit, whereas optimal classification performance can be achieved by any classifier that learns the same decision boundary as the optimal classifier, even if its logits differ from the optimal classifier. Accordingly, we found that for many configurations approximations of the density ratio. This was particularly true when the design and labeled distributions were far apart, because the classification problem is too easy given finite samples: many different classifiers can minimize the empirical risk, none of which may happen to coincide with the population-optina. one whose logits are equivalent to density ratios. Telescoping density ratio estimation (Rhodes et al., 2020) tackles this problem by constructing intermediate distributions that interpolate between the numerator and denominator distributions, creating sequence of "harder" binary classification
the menu, classifiers with very low training and validation losses often yielded poor
23

---

Reliable Algorithm Selection for Machine Lcarning-Guided Design
problems for which there are fewer empirically optimal classifiers, and whose resulting estimated density ratios can be combined through telescoping sum to approximate the original density ratio of interest. MDRE is similarly motivated but constructs single multi-class classification problem between the intermediate distributions, justified by theoretical connections between the population-optina classifier and the density ratio analogous to those described above (Srivastava et al., 2023). Concretely, let (x) denote the unnormalized log-probability according to a trained classifier that x belongs to distribution с E (I,..., C),, where denotes the total number of distributions. MDRE uses exp (x) 1 (x)) to approximate the density ratio between distributions (numerator) and j (denominator).
Note that the design algorithm selection problem naturally lends itself to the construction of the intermediate dis- tributions used by MDRE, because many design algorithms have hyperparameters that navigate how far the de- sign distribution strays from the labeled distribution. It is often of interest to include different settings of such hyperparameters on the menu, in which case all of these configurations' density ratios with the labeled distribu- tion can be approximated using one MDRE classifier. For the RNA binder experiments, denote configurations by [design algorithm] (prrddictvve model]- (byperppptpprrraeeeeee value] for example, Biswas-CAN-C.CO refers to running Bi swas with the convolutional ensemble predictive model and temperature hyperparameter = 0.02. We fit separate MDRE model for each combination of design algorithm and predictive model is, separate multi-class classifier for Adalead-ridge-+, for AdaLead-FC-+, for AdaLead-CNN-+, for Bisxas-ridge++, for Biswas-FC-+, for Bi Swas-CNN,,,, for CbAS-ridge-s, for CbAS-FC-s, for CbAS-CNN-v, for DiAS-ridges for DbAS-FC-x, for DbAS-CNN- for PEX-ridge for PEX-FC, and for PEX-CNN, where the last three reduced to binary classification problems. Each of these classifiers was fit on the 5k training sequences and N = 50k designed sequences from each included configuration. For example, the 10-category classifier for estimating density ratios for CbAS-ridge-* was fit on the 5k training sequences and 50k sequences each from CbAS-ridge-O 1, ObAs-ridge-00 0. .... ChAS-ridge-C, 8, and CbAS-ridge-00 Each classifier was a model with one 256-unit hidden layer and quadratic final layer, trained for 100 epochs using Adam with learning rate of 10-3
24


Reviews:
[{"review_id": "yhyNaH3h4j", "reviewer": "Reviewer_Bfc7", "summary": "When performing model-guided design, the goal is to propose new objects x that have some desired property, where the relationship between x and the property is approximated by a predictive model f(x). The problem is that f(x) may be unreliably when proposing x far from the training data. This makes it difficult to choose among design algorithms. Each defines a distribution over potential designs. How do we estimate the performance of a design algorithm without actually measuring the property for these designs (which could involve an expensive wet-lab experiment)?\n\nThe authors propose an appealing frequentist formalism for the problem and leverage the recent 'prediction powered inference' framework to propose an algorithm with desirable properties.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the positive evaluation of our work, and are glad the reviewer found our method and experiments well-motivated and appealing. Our responses to specific comments and questions follow.\n\n- \"The hyper-parameter range is also discretized at an arbitrary resolution of 100 values, but the menu size drives the performance of the algorithm due to the multiple testing correction.\" We appreciate your concern regarding this point. See [Fig. R1](https://shorturl.at/qE6TG) for new results where the menu for the protein GB1 experiments is discretized to contain 501 values (np.arange(0.2, 0.7, 0.001) instead of the original np.arange(0.2, 0.7, 0.005)). Our method still keeps error rates below the user-specified level, consistent with the guarantees. Interestingly, it does so with *higher* selection rates for greater values of tau (x-axis) than with the original smaller menu. That is, the multiple testing correction did not make the method more conservative with a 5X-larger menu; in fact, the method actually selected successful configurations more frequently, because the expanded menu contains more configurations with greater mean design labels.\n\n- \"I was surprised to find that the 'menu' corresponded to a range of values for a single real-valued hyper-parameter. There is a lot of structure across this menu that the method doesn't exploit.\" Thank you for this insightful observation. Please see [Fig. R2](https://shorturl.at/mOSna) for new RNA experiments with an expanded menu of size 249, containing the more complex hierarchical configuration space illustrated in [Fig. R3](https://shorturl.at/X0EOu). Our method empirically performs well in these experiments, but we agree that it would be possible to better exploit the structure in the menu induced by the relationships between different configurations. This is a conceptual strength of formalizing design algorithm selection as a multiple testing problem: any multiple testing procedure that controls family-wise error rate (FWER) can actually be used in place of the Bonferroni correction in Line 6 of Alg. 1, meaning one can use FWER-controlling procedures that respect, e.g., the hierarchical organization of the configuration space [1] or the correlations between configurations [2]. We instantiated Alg. 1 with the Bonferroni correction for full generality, as it doesn't require any assumptions about how configurations are related, but replacing this with procedures that exploit these relationships is a great direction for future work. Note that using such structure-respecting multiple testing procedures would not improve the existing guarantees on the error rate, but might improve (i.e., increase) the selection rate by yielding a less conservative multiplicity correction.\n\n- \"For problems where the density ratio was tractable, it would have been helpful to see what the performance difference is between the case when using the true density ratio vs. an approximated one.\" Thank you for this suggestion. See [Fig. R1](https://shorturl.at/qE6TG) for new results where we estimated the density ratios (DRs) for the protein GB1 experiments. Specifically, we separately estimated the labeled distribution and the design distribution corresponding to each configuration. For the former, we performed maximum-likelihood estimation with Laplace smoothing (with pseudocounts of 1) to estimate the site-specific categorical distributions using the held-out labeled sequences; for the latter, we did the same using the design sequences. For a given sequence, we then took the ratio of its densities under these two estimated distributions as the estimated DR. The results from our method using these estimated DRs are very similar to the original results using the known DRs.\n\n- Figs. 3A, 4A: Thank you for noting that the presentation was confusing. To clarify, would the reviewer prefer that we zoom-in the y-axes of plots closer to 0 for error rate and 1 for selection rate? Note that the methods have error rates and selection rates that span the entire [0, 1] range, because we wanted to thoroughly compare their performance across a wide range of success criteria (x-axis values) that practitioners might be interested in. However, we have zoomed-in the error rate plot of [Fig. R1](https://shorturl.at/qE6TG) closer to 0 as suggested, since the plotted rates are all less than 0.1.\n\n[1] Bretz et al. A graphical approach to sequentially rejective multiple test procedures. Stat. Med. 2009.\n\n[2] Dudoit et al. Multiple Hypothesis Testing in Microarray Experiments. Stat. Sci. 2003.", "claims_and_evidence": "The evaluation is a bit confusing at first, as it requires really understanding the frequentist formulation of the design algorithm selection problem. However, given this problem formulation, the way that algorithms are evaluated makes a lot of sense.", "methods_and_evaluation_criteria": "The proposed method is well motivated and appealing. I appreciate that subtle variations are also provided for regimes where certain required quantities (the density ratio term) are note available. \n\nI found the set of baselines to be very well presented and well motivated.", "theoretical_claims": "I do not have the required technical background to assess the correctness of the proofs in the appendix.", "experimental_designs_or_analyses": "Yes, I think it was posed correctly.", "relation_to_broader_scientific_literature": "It does a good job of framing things in terms of related work", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "I found figures 3A and 4A very confusing. I feel like there should be a better way to present this data that is mostly saturated at y = 0 or y = 1.", "other_comments_or_suggestions": "When I got to the experiments section, I was surprised to find that the 'menu' corresponded to a range of values for a single real-valued hyper-parameter. There is a lot of structure across this menu that the method doesn't exploit. The hyper-parameter range is also discretized at an arbitrary resolution of 100 values, but the menu size drives the performance of the algorithm due to the multiple testing correction. I worry that this choice of 100 could have had a big qualitative impact on the results.\n\nFor problems where the density ratio was tractable, it would have been helpful to see what the performance difference is between the case when using the true density ratio vs. an approximated one.", "questions_for_authors": "Can you please show that the experiments' outcomes are not sensitive to the discretization resolution discussed above?\n\n\n## After Authors' response ##\nThanks for addressing my comments. I think the proposed changes will strengthen the manuscript.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nuIWuXGR1y", "reviewer": "Reviewer_EEWq", "summary": "Hyperparameter tuning and algorithm selection can be really tricky in real scenes. This paper proposes a method based on prediction-powered inference techniques for design algorithm selection, aiming to choose some settings (configurations) that satisfy users' demands. Two practical experiments demonstrate this method can effectively select ideal settings.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We are glad the reviewer found the idea of selecting reliable algorithms to have strong potential to benefit the ML community. We believe there has been a misconception regarding our primary contributions.\n\n**[First to formalize + address DAS]** To our knowledge, although ML has revolutionized the design of proteins, small molecules, and other modalities, this work is the first to formalize the problem of design algorithm selection (DAS) and propose a solution with any guarantees. None of the reviewers have suggested any baselines or alternative approaches, which testifies to the novelty and difficulty of DAS and the creativity of our solution.\n\n**[Not just HP tuning]** This work is not simply a variant of hyperparameter tuning as it is performed in supervised learning. Tuning cannot be performed for design tasks the same way, because we never have the labels of designs needed to evaluate each design algorithm configuration. This is why our major technical innovation is to rigorously evaluate which configurations will be successful, in the complete absence of labels for their designs.\n\n**[Formalized population-level success criteria w/ high-probability guarantees]** Another innovation is formalizing + achieving a common goal in practice: to produce a *population* of designs that satisfies a desired criterion (e.g., that at least 10% of designs' labels surpass some threshold, or that the average design label does so). Our method provides high-probability guarantees for a broad class of such population-level success criteria. Prior work has focused instead on uncertainty over individual designs, rather than population-level criteria often sought in practice.\n\nWe are glad the reviewer found the paper well-written, and appreciate the concerns regarding the method's practicality, addressed below.\n\n- \"Menus ... only contain roughly 100 choices\": See [Fig. R1](https://shorturl.at/qE6TG) and [Fig. R2](https://shorturl.at/mOSna) for new protein and RNA experiments with menu sizes 501 and 249, respectively, where our method still performs well. However, even for the original smaller menus, our method outperformed the baselines. Are there alternative approaches the reviewer might propose? We are happy to try them.\n\n- \"[Density ratios can be] impossible to estimate\": Fig. 4 showed empirically that our method is still effective with a relatively simple classifier-based density ratio estimation (DRE) technique. Also see [Fig. R1](https://shorturl.at/qE6TG) for new results showing our method performs well with DRE in the protein GB1 experiments. Note that DRE is routinely used in ML applications involving importance weighting [1,2] and yields consistent estimators of the DRs if the model class is correctly specified (flexible enough) [3]. We agree that the guarantees don't hold with DRE, but this doesn't nullify the value of having guarantees when DRs are known, in contrast to baselines without guarantees at all.\n\n- Requires \"held-out labeled data (HLD)\": Figs. 3&4 compared to baselines that do not require HLD (prediction-only and GMMForecasts), which instead used *all* the labeled data to train the predictive model. Our method outperformed these baselines. We also note that the entire literature on post-hoc calibration for uncertainty quantification (including e.g. conformal prediction) requires HLD. Indeed, there is a fundamental limit to how calibrated models can be w/o using HLD [4]. In practice, collecting some extra labeled data can be a reasonable investment when the method's guarantees can help justify costly resources for synthesizing designs to project stakeholders. Lastly, our method is amenable to cross-fitting variants that do not completely hold out data [5] (happy to elaborate).\n\n- Can the method handle \"categorical parameters and ... multiple parameters?\": See [Fig. R2](https://shorturl.at/mOSna) for new RNA experiments with an expanded menu of size 249, containing the more complex hierarchical configuration space illustrated in [Fig. R3](https://shorturl.at/X0EOu). Our method performs well here (similar error rates with higher selection rates), as the expanded menu contains configurations with higher mean design labels than the original menu.\n\n- Fig 3B,4B: The \"=\" is an equals sign, indicating that the diagonal line is the y = x line. Results on/above the diagonal mean that selected configurations are successful. \n\n- Line 275: Thank you for noting the broken link! Now fixed.\n\n[1] Sugiyama et al. Density ratio estimation in machine learning. 2012. \n\n[2] Grover et al. Bias correction of learned generative models using likelihood-free importance weighting. NeurIPS 2019.\n\n[3] Gutmann & Hyvarinen. Noise-contrastive estimation of unnormalized statistical models. JMLR 2012.\n\n[4] Bengs et al. Pitfalls of epistemic uncertainty quantification through loss minimisation. NeurIPS 2022. \n\n[5] Zrnic & Candes. Cross-prediction-powered inference. PNAS 2024.", "claims_and_evidence": "Yes. The authors use p-values to support their design further.", "methods_and_evaluation_criteria": "The method makes sense.", "theoretical_claims": "I check the theoretical claims in the main context, showing their framework can guarantee a high success rate for selecting desirable settings.", "experimental_designs_or_analyses": "Yes, both experiments seem sound to me.", "relation_to_broader_scientific_literature": "The method can speed up the process of finding optimal hyperparameter and algorithm combinations and significantly reduce the cost of wet lab experiments.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of selecting reliable algorithms has a strong potential to benefit all machine learning communities.\n2. The paper is well-written and easy to comprehend.\n\nWeaknesses:\n1. The major concern lies in that the authors have claimed a big idea, but the fact is that the menus used in both experiments only contain roughly 100 choices, which is far less in practical scenarios. A single method can require multiple hyperparameters to tune when applied to a new task, let alone we are in an era where new methods for a single task are merging every day.\n2. The method is guaranteed by the assumption that the density ratios between the design and labeled data distributions are known, which can vary in real experiments or even be impossible to estimate.\n3. The method uses held-out labeled data to characterize how prediction error affects the evaluation, which acquires lots extra information.", "other_comments_or_suggestions": "The reference in line 275 cannot jump.", "questions_for_authors": "1. What does '=' stands for in Figure 3(b) and Figure 4(b)?\n2. Can your method be extended to more complicated tasks? For now, each of the RNA binder design algorithms contains one or zero hyperparameters, and all parameters are continuous. Is this method still satisfying when facing categorical parameters and methods with multiple parameters? Proving this would greatly help this paper to be more convincing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sem0qz6ecV", "reviewer": "Reviewer_pgJa", "summary": "This paper proposed a method for design algorithm selection which combines designs' predicted property values with held-out labeled data to reliably assess whether a candidate design algorithm configuration produces successful designs. Specifically, the method selects configurations for design algorithms such that the resulting outcome (label) satisfies a pre-defined success criterion (e.g., having a sufficient fraction of designs exceeding a certain threshold) with error rate at most $\\alpha$.\n\nTo tackle this problem, the paper first formalizes the problem as hypothesis testing over a menu of candidate design algorithms, and then using prediction-powered inference techniques adapted to covariate shift (from held-out labeled data) to obtain statistically valid p-values for the hypothesis tests. \n\nUnder the assumption that the density ratio between the design and labeled distributions are known, the paper gives theoretical guarantee that the selected configurations achieves the success criterion for any error rate. Without known density ratio, the paper provides empirical evidence of the procedure by simulation on two biologically motivated examples that their method successfully controls the error rate while identifying configurations that produce high-quality designs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback! We would like to refer the reviewer to the first 3 paragraphs of our response to reviewer EEWq, as we believe there has been a misconception regarding our primary contributions. \n\nIn particular, to our knowledge, our work is the first to formalize and propose a rigorous solution to the problem of design algorithm selection (DAS): a principled method for making the decisions that every practitioner of ML-guided design must make, and which might otherwise be biased by personal preference or convenience. None of the reviewers have suggested any alternatives, which testifies to the novelty and difficulty of DAS and the creativity of our solution.\n\nOur responses to specific points follow.\n\n- \"Would [the Bonferroni correction] lead to overly conservative selection?\" Empirically, this was not a problem. Our method had low error rates while maintaining high selection rates, compared to baselines (Figs. 3a,4a). In new results with a 5X-larger menu for the protein GB1 experiments ([Fig. R1](https://shorturl.at/qE6TG), see first comment to reviewer Bfc7 for details), our method actually has higher selection rates than with the original menu, as the expanded menu contains more configurations with higher mean design labels. If one is concerned about conservatism, however, a conceptual strength of our formalization of DAS as a multiple testing (MT) problem is that *any* MT procedure that controls FWER can be used instead of Bonferroni. E.g., one can use procedures that respect the hierarchical [1] or correlation structure between configurations [2], which may yield less conservative multiplicity corrections. Since our goal was to establish a first, general solution to DAS, we instantiated Alg. 1 with Bonferroni, as it doesn't require assumptions about how configurations are related.\n\n- \"Can you get theoretical guarantees on the selection rate?\" No, because if the practitioner specifies an impossible success criterion, then the selection rate must be zero. That is, no non-zero lower bound exists in general.\n\n- Potential issues with estimated density ratios (DRs): While the guarantees in the paper do not hold with DRE, this does not nullify the value of having guarantees when DRs are known, in contrast to baselines w/o guarantees at all. We appreciate your concern, however, and present new results in [Fig. R1](https://shorturl.at/qE6TG) showing that our method performs similarly in the protein GB1 experiments using estimated vs. known DRs. Both there and in our RNA experiments in Fig. 4, our method with DRE outperforms the baselines. Also note that DRE yields consistent estimators of the true DRs if the model class is correctly specified (flexible enough) [3], and is routinely used in ML applications involving importance weighting [4].\n\nTo further address the concerns regarding Bonferroni and DRE, we also present new RNA experiments in [Fig. R2](https://shorturl.at/mOSna), which use DRE with an expanded menu of size 249 containing the hierarchical configuration space shown in [Fig. R3](https://shorturl.at/X0EOu). Our method performs well in these experiments.\n\n- Thank you for suggesting the relevance to Bayesian optimization and active learning! We now mention them in \"Related Work,\" condensed here:\nBayesian optimization (BO) is a paradigm for iteratively selecting designs, acquiring their labels, and updating a predictive model in order to optimize a property of interest. In each round, BO chooses the design that globally maximizes some acquisition function quantifying desirability based on the model's predictions. A typical goal of BO is to converge to the global optimum as the rounds progress, under regularity conditions [5]. In contrast, the goal of DAS is to achieve criteria on the distribution of design labels to be imminently proposed. Such guarantees can help justify designs to stakeholders when acquiring labels for even one round is resource-intensive, and the priority is to achieve specific criteria within one or a few rounds. However, nothing precludes BO from being used as design algorithms within our framework: configurations of BO with, e.g., different HPs or acquisition functions can be on the menu. \n\nFinally, we emphasize that our method empirically outperforms the baselines. Despite the impact of the DAS problem, no reviewer has suggested any alternative approaches. If the reviewer would like to suggest one, we are happy to include it!\n\n[1] Bretz et al. A graphical approach to sequentially rejective multiple test procedures. Stat. Med. 2009.\n\n[2] Dudoit et al. Multiple hypothesis testing in microarray experiments. Stat. Sci. 2003.\n\n[3] Gutmann & Hyvarinen. Noise-contrastive estimation of unnormalized statistical models. JMLR 2012.\n\n[4] Sugiyama et al. Density ratio estimation in machine learning. 2012. \n\n[5] Srinivas et al. Gaussian process optimization in the bandit setting. ICML 2010.", "claims_and_evidence": "The central claim of the paper is that the proposed algorithm (hypothesis testing procedure combined with prediction powered inference for valid p-values) is able to select design algorithm configurations that achieves the success criterion with guarantees on any error rate (i.e. probability of selecting algorithms that don't match the success criterion). \n\nUnder the assumption that the density ratio between the design and labeled distributions are known, Theorem 3.1 and Theorem A.1 give theoretical guarantees that using finite-sample or asymptotically valid p-values, the proposed procedure controls the error rate at any level $\\alpha$.\n\nWhen this assumption doesn't hold, the paper combines the approach with a multinomial logistic regression-based method for estimating the density ratio. Empirically, the paper compares their approach against other baselines (prediction-only model, GMMForecasts) and \nshows that on the protein and RNA design tasks, their method outperforms others in selecting successful configurations.", "methods_and_evaluation_criteria": "The paper proposes to use a multiple hypothesis testing approach to test if each configuration fits the user-defined success criterion. To obtain statistically valid p-values for the hypothesis test, the paper uses prediction-powered inference adapted for covariate shift to extract information from these predictions without being misled by prediction error. It then usees a Bonferroni correction to bound the overall error rate. \nThe overall algorithm makes sense and gives guarantees under the assumption of known density ratio between design and labeled distributions.", "theoretical_claims": "I checked the proofs for the theoretical claims (Theorem 3.1 and Theorem A.1) and think it's correct.", "experimental_designs_or_analyses": "Without known density ratio, the paper provides empirical evidence of the procedure by simulation on two biologically motivated examples that their method successfully controls the error rate while identifying configurations that produce high-quality designs.\nIn the experiments, the paper measures both the error rate of the selected configurations (probability of selecting at least one unsuccessful configuration) as well as the selection rate to ensure the algorithm is not overly conservative that outputs no configurations.", "relation_to_broader_scientific_literature": "This work proposes the use of multiple hypothesis testing and prediction-powered inference applied to covariate shift to address machine learning-guided design problems where out-of-distribution predictions are common. Their proposed method gives a systematic way to choose design algorithms that fits the success criterion with guaranteed error rates.", "essential_references_not_discussed": "The paper discusses their method within the context of machine-guided design and prediction-powered inference. \nThe overall problem formulation might also be related to the area of Bayesian optimization and active learning.", "other_strengths_and_weaknesses": "n.a.", "other_comments_or_suggestions": "n.a.", "questions_for_authors": "- Algorithm 1 (line 6) selects p-values by a Bonferroni-style threshold. Would this possibly lead to overly conservative selections? \n- Related, can you get theoretical guarantees on the selection rate?\n- Do you have any theoretical guarantees if the density ratio is not known? Or, if the support of the densities are different? Are there possible problems that might arise in your analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies a pretty neat problem: how do we actually select optimization/design algorithms without actually making many more measurements of the objective function? The authors present a frequentist approach to this problem with guarantees under the assumption that the density ratio between the design and labeled distributions are known. The authors responded comprehensively to the questions raised by the reviewers during the author feedback period. I generally agree with the authors that Reviewer EEWq appears to have overlooked the main contributions of this work, and that the setting considered here is pretty substantively different from hyperparameter tuning (although I suppose could reduce to hyperparameter tuning in the setting where the set of design algorithms differed only by hyperparameter values?)
Decision: Accept (poster)