Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient
Authors: Natasa Tagasovska
Vladimir Gligorijevic
Kyunghyun Cho
Andreas Loukas
Venue: NeurIPS 2024 poster (2024)
Keywords: matching
property enhancement
gradient approximation
design optimization
shape optimization
antibodies
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: A method inspired by ``matching'' that allows for implicit guidance, sidestepping the need for a discriminator. Training with a matched dataset approximates the gradient of a property and applies to design optimization in real-world datasets.

Abstract: Across scientific domains, generating new models or optimizing existing ones while meeting specific criteria is crucial. Traditional machine learning frameworks for guided design use a generative model and a surrogate model (discriminator), requiring large datasets. However, real-world scientific applications often have limited data and complex landscapes, making data-hungry models inefficient or impractical. We propose a new framework, PropEn, inspired by ``matching'', which enables implicit guidance without training a discriminator. By matching each sample with a similar one that has a better property value, we create a larger training dataset that inherently indicates the direction of improvement. Matching, combined with an encoder-decoder architecture, forms a domain-agnostic generative framework for property enhancement. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution, allowing efficient design optimization. Extensive evaluations in toy problems and scientific applications, such as therapeutic protein design and airfoil optimization, demonstrate PropEn's advantages over common baselines. Notably, the protein design results are validated with wet lab experiments, confirming the competitiveness and effectiveness of our approach. Our code is available at https://github.com/prescient-design/propen.

Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient
Nataša Tagasovska*
Vladimir Gligorijevié
Kyunghyun Cho
Andreas Loukas
Abstract
Across scientific domains, generating new models or optimizing existing ones while meeting specific criteria is crucial. Traditional machine learning frameworks for guided design use generative model and a surrogate model (discriminator), requiring large datasets. However, real-world scientific applications often have limited data and complex landscapes, making data- hata-hungry models inefficient impractical. We propose a new framework, PropEn, inspired by "matching", which enables implicit guidance without training discriminator. By matching each sam- ple with a similar one that has better property value, we create a larger training dataset that inherently indicates the direction of improvement. Matching, combined with an encoder-decoder architecture, forms a domain-agnostic generative frame- work for property enhancement. We show that training with matched dataset approximates the gradient of the property of interest while remaining within the data distribution, allowing efficient design optimization. Extensive evaluations toy problems and scientific applications, such as therapeutic protein design and airfoil optimization, demonstrate PropEn's s advantages over common baselines. No- tably, the protein design results are validated with wet lab experiments, confirming the competitiveness and effectiveness of our approach. Our code is available https://gghbbb.coom//sssseeedsgs
Introduction
Navigating the complex world of design is a challenge in many fields, from engineering 35 to
material science [33 40 and life sciences (6]. life sciences, the goal may be to refine molecular structures for drug discovery [6] focusing on properties like binding affinity or stability. In engi- neering, optimizing the shapes of aircraft wings or windmill blades to achieve desired aerodynamic traits like lift and drag forces is crucial [35]. The common thread in these fields is the design cycle: experts start with an initial design and aim to improve specific property. Guided by intuition and expertise, they make adjustments and evaluate the changes. If the property improves, they continue this iterative optimization process. This cycle is repeated multiple times, making it time-consuming and resource- intensive.. MI. holds promise to reduce these costs, speed up design cycles, and create setter-performing designs B
22
Yet, progress in ML methods for design is hindered by practical challenges. The first challenge is limited data availability. Since gathering label measurements is resource intensive [46] 28 50 19),
designers are more often than not constrained to very small-scale datasets. In addition, there are often non-smooth functional dependencies between the features and outcome, complicating approximation, even for deep neural networks 30 (45.. Traditional methods use two part frameworks, requiring discriminators guide the property enhancement for examples produced by generative model. Such discriminators should be able to reliably predict the property of interest given some training data or latent representations. Because of the dependency on training a discriminator for guidance,
Prescien//MLDDD, Genentech Research and Early Development
Department of Computer Science, Center for Data Science, New York University
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Implicit guidance
Step 1: Match dataset
Step Train PropEn
Legend
A
A
samples
DA
neural net
embedded samples
W
Propts User spoce
Explicit guidance
W
Wivt space
Step 1: Train a generative model
Step 2: Train property predictor
Step 3: Guide optimization latent space
Figure 1: Conceptual summary of implicit and explicit guidance. The task increasing the size the objects. Top in implicit guidance, first we match the training dataset, by pairing each sample with the closest one w.r.t its shape, which has a better property (size). Then, we train encoder-decoder framework which due to the construction of the dataset learns lower dimensional manifold where the embeddings are ordered by the property value. Bottom in explicit guidance, we train two separate models: generator and discriminator that guides the optimization in latent space.
we denote these methods as explicit guidance. While Genetic Algorithms were once prevalent [14].
contemporary models like auto-encoders [47). GANs [49], and diffusion models now dominate both research and practice [31] the role of generative models. Despite their flexibility, such models face challenges typical to deep learning: they are "data- "data-hungry" and unreliable when encountering
out-of-distrin dittr--diitribution examples 29
37..
Motivated by these challenges, propose a new approach inspired by the concept of "matching" Matching techniques in econometrics are used to address the challenge of selection bias and confound- ing when estimating causal effects in observational studies 2
38
39 43 These techniques aim
create comparable groups of units by matching treated and control observations based on observable characteristics. The basic idea behind matching is to identify untreated units that are similar to treated units in terms of observed covariates, effectively creating a counterfactual comparison group for each treated unit. Matching techniques in ML as in econometrics have only been used provide more robust, reliable causal-effect estimation 24 48 19.
This work argues that, in lack of large datasets, matching allows for implicit guidance, completely sidestepping the need for training a discriminator (differentiable surrogate model). We match each sample with similar obtain a much larger training dataset, inherently embedding the direction of property enhancement. We name method Propłi and illustrate in Figure 1 By leveraging this expanded dataset within a standard encoder-decoder framework, we circumvent the need for separate discriminator model. We show that PropEn is domain agnostic, can be applied any data modality continuous or discrete. Additionally, PropEn alleviates some common problems with explicit guidance such as falling off the data manifold or requiring complex engineering.
that has superior value for the property of interest. By doing so,
Overall, our contributions are as follows:
We propose "matching" (inspired by causal effect estimation) to expand small training datasets (subsection 2)
We provide a theoretical analysis on how training on a matched dataset implicitly learns an approximation of the gradient for property of interest (subsection 2.2):

---

We provide guarantees that the proposed designs are as likely as the training distribution, avoiding common pitfalls where unreliable discriminators lead to unrealistic, pathological designs (subsection 2.3);
We demonstrate the effectiveness and advantages of implicit guidance through extensive experiments in both toy and real-world scientific problems, using both numerical and wet lab validation (section 3).
2 Property Enhancer (PropEn)
Given set of initial examples, objective is to propose new design which is similar the initial set, but, exceeds it in some property value interest.
Problem setup. Concretely, we start with dataset D = { V consisting of observed examples 24 E I!' drawn from a distribution p together with their corresponding properties y: = g(x4)) R. Our objective is to determine ways improve the property of test example. Concretely, at test time, we are given point Ty and aim to identify some new point x"e ~ = close to zy such that (x=""))= > g (.) In effect, our problem combines constrained optimization (maximize g(2"(") while staying close to 2o) with sampling from a distribution (point z"r should likely according to p).
Hereafter, we will refer to the initial example we wish optimize as seed design and the model's proposal as candidate design. Our method, PropEn, entails three steps: (i) matching dataset, (ii) approximating the gradient by training model with a matched reconstruction objective and (iii) sampling with implicit guidance.
2.1 Match the dataset
We view the group of samples with superior property values the treated group and their lower-value counter part as the control group. This motivates us to construct "matched dataset" for every (z, v) within D:
2, E D (z)
|| - =112 S.. -(x)) (0, A0 }
M
(I)
where A, and Ag are predefined positive thresholds.
The matched dataset gives us new and extended collection M whose size N 0(n²) >> n can significantly exceed that of the training set, depending on the choice of matching thresholds.
2.2 Approximate the gradient
After matching the dataset, we train a deep encoder-decoder network So M by minimizing the matched reconstruction objective:
{(fo;M)
1
Σ e(fo(z),x',,
(matched reconstruction objective)
[M (I,T")MM
where L is appropriate loss for the data in question, such mean-squared error (MSE) or cross-entropy loss.
Before illustrating the properties of our method empirically, we perform theoretical analysis. We show that minimizing the matched reconstruction objective yields a model that approximates the direction of the gradient of g(.), even if property predictor has been explicitly trained:
Theorem 1. Let f* be the optimal solution of the matched reconstruction objective with a sufficiently small Ag. For any point
the matched dataset for which p uniform within ball of radius Ag. we have (x) + cV (z(z) for some positive constant c.
The detailed proof is provided in /uusecctionn A.I Remark 1. The proof is founded on the assumption that distribution is uniformly distributed within a ball of radius A, around point z. This assumption is made to maintain the generality of the theorem without specific information about the sampling distribution, assuming uniformity avoids
3

---

ID Beed
(4) uplimication inseettry (ID smsdls)
(b) oplimizaiion Inserdory (0OD swnds)
000 Seed
(3) NLL during optimiatiion
Figure 2: Illustration of PropEn the pinwheel toy example with only 72 training examples. The training data are circles in grey, colored by the value of the property. With pink we mark the initial hold out test points and in orange 'x' the PropEn trajectories. The color of the candidates intensifies with each iteration step. On the right-hand-side, we depict the sum of negative log likelihoods of the seeds and optimized designs across optimization steps.
introducing any biases that could arise from other distributional assumptions, such as symmetry, finite variance etc.
Remark 2. can be shown that with with matched reconstruction objective we learn a direction that is a-colinear with the gradient of g, avoiding the isotropy assumption. This leads to additional analysis on understanding the implications of the choices while matching, all included in Appendix A
2.3 Optimize designs with implicit guidance
Training on matched dataset allows for auto-regressive sampling. Starting with a design seed zo, for I .... we can generate 2; = ʃo (2==-) )) until convergence, fo(x)) Te. g(rre g( At test time, we feed a seed design zo to PropEn, and read out an optimized design æ1 from its output. We then proceed to iteratively re-feed the current design to PropEn until fe(x;) = Iı. which is analogous to arriving at stationary point with Vg(xz) = 0 and we have exhausted the direction of property enhancement given the training data. Exploiting the implicit guidance from matching results in trajectory of multiple optimized candidate designs.
We next show that optimized samples almost likely training according to the data distribution p. This serves as a guarantee that the generated designs lie within distribution, as desired: Theorem 2. Consider model f* trained minimize the matched reconstruction objective The probability of f*(x) at least
(()* 2 Exr-aa [I1,[(())\la
where fiy is the empirical measure on the dataset, (x) is the Hessian of p at x and (2 = Exrrp, |||=" Egra, [a]]][]] is the variance induced by the matching process.
The detailed proof is provided in subsection A.3
We synthetic example to illustrate optimizing designs with PropEn. We choose a 2d pinwheel dataset. As property to optimize, we choose the likellioddood of the data as estimated by a KDE with Gaussian kernel with o 0.01. Figure depicts in gray the training points, with the color intensity representing the value of the property hence higher/darker value is better in this example. After training PropEn, we take held out points (pink squares) and use them as seed designs. With orange x- x-markers, we illustrate PropEn candidates, with the color intensity increasing at each step t. We notice that PropEn moves towards the regions of the training data with highest property value, consistently improving at each step (right-most panel). Additionally, we also use out-of-distribution seeds, and we demonstrate in the middle panel that PropEn chooses to optimize them by proposing designs from the closest regions in the training data.

---

Table 1: Overview of the datasets experiments.
Dataset
Domain
Size
Type
Metric
Property
Preview
Toy
RD 1 100
50, 100
cont.
L2
log- likelihood
Airfoil
R4000
200,500
cont.
L2
dift-to-drag
ratio
binding
Antibodies
20
200 400
discrete
Levenshtein
affinity
Experimental Results
We empirically evaluate PropEn synthetic and real-world experiments answer the following main questions: (i) Can PropEn be applied across various domains and datasets? (ii) Does PropEn provide reliable guidance, especially in situations with limited data and when dealing with out-of- distribution examples? (iii) How effective is PropEn in recommending optimal designs? Can suggest candidates with property values exceeding those in the training set? (iv) How does PropEn's performance vary with different data characteristics (e.g., dimensionality. sample size, heterogeneity) and hyperparameters (such as Ag. 4y, and regularization terms)? Our code available at hhtpps //github. com/ wwtam//pesientt..m desi www.ii/proppeenne.eeeeeeeep http:/ppppettt////ggtuubbo..o pprpscientddssinnppeeeee
Datasets. We consider three different data types: synthetic 2d toy datasets and their higher dimension transformations, NACA airfoil samples, and therapeutic antibody proteins. An overview of the data is given in Table I] We present results in two settings, in silico where rely experimental validation using computer simulations and solvers, and in vitro experiments where candidate designs tested in wet lab. Each of the experiments is evaluated under the baselines and metrics suitable for the domain.
PropEn variants. We investigate the utilization of matching and reconstruction within the PropEn framework. Two key considerations emerge: first, whether to reconstruct solely the input features (x2x) or both the input features and the property (xy2xy); second, the proximity to the initial sample, regulated by incorporating straightforward reconstruction regularizer into the training loss (()) 2). This regularized variant will be referred to mixup/mix.
3.1 In silico experiments
3.1.1 Toy data
We choose two well-known multi-modal densities: pinwheel and 8-Gaussians. These are 2d datasets, but, in order to make the task more challenging, we expand the dimensionality to d E {10, 50, 100} by randomly isometrically embedding the data within a higher dimensional space. Our findings are summarized in Figure 3) and we include the tabular results in infitem B.7 We empirically validate the four variants of PropEn and we compare against explicit guidance method: for consistency, we chose an auto-encoder of the same architecture PropEn augmented with discriminator for guidance in the latent space. We denote this baseline Explicit. We compare the methods by ratio improvement the proportion of holdout samples for which PropEn baselines demonstrate enhanced property values. To assess the quality of the generated samples we report uniqueness and novelty in tables. We likelihood model derived from a Kernel Density Estimation (KDE) fit the training data. The negative log-likelhhood scores under this model serve as an indicator of in-distribution performance. Higher values for all metrics indicate better performance.
Results. Several insights can be gleaned from these experiments. When analyzing the results based on the number of samples, a clear trend emerges: the number of training samples increases, PropEn consistently outperforms explicit guidance across all metrics, except for average improvement, where all methods exhibit similar behavior. The choice of the preferred metric may vary depending on the specific application; however, is noteworthy that while explicit AE guidance improves approximately 50% of the designs for all datasets, PropEn demonstrates the potential to enhance up to 85% of the designs. Importantly, this improvement trend remains consistent regardless of the
5

---

$4.00
PROCE
3-100
2-100
C1.00
Figure 3: PropEn in toy examples in d E {50, 100), left side: 8-Gaussians, right side: pinwheel. Distribution of evaluation metrics from 10 repetitions of each experiment.
dimensionality of the data. Furthermore, an intriguing observation pertains to the performance of different variations of PropEn. It is noted that as the sample size increases, PropEn xy2xy does not exhibit an advantage over PropEn x2x. Moreover, the impact iterative sampling with PropEn is notable. With each step, the property improves until it reaches a plateau after multiple iterations, albeit with a simultaneous drop in the uniqueness of the solution to around 80%. Nevertheless, iterative optimization can be continued until convergence, with all designs saved along the trajectory for later filtering according to user needs.
3.1.2 Engineering: airfoil optimization
In engincering context. shape optimization entails altering the shape of an object to enhance its efficiency. NACA airfoils (National Advisory Committee for Aeronautics) [4], rooted in aerodynamics and parameterized by numerical values, serve well-documented benchmark due their versatility. These airfoils span diverse aerodynamic characteristics, from high lift to low drag, making them ideal for exploring different optimization objectives. Given their integral role in aircraft wings, optimizing airfoil shapes can significantly impact aerodynamic performance by improving lift, drag, and other properties essential for aerospace engineering.
Data & experimental design. We generate NACA 4-digit series airfoil coordinates by choosing these parameters: . (maximum camber percentage), P (location of maximum camber percentage), and T (maximum thickness percentage). Each airfoil is represented by 200 coordinates, resulting in a 400 vector representation when flattened. Our objective is to optimize the lift-to-drag ratio (Ci/C, ratio) for each shape. We calculate lift and drag using NeuralFoil [1], a precise deep-learning emulator of XFoil
Note that the lift-to-drag ratio is pivotal in aircraft design, reflecting the wing's lift generation efficiency relative to drag production. A high value signifies superior lift production with minimal drag, translating to enhanced fuel efficiency, extended flight ranges, and overall improved performance. This ratio is paramount in aerodynamic design and optimization, facilitating aircraft to travel farther and more efficiently through the air. Traditionally, engineers have relied on genetic algorithms guided by Gaussian Process models (kriging) (22). however, in recent years the community has moved towards ML.-based methods which consist of generative model that can be GAN-based (1 49 or a variation of VAE [26 52 31 47 Similarly, for the surrogate, guidance model, GPs and numerical solvers have been replaced by deep models 55 41). For our experiments we follow this standard setup: we choose a VAE-like baseline
is the most similar architectural choice PropEn, and for guidance we use a MLP. All networks (encoder, decoder and surrogate) are fully connected 3 layer MLPs with 50 units, ReLU activations per layer and a latent space of dimension 50. We randomly select 0.1 % as holdout dataset for seeds, and use the rest for training.
Results. Our numerical findings summarized in ??. Similar to the toy dataset, the designs produced by PropEn variants demonstrate enhanced properties compared to those guided explicitly. Delving into further analysis with ablation studies, depicted in Figure 4(b) and (c), we observe that increasing the matching thresholds in PropEn correlates with higher rates of improvement. Remark- ably, all designs within PropEn trajectory are deemed plausible, depicted in the accompanying figure. Moreover, a consistent enhancement in the fot-to-drag ratio C,/C is noted along the opti- mization trajectory until convergence. This consistent trend underscores the effectiveness of PropEn in progressively refining airfoil designs to bolster their aerodynamic performance.
Interestingly, find that in larger training datasets the threshold for property improvement (Ay) may not be necessary for optimization, as the PropEn x2x demonstrates satisfactory performance.

---

100
48
C/CC ratio 1710
96
84
50
92
40
Phowshold
140
11
th
80
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Unrestuudd value
Uhresitold value
Figure 4: Ablation studies for PropEn on Airfoils. (a) PropEn improves CI/Cd ratio along its trajectory and produces realistic/valid airfoil shapes. and e) the impact of choice of threshold A5 and A, in the matching phase. When varying Az. Ag is set to I, and vice versa.
We also notice that the mix variant of PropEn may require longer training. This observation is consistent with the notion that mix, by introducing a regularization term in the training loss, may improve at a slower rate compared to other PropEn variants. This slower improvement can be attributed to the regularization term's tendency to pull generated designs closer the initial seed, thereby limiting the extent of exploration in the design space.
3.2 In vitro experiment: therapeutic protein optimization
The design of antibodies good functional and developability properties is essential for developing effective treatments for diseases ranging from cancer to autoimmune disorders. We here focus on the task of optimizing the binding affinity of starting antibody (the seed) while staying close to it in terms of edit distance. The task is referred to as affinity maturation the drug design literature and constitutes an essential and challenging step in any antibody design campaign.
Antibody binding affinity refers to the strength the interaction between an antibody molecule and its target antigen. High binding affinity is crucial in antibody-based therapeutics as it determines the antibody's 's ability to recognize and bind to its target with high specificity and efficiency. We follow the standard practice of quantifying binding affinity by the negative log ratio of the association and dissociation constants (pKD). which represents the concentration of antigen required to dissociate half of the bound antibody molecules. Higher pK1) indicates tighter and more stable interaction, leading to improved therapeutic outcomes such as enhanced neutralization of pathogens or targeted delivery of drugs specific cells.
Data experimental design. The data collection process involved conducting low-throughput Surface Plasmon Resonance (SPR) experiments aimed measuring with high accuracy the binding affinity of antibodies targeting three different target antigens: the human epidermal growth factor receptor (HER2) and two additional targets that denote TI and T2. For each of those targets, one or more seed designs were selected by domain experts. In the case of HER2, we used the cancer drug Herceptin seed. We ensured the correctness of the SPR measurements by validating the fit of the SPR kinetic curves according to standard practices.
As the targets differ in the properties of their binding sites, we trained PropEn model per each target (but for all seeds for that target jointly). For this application, we opted for the PropEn x2x mix variant. The reconstruction of the original sequence (mix) complies with antibody engineering wisdom that a candidate design should not deviate from a seed by more than small number of mutations. Similar to 34 used one-hot encoded representation of antibodies aligned according to the AHo numbering scheme [20] determined by ANARCI [1]. The encoder-decoder architecture based on a ResNet [18] with 3 blocks each. We compare PropEn with four strong baselines: two state-of-the- art methods for guided and unguided antibody design namely walk-jump sampler [33] and lambo [17] well two variants of diffusion model trained on AHo antibody sequences differing their use of guidance. The first one (labeled as diffusion) is based on a variational diffusion model (5 trained on latent space obtained by projecting AHo 1 1-hot representation using an encoder-decodee type of architecture similar to PropEn's architecture; encoder-decoder model is trained simultaneously with the diffusion model. The second one (labeled as diffusion(guidee) a variant of the first one with
7

---

3.0
(per 54043
Propen
2.5
walle jump
brdnng aftraty mnroonmerr
2.0
diftusion (guitedi
diftusion
undiing arterty morovemen
1.5
lambo (guided)
1.0
0.5
0.0
-0.5
-.00
0.0
0.2
0.4
0.8
1.0
Proph
walejomp lambe (pudee) dffiiiaon ciffasion (puides)
binging rate (por sced)
((a))
((b)
Figure 5: Therapeutic protein optimization results: (a) The left figure contrasts the binding rate with the 90-th percentile of the binding affinity improvement for each method and seed. Points on the top-right are on the Pareto front. (b) The right figure focuses on binders and reports the histograms of binding affinity improvement across all designs and seeds.
Table 2: Binding rate (and number of designs submitted). Higher better.
Horcaptin
TISI
T153
T281
T252
T253
T2S4
Proptia
90.9% (11)
100.0% (4)
100.56 (6)
100.%% (74)
20.0% (5)
100,0% (23)
100.0% (16)
100.0% (4)
94.6% (93)
walk--jump [+]
25.0% (4)
80.0% (15)
100.0% (18)
26.7% (30)
41.756 (12)
100.0% (15)
63.6% (11)
62.9% (105)
fambbo (guided) 1+
50.0% (10)
0.0% (4)
100.0% (5)
0.0% (9)
100.0% (1)
57.1% (14)
44.2% (43)
difTusion
X5.7%
88.2% (17)
66.7%
diffusionn (guided)
85.25 (27)
96.95 (32)
93.356 (15)
100.0% (10)
92.9% (84)
added guidance based on the iterative latent variable refinement idea described in the paper by Choi et al. [8), which ensures generating samples that are close to the initial seed.
We evaluate the set of designs in terms of their binding rate (fraction of designs tested that were binders), the percentage of designs than improve the seed, and their binding affinity improvement (pKD) design pKD of seed).
Table 3: Fraction of designs improving the seed and total designs tested. Higher is better.
Herceptin
TISI
T1S2
TIS3
T2S1
T282
T2S3
T2S4
overall
PropEn
0.0% (11)
100.0% (4)
33.3%
41.7% (24)
0.0% (5)
69.6% (23)
0.0% (16)
0.0% (4)
(93)
walk-jump [
25.0% (4)
6.7% (15)
5.6% (18)
3.3% (30)
8.3% (12)
0.0%% (15)
0.0% (11)
4.8% (105)
lambo (guided)
10.0% (10)
0.0%
0.0%
0.0% (9)
0.0%
35.7% (14)
14.0%
diffusion
62.5% (8)
14.3% (14)
0.0%% (17)
0.0% (6)
15.6% (45)
diffuslon (guided)
51.9% (27)
15.6% (32)
0.0% (15)
0.0% (10)
22.6% (84)
Results. As seen in Tables 2 and 3. PropEn excelled in generating functional antibodies with
consistently high binding rates (94.6%) and 34.5% of the tested designs showed improve binding than the seed, outpacing other models in overall performance. To account for the trade-off between binding rate and affinity improvement (larger affinity improvement requires making risky mutations that might end-up killing binding), we visualize the Pareto front in Figure 5(a) In the plot, we mark the performance of each method for specific seed design by placing marker based on the achieved binding rate (X-axis) and maximum affinity improvement (x-axis). Compared to baseline methods, PropEn struck a beneficial trade-off, on average achieving a larger affinity improvement than methods with high binding rate.
Figure 5(b) takes closer look at the affinity improvement on the subset of designs that bound. As observed, all models produced some binders that were better than the seed, speaking for the strength of all considered models. Interestingly, none the top three models in terms of binding affinity improvement relied on explicit guidance, which aligns with our argument about the brittleness of explicit guidance in low-data regimes. Out of the those three models, PropEn generated two designs that improved the seed by at least one pKD unit (10 times better binder) followed by the walk-jumm and the unguided diffusion model, that generated one such design each.
8

---

4
Related work
As design optimization has been ubiquitous across science domains, naturally our approach relates to a variety of methods and applications. In the molecular design domain, data are bound to discrete representation which can be challenging for ML methods. A natural way to cireumvent that is by optimization in latent continuous space. Bombarelli et al. [15] presented such an approach in their seminal work on optimizing molecules for chemical properties, and it has since spawn across different domains ( Recently, one of the common way for obtaining embeddings for explicit guidance relies on language models 27 [2]. One of the challenges of using a latent space the issue for blindly guiding the optimization into ambiguous regions of the manifold where training data was available
[1 [9]] Follow up works attempt to address this problem 7 16 37 by incorporating uncertainty estimates into black-box optimization.
Another line of work perhaps more close to our approach is the notion of neural translation, where the goal is to go from one language to another by training on aligned datasets. [23 I0] have build on this idea to improve properties for small molecules translating one graph or sequence to another one with better properties. These works propose tailored approaches for domain specific applications. With PropEn we take a step further and propose a domain-agme aamaan-agnostic framework that empirically validated across different domains (uniquely including wetlab experiments). We also derive novel theoretical guarantees that illustrate the relation of the generated samples with the property gradient, as well as provide guarantees that our designs fall within distribution.
Previous works have also considered learning an optimizer for some function based on observed samples 42 1l (3]] This is usually achieved by either (i) rendering the optimizer differentiable and training some of its byperparameters; or (ii) by unfolding the iterative optimizer and treating each iteration as a trainable layer. Our approach is different, thanks to the matched reconstruction objective that lets us implicitly approximate the gradient of function of interest.
Conclusion
method for implicit guidance in design optimization that approximates the gradient for a property of interest. We achieve this by leveraging matched datasets, which increase the size of the training data and inherently include the direction of property enhancement. Our findings highlight the versatility and effectiveness of PropEn in optimizing designs in engineering and drug discovery domains. We include wet lab in-vitro results for comparison with state-of-the-arr baselines in therapeutic protein design. By utilizing thresholds for shape dissimilarity and property improvement, PropEn efficiently navigates the design space, generating diverse and configurations. We believe our method offers a simple yet effective recipe for design optimization that can be applied across various scientific domains.
Strengths. We introduced PropEn,
Limitations. The matching step adds some computation overhead with complexity depending on the choice of distance metric. Since PropEn is targeting low fata-regime applications, scalability is out of scope for the current work. However, we are considering on-the-fly distance evaluation or parallelisation across multiple nodes. The choice of distance metric for matching certain extent, can be considered limitation because it requires understanding of the context for the application. However, this choice is also what allows for incorporating domain knowledge and constraints, which can be meaningful and necessary in the domain of interest (edit distance for antibodies, deviations only in the camber of the airfoil etc).
Future work. Immediate extensions of PropEn are applications to other molecular modalities, such as small molecules, material discovery, and optimization of adeno-associated virus vectors for gene-therapy. Additionally, we are keen to explore how different similarity metrics incorporate various inductive biases that can be leveraged for property optimization. Ongoing worthwhile efforts include developing a multi-property PropEn framework to address the optimization of propertics simultaneously, offering more comprehensive approach to the design process.
References
[1] Akshay Agrawal, Shane Barratt, Stephen Boyd, and Bartolomeo Stellato. Learning convex optimization control policies. In Learning for Dynamics and Control, pages 361-373. PMLR,

---

2020.
[2] Robert P Althauser and Donald Rubin. The computerized construction of a matched sample. American Journal of Sociology, 76(2):325-346, 1970.
[3] Prasanma Balachandran, Benjamin Kowalski, Alp Schirlioglu, and Turab Lookman. Ex- perimental search for high-temperature ferroelectric perovskites guided by two-step machine learning. Nature communications, 9(1):1668, 2018.
[4] Roger E. Bilstein. Orders of Magnitude: A History of the NACA and NASA, 1915-1999 volume 4406. National Acronautics and Space Administration, Office of Management .... 1989.
[5] Mohamed Amine Bouhlel. Sicheng He, and Joaquim RRA Martins. Scalable gradient audient--nnhnned artificial neural networks for airfoil shape design in the subsonic and transonic regimes. Struc- tural and Muliddiscppinnay Optimization,
2020.
[6] John Bradshaw, Brooks Paige, Matt Kusner, Marwin Segler, and José Miguel Hernández- Lobato. A model search for synthesizable molecules. Advances in Neural Information Processing Systems, 32, 2019.
[7] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pages 773-782. PMLR, 2019.
[8] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 2021.
[9] Oscar Clivio, Fabian Falck, Brieuc Lehmann, George Deligiannidis, and Chris Holmes. Neural score matching for high-dimensional causal inference. International Conference on Artificial Intelligence and Statistics, pages 7076-7110. PMLR, 2022.
[10] Farhan Damani, Vishnu Sresht, and Stephen Ra. Black box recursive translations for molecular optimization. arXiv preprint arXiv: 10/2. 2019.
[11] Mark Drela. Xfoil: An analysis and design system for low reynolds number airfoils. In Low Reynolds Number Aerodynamics: Proceedings of the Conference Notre Dame, Indiana, USA, 5-7 June 1989, pages 1-12. Springer, 1989.
[12] James Dunbar and Charlotte M. Deane. ANARCI: antigen receptor numbering and receptor classification. Bioinformatiis, 32(2):298-333 300, 09 2015.
[13] Nathan C. Frey Dan Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien Lafrance- Vanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, Andreas Loukas, Vladimir Gligorijevic, and Saeed Saremi. Protein discovery with discrete jalk-jump sampling. In The Twelfth International Conference Learning Representations, 2024.
[14] Mitsuo Gen and Runwei Cheng. Genetic algorithms and engineering optimization, volume 7. John Wiley & Sons, 1999.
[15] Rafael Benjamín Sänchez-Lengeling, Dennis Sheberla, Jorge aguilera-Iparraguirg Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Gzik, Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276,, 2018.
Jennifer N Wei, David Duvenaud, José Miguel Hermández-Lobaue
[16] Ryan-Rhys Griffiths and José Miguel Hernández-Lobate Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science, 11(2):577-586 2020.
[17] Nate Gruver, Samuel Don Stanton, Nathan C. Frey, Tim G. J. Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
10

---

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Jordan Hoffmann, Yohai Bar-Sinai, Lisa M Lee. Jovana Andrejevic, Shruti Mishra, Shmuel M Rubinstein, and Chris H Rycroft. Machine learning in data-limited regime: Augmenting exper- iments with synthetic data uncovers order in crumpled sheets. Science advances, 5(4):eaau6772, 2019.
[20] Annemarie Honegger and Andreas PluEckthun. Yet another numbering scheme for immunoglob- ulin variable domains: an automatic modeling and analysis tool. Journal of molecular biology, 309(3):657-67) 670, 2001.
[21] David Janz, Jos van der Westhuizen, and José Miguel Hernández-Lobato. Actively learning what makes a discrete sequence valid. arXiv preprint 17Xiv:170.,044200 2017.
[22] Jingchao Jiang. A survey of machine learning in additive manufacturing technologies. Interna- lional Journal of Computer Integrated Manufacturing, (6(9):1258-1288) 1280, 2023.
[23] Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to-graph translation for molecular optimization. arXiv preprint arXiv: 2018.
[24] Nathan Kallus. Deepmatch: Balancing deep covariate representations for causal inference using adversarial training. In International Conference on Machine Learning, pages 5067-5077. PMLR, 2020.
[25] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
[26] Jiaqing Kou, Laura Botero-Bolívar, Román Ballano, Oscar Marino, Leandro De Santana, Eusebio Valero, and Esteban Ferrer. Aeroacoustic airfoil shape optimization enhanced by autoencoders. Expert Systems with Applications, 217:119513, 2023.
[27] Ben Krause, Subu Subramanian, Tom Yuan, Marisa Yang, Aaron Sato, and Nikhil Naik. Improving antibody affinity using laboratory data with language model guided design. bioRxiv, pages 2023-09 09, 2023.
[28] Rafat Kurezab, Sabina Smusz, and Andrzej Bojarski. The influence of negative training set size on machine learning- baased virtual screening. Journal of cheminformatics, 6:1-9, 2014.
[29] Matt Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. In International conference on machine learning, pages 1945-1954. PMLR, 2017.
[30] Karolina Kwapien, Eva Nittinger, Jiazhen He, Christian Margreitter, Alexey Voronov, and Christian Tyrchan. Implications of additivity and nonadditivity for machine learning and deep learning models in drug design. ACS omega, 7(30))26573-26681 2022.
[31] Jichao Li, Xiaosong Du, and Joaquim RRA Martins. Machine learning in aerodynamic shape optimization. Progress in Aerospace Sciences, 134:100849, 2022.
[32] Lin Li, Esther Gupta, John Spaeth, Leslie Shing, Rafael Jaimes, Emily Engelhart, Randolph Lopez, Rajmonda S Caceres, Tristan Bepler, and Matthew E Walsh. Machine learning opti- mization of candidate antibody yields highly diverse uub-nanomolar affinity antibody libraries. Nature Communications, 14(1):3454, 2023.
[33] Turab Lookman, Stephan Eidenbenz, Frank Alexander, and Cris Barnes. Materials discovery and design: By means of data science and optimal learning, volume 280. Springer, 2018.
[34] Karolis Martinkus, Jan Ludwiczak, WEI-CHING LIANG, Julien Lafrance- Vanasse, Isidro Hotzel, Arvind Rajpal, Yan Wu, Kyunghyun Cho, Richard Bonneau, Vladimir Gligorijevic, and Andreas Loukas. Abdiffuser: full-atom generation of in-vitro functioning antibodies. In seirth- Conference on Neural Information Processing Systems, 2023.
[35] Joaquim RRA Martins. Aerodynamic design optimization: Challenges and perspectives. Com- puters & Fluids, 139:105391, 2022.

---

[36] Vishal Monga, Yuelong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2):18-44 2021.
[37] Pascal Notin, José Miguel Hemánneez-Lobtto, and Yarin Gal. Improving black-box optimization in vae latent space using decoder uncertainty. Advances in Neural Information Processing Systems, 34:802 814 14, 2021.
[38] Donald B Rubin. Matching to remove bias in observational studies. Biometrics, pages 159-183, 1973.
[39] Donald B Rubin. The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Statistics in medicine, 26(1):20-36, 2007.
[40] Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques. Recent advances and applications of machine learning in solid-state materials science. npj computational materials, 5(1):83, 2019.
[41] Peter Sharpe. NeuralFoil: An airfoil acrodynamics analysis tool using physics-informet machine learning. httts://githu: cww. ptteerdaaaarppp/eerraaolooll 2023.
[42] Nir Shlezinger, Yonina C Eldar, and Stephen P Boyd. Model-based deep learning: On the intersection of deep learning and optimization. IEEE Access, 10: 15384-115399 2022.
[43] Elizabeth A Stuart. Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25(1):1, 2010.
[44] Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learning in drug discovery and development. Nature reviews Drug discovery, 18(6):463-477, 2019.
[45] Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of chemical information and modeling, 62(23):5938-599 5951, 2022.
[46] Derek van Tilborg, Helena Brinkmann, Emanuele Criscuolo, Luke Rossen, Riza Özęelik, and Francesca Grisoni. Deep learning for low-data drug discovery: hurdles and opportunities. Current Opinion in Structural Biology, 86: 66:102818. 2024.
[47] Jing Wang, Cheng He, Runze Li, Haixin Chen, Chen Zhai, and Miao Zhang. Flow field prediction of supercritical airfoils via variational autoencoder based deep learning framework. Physies of Fluids, 33(8), 2021.
[48] Tianyu Wang, Marco Morucci, M Usaid Awan, Yameng Liu, Sudeepa Roy. Cynthia Rudin, and Alexander Volfovsky. Flame: A fast large-scale almost matching exactly approach to causal inference. Journal of Machine Learning Research, 22(31): 1-41,, 2021.
[49] Yuyang Wang, Kenji Shimada, and Amir Barati Farimani. Airfoil gan: encoding and syn- thesizing airfoils for aerodynamic shape optimization. Journal of Computational Design and Engineering, 1050):335 2023.
[50] Zhenpeng Yao, Yanwei Lum, Andrew Johnston, Luis Martin Mejia-Mendowe Xin Zhou, Yonggang Wen, Alán Sapuru-Guzkk Edward H Sargent, and Zhi Wei Seh. Machine learning for a sustainable energy future. Nature Reviews Materials, 8(3):202-211 2023.
[51] Jianmiao Yi and Feng Deng. Cooperation of thin-airfoil theory and deep learning for compact airfoil shape parameterization. Aerospace, 10(7):650, 2023.
[52] Kazuo Yonckura, Kazunari Wada, and Katsuyuki Suzuki. Generating various airfoil shapes with required lift coefficient using conditional variational autoencoders. arXiv preprint 1xxiv:2/06.09901, 2021.
12

---

/ supplemental material
A Deferred proofs
A.1 Proof of Theorem
Proof. The matched reconstruction objective for the MSE loss can be expressed
arg min
Σ
Σ
(l.a(z) (z + 8 11 ((((z) g(x + 6z)).
(2)
(0,
Assuming a model that is sufficient overparametrized, we can w.l.ogg. suppose that there some 0 which minimizes every and thus swap the and argmin. We are left with the objective of minimizing the inner sum:
arg min
Σ
(2) (ar 5,)11 < g(x - 5z))
Og
6a (0.A.,)
Noting that the the squared loss is minimized by the expected value and taking the data limit get:
fe. (z) (x)
Σ
82 Ag(x)) g(x 8z))
C
82 ()0,)()
(0 (dx g(x) g(x + 8z))
(4)
where constant C' is needed because the expected value should only be computed on the non-zero terms. To proceed, we consider rotation matrix R₂ for which the following holds:
R₂ Vg())=
(5)
and introduce the reparametrization z = 8g The last vector is also uniformly distributed
R2 A,) u(0, Az)
due to the uniform distribution on a ball being rotation invariant. Note that above we write (0, Am mean the push-forward of the distribution through the inverse rotation.
can now rewrite Equation as follows:
E,, Es_ (6x))o(z) g(x + 5n) = (B.
< g + 5
(6)
=R R. g(() g(x R
(7)
RR ".. Vg (x) R + elg, x. An))
(8)
Above, to go from line 7 to line 8 we Taylor expand g around
g(x + R2 = g(x) + Vg(z) R + e(g. x, Az)
with |e(g, x, A+) = ( and being the Lipschitz constant Vg.
The conditional in Equation is equal to
Vg(x))ll>
-egg, An) < Vg(x) R = ZR (x) =
(9)
or equivalently
-e(g, A.)
(10)
13

---

where we have denoted the first coordinate of as Thus, setting c
Equation 8 can be re-written as
B c)
R EU,,, (2/21 c) RØ²
:
(11)
E(zml 21 111001>00 c))
B(zi|zı > c) Ri²
(12)
E(2//2 c) R2,"" R. Vg(a) = B >c)
Vg(x).
(13)
[VV((>)l2 2
(V((2)22
We further note that [2 > c) (A, -(g. 1/2 due 21 being distributed uniformly in UO, Az). We have thus shown that
(A, + Vi(l)ila
foz (x)
Vg(x),
(14)
Vg<lll
and the constant in front of the gradient is positive for Ag whenever Ag = ( V Vo(elll)).
dg
The latter condition met
[a(())
b
A.2 Understanding the relation between the learned direction and the property gradient
Our approach entails constructing conditional 'matching distribution':
((x') if x < Az, g(x') g(x) E (6g, Ayl
(x') x
(15)
0 otherwise,
and then training a model to optimize the following regularized matched reconstruction objective:
{(f.p) Ez-p E [e(x', f()) + 8 [(x f()))]]
Note that by p(x) we denote the empirical density supported the training set p(x) - 8(x1- x) (and analogously for (.)).
Theorem 3. Let f* the optimal solution of the matched reconstruction objective matched recon
struction objective For any point v, the global minimizer is given by
Eg' [xc]] + fx
f' (x)
(16)
1 3
Further, for a λ₃ -i-Lipschitz and Am-sthoth function g. the vector f (x) is a-colinear with the gradient of g:
(F())))
VV( (22x))))ll x/1
whenever A <2 6, -0 12 in (1)-(1-))
Proof. The global minimizer of the matched reconstruction objective for the MSE loss is given by f*(x) Egao,, [a']] Ax
(17)
+ 3
which directly follows by taking the gradient of the mean-square error loss and setting it to zero. We next consider a C2 function g and Taylor expand it around
g(x') g(x) + Vg(x) (x' x)) c(x,x'),
14

---

where the norm of the approximation error is at most (e(x, - X2 v - x/11//2 - 22Ax/2.
Taking the expectation w.r.t. Pe yields
Ey' Ex' - g(x) Vg(x) (Bx')) x) Ey Bx (e(z,,
+ E L - <(x,
g(x) = (f(x)
- x)(1+ 3),
where in the last step we substituted the expectation by f((x). We notice that, as long as Bx'n [g(x') - ((x,x') g(x) > O, the learned direction is pointing towards a similar direc- tion the gradient:
Vo(x) -))
{()()), - x) - are cos
Vo(+))allM)) - x1/2 S $90
(18)
We expand the term within the condition in the following:
Ex'ra (g(x") E( x')] g(z) 2 inf E Ex (g(aa) (e(x, -({x)
8y
to determine that the following sufficient condition for <(Vg(z) ,f())) -x) to be below 90 degrees:
26y
A
A2
More generally, since
(f(=) 2 (1 [V)) I1]lf(=)) x|
6g d2
fl (a) x||2
sufficient condition for the normalized inner product to be above is
S Tf - I)
2 6 a(1 3))
1122
A2
One may also set (x) 2||2 Be'mps (x'1-11-01 Bl
in the equation above obtain the condition
1+0
A 2 by -ad1
Er' [o']
-1-88x=12
A2
as claimed.
A.3 Proof of Theorem
Proof. Set x' = (z).. Taking a Taylor expansion of around x', we deduce that for every x" E D the following holds:
p(a.)) S p(x') Vp(a') (a" -x) HH,('' 122 ||=" =012
with (z')l2) being the Hessian of p at x'. Taking the expectation w.r.t. Puz yields
[p(="))( & p(=') + Vp(z') [Br""aal"") - x') + [H, 1|2 Ey |||=""
=x'113)
2
or equivalently
p(x') 2 Bp [p("")] Vp(x') (E"""a" - x') [H 122 Ez" [||=" 2113
2
we further assume that the model trained to minimize the matched reconstruction objective with an MSE loss, we obtain the claimed result:
p(x') > En [p(a")] (,((''))20(2M
2
where o² (MM.) Egtrup, I1|x0 Egnmp [x'] is the variance induced by the matching process.
Li
Observe that, since the variance is upper bounded by (MM,) < 4A,, one may provide higher likelihood samples by restricting the matching process to consider closer pairs. Further, as perhaps expected, the likelihood is higher for smoother densities.
15

---

A.4 Understanding the matched reconstruction objective
The following analysis analyses the key characteristics of the proposed method. Though our analysis does not take into account the training process with stochastic gradient descent and the potentially useful inductive biases of the neural network fa. provides useful insights that qualitatively align with the practical model behavior.
Property The learned direction follows the property gradient. We first show that by optimizing the matched reconstruction objective, the model will learn to adjust an input point by approximately following the gradient Vg(æ)) of the property function.
For ease notation, we denote by Ma the subset of our paired dataset with all pairs that start from point æ.
Theorem 4. Let f3 be the optimal solution of the matched reconstruction objective For Å1- Lipschitz and sm-smooth function 9, the vector fz(Ø) is a-colinear with the gradient of g:
Vg(z)
a
2 (6, ai1(1+ (1)l -x112) whenever Ag < A2
The theorem reveals the role of the introduced thresholds in controlling how close we follow the property gradient: the larger the required property change 6y and the smaller the allowed input deviation A the closer U, tracks the gradient of g. We thus observe a trade-off between approximation and dataset size: selecting less conservative thresholds will lead to a larger |M| the expense of rougher approximation.
Property Regularization controls the step size. Hyperparameter 3 enables control the magnitude of the allowed input change (the step size in optimization parlance), with larger B encouraging smaller changes:
Corollary 1. In the setting of Theorem we have: lzf - æ112 = [[((())=(1a
1+3
Therefore, setting f to larger value can be handy for conservative design problems that prioritize distance constraints or when the property function is known be complex.
B Additional results for experiments
B.1 Details on Matched datasets
Table 4: Overview of matched dataset and choice of A, and A,
Dataset
Pairs
# Unique Samples in Train
Y Range (Control)
Y Range (Treatment)
8 Gaussians
746
96
[0.14, 0.86]
1]
Airfoils
125
200
[56.3, 82.8]
[65.2, 90.6]
Antibodies (T1)
362
268
[4, 8.9]
[6.5, 9.3]
B.2
Toy experiment Gaussians with anti-clockwise increasing property value
We include a toy example where the property is disentangled from the likelihood of the data. The results in Figure B.2 and Figure B.2 consistent with the discussion in the main text.
B.3 Setting A, and A
The choice parameters A5 and A, should be informed by the specific application. For example, in antibody design, domain experts recommend not using thresholds above a Levenshtein distance of as such differences are considered biologically irrelevant. Similarly, for Ay. knowing that noise in binding measurements can be up 0.3, chose 0.5 to ensure proper matching. When faced
16

---

Figure 6: New toy example d 50. Explicit guidance suffers in higher dimensions (shown here for d = 50) as there are increasingly more directions around the data in which the surrogate model is erroneous. Implicit guidance robust the higher data dimensionality because generated samples lie within distribution (as proven).
6-53
Mello improvement
NIL
ratlo of improvement
MET
rallo of imprvvement
MI
Figure 7: New toy example in d {2, 10, 50}, for 8-Gaussians with property increasing in anti- clockwise direction. Distribution of evaluation metrics from 10 repetitions of each experiment. ratio of improvement, higher is better, NLL lower is better.
with new dataset of n unique data points, good initial approach is to use the standard deviation of pairwise distances for æ, and the mean or median for property y:
2
Σ
d(z,,
Σ
n(n 1)
n(n 1)
Isiejšn
Igkelsn
Σ d(zi, Ii) A, S n(n - 1) Igi<jsn
How g(-) influences the calibration/selection of A When the g(.) non-smooth, or we have a very few datapoints, this certainly influences the choice of Ay, this case we opt for as small steps as possible since this will create many pairs which should give sense of the direction of the gradient around the sparse regions.
provide additional ablation study on how threshold choices affect the number of training pairs, showing that sufficiently large thresh- olds include all unique training points. For perfor- mance impact, please refer the ablation study in Figure
In the Figure 8
Figure 8: Offect of thresholds As and Ag the number of pairs. Though the number of pairs in- creases with larger thresholds, the benefit saturates (especially when increasing Ay).
17

---

Figure 9: Airfoils optimized with PropEn from multiple seeds. Red: seed designs. blue: PropEn candidates. To reduce the lift-drag ratio, PropEn tends to flatten the bottom of the airfoil to reduce the drag, while extend the front upwards increase lift.
Table 5:
Diffusion models do improve almost all hold out designs (RI ratio of improvement), however, only by a very small value (Al-average improvement.)
Airfoil experiment
N 200
AI
RI
Explicit guidance
15.04 + 1.21
6.53 6.35
Diffusion T 5 4.297 0.7
97.6 - 0.3
Diffusion T 15
4.3
99.2 + 0.2
Diffusion 7' 150
0.3
97.38
PropEn mix x2x
29.49 8.02
7.23
PropEn x2x
41.30 + 14.83
14.83
PropEn mix xy2xy
5.91 + 8.72
6.06 + 7.75
PropEn xy2xy
55.41 14.07
29.81
B.4 Airfoils
B.4.1
Additional baseline for airfoils
B.5
Example of PropEn designs for antibodies
PTRSY
Figure 10: HER2 binders from PropEn validated in wet lab experiments. Up: sequence alignment of the heavy chains the seed which the top sequence. The positions marked black correspond to mutational differences from the seed. Bottom: folded structures of the corresponding designs with the mutations to seed marked in green.
B.6
Details on experimental setups
Toy datasets
ablation studies: N E (100, 200), {2, 10,
matching thresholds: A₂ = A, = 1,
number of epochs: 500, batch size: 64
Criterion: MSELoss ()
Encoder:
18

---

Sequential
(0): Linear (in_uueeeees=22 out_featuuee=-0, bias-True)
(1) ReLU()
(2) : Linear (in (in_features=30, out out_ fut_features=30, bias=True)
(3) ReLU()
(4) : Linear Linear (in_featture==30, out_features=:0, bias=True)
(5) ReLU()
(6) (ine
out fut_feaaures=15, bias=True)
Decoder: Analogous to the encoder, starting with the compressed representation (15 features) and progressively reconstructing the original input size (2 features).
Toy Example Discriminator:
Sequential(
(0): Linear
fut_feaaures=30, bias=True)
(1): : ReLU()
(2) Linear fin_ fut_feaaures=30, bias=True)
(3) : ReLU()
(4): : Linear (in (in_features-=3, out_features=30, bias=True)
(5) : ReLU()
(6): Linear fineatain,, out_features=1, bias=True)
)
Airfoil
ablation studies: N E {200, 500, 1000),
matching thresholds: A Ag E {0.3, 0.5, 0.7,
number of epochs: 1000, batch size: 100
Criterion: MSELoss ()
Encoder:
Sequential
(0): Linear (in_fariin out_featuues=1000 bias=True)
(1): ReLU()
(2) : Linear (in_features= out out_features=100 100, bias-True)
(3) ReLU()
(4) : Linear (in_ sin_featuurs=100, out aut_featurrs=100, bias=True)
(5) ReLU()
(6) Linear (in_features=100, out out_features=50, bias=True)
Decoder: Analogous to the encoder, starting with the compressed representation (50 features) and progressively reconstructing the original input size (400 features).
AirFoil Discriminator:
Sequential
(0): Linear (in_eearrse,,,,, out_features=1000 bias=True)
(1): ReLU()
(2): : Linear (in_features=100 out_features=100, bias=True)
(3) : ReLU()
(4): Linear (in_features=100, out out_features=1000 bias=True)
(5) : ReLU()
(6) Linear (in features= out fut_features=1, bias=True)
)
19

---

Hyper-parameter choice. For optimizing the parameters of the baselines in the toy and engineering experiments, we conducted grid search over the learning rate ([le-2, le-5)), weight decay (l1e-2. le-5)), number of epochs ([300, 1000, 5000)), batch size (32, 64, 128), and number of neurons per layer ([30, 50, 100]).
Therapeutic Proteins
batch size: 32, training epochs: 300
matching thresholds: A {10, 15} mutational, edit distance, Ay 0.5
baselines: For WJS, used 1D Conv architecture (please A.1 in [38]), and for Lambo,
a bert-small transformer (please see C.1 in [39]).
Encoder(
(mlp): Linear (in featuressssss5555, out_featturs=128, bias=True)
(blocks): ModuleList(
ResNetBlock
(1n1): LayerNorm ( (128,), eps=1e-05, elementwise elementvise__fffiee affine=True)
(mlp1): Linear (in features=128,
featuuee==256, bias=True)
(1n2) LayerNorm ((trNoorm((266)) (256,), eps=1e-05,
(mlp2): Linear fet_ufeetueee-5566 out_features=128, bias=True)
(act):
Decoder: Analogous to the encoder, starting with the compressed representation (256 features) and
progressively reconstructing the original input size (6556).
Comparison to Bayesian Optimization. While both BO and PropEn aim optimization in small sample sizes, the two frameworks solve different problems. The goal of PropEn is to generate designs, whereas in BO/AL the goal is to choose the most promising designs that should be labeled in order to imporve predictors performance or find the best candidate, i.e. the focus is selection. In the context of optimizing designs, one would have a suite of (1) generative models, (2) property predictors and (3) BO/AL module that will do the final selection across pool of candidates. PropEn falls in (1), the category of generative models section that will contribute to the library of potential candidates. As side note, Lambo, a method we compared to, uses a BO inspired acquisition function guide the search for better designs, and we do compare it (favorably). but, we must highlight the difference, **PropEn and Lambo are generative models not BO/AL methods. We hope this clarifies the differences between the two frameworks and highlights their complementarity
B.7 Ablation study for toy experiments
Additional metrics included:
1. Uniqueness: number of unique designs divided by number of generated designs.
2. Novelty: number of designs proposed by the method that don't 't appear in the training data, divided by the number of generated designs.
20

---

Table PropEn Explicit guidance results on toy datasets, end of optimization trajectory. N the number of samples, d is the number of dimensions for projection (2D - dD datasets). Mean (std) over 10 repetitions of the experiment. For all metrics, higher is better.
N-100
N=200
d=10
d=50
d=100
d=10
d-50
d=100
8- N-Gaussions
ratio imp
PropEn mix x2x
85.71 5.01
77.89 16.79
81.58 12.58
92.882 5.64
93.16 6.66
91.88 8.49
PropEn mix xy2xy
68.42 18.23
65.41 9.04
80.70 6.37
79.19 10.
75.78 + 13.27
79.49 13.37
PropEn x2x
90.06 11:30
91.81
10:23
85.09
11.25
93.33 4.87
94.36 +: 5.64
93.77 4: 5.51
PropEn xy2xy
73.03 13.91
79.70
11.91
82, + 9.56
89.01 +: 5.87
78.39 +: 6.92
79.77 :: 10.76
Explicit
48.80 8.52
49.76 5.45
51.67 + 6.58
48.02 1: 4.88
48.02 :: 4.15
50.58 +: 5.01
loglike
PropEn mix x2x
-36.34 1.28
-37.06 2.53
-37 07.08 3.22
77.44 :: 2.53
77.61 4.25
77.550 2.68
PropEn mix xy2xy
10..11 4.90
-13.51 5.76
33.77
2.67
80.17 3.19
-33.16
2.20
-2..52 3.73
PropEn x2x
36.13 2.68
-35.12
2.76
36.04 2.91
76.995 2.70
76.80 4.83
-77.55
3.27
PropEn xy2xy
38.51 3.44
2.18 -38.70
38.47
5.12
-79.00 + 3.67
-81.79 1.87
82.992 1.05
Explicit
45.00 2.50
2.46 -44.11
-44.64 1.71
-99.34
3.02
-99.62 3.84
-98.88 3.63
uniqness
PropEn mix x2x
93.98 9.33
97.37
3.72
89.47
12.26
82.05 7.15
77.78 6.01
78.20 12.00
PropEn mix xy2xy
96.84
5.08
98.50 3.98
93.86
3.96
81.09
10.16
84.62 +: 8.79
86.08 +: 6.08
PropEn x2x
79.53 10.67
74.27
14.76
77.19
14.38
57.69
8.48
57.95
12.69
56.04 8.69
PropEn xy2xy
78.95 9.75
72.93
10.71
81.05 + 16.48
65.57
17.19
55.68 + 8.34
54.70 +: 11.75
Explicit
100.00 = 0.00
100.00 = 0.00
100.00 = 0.00
100.00 + 0.00
100.00 0.00
100.00 = 0.00
novelty
PnopEn mix x2x
100.00 0.00
100.00 = 0.00
100.00 : 0.00
100.00 1 0.00
100.00 +: 0.00
100.00 = 0.00
PropEn mix xy2xy
100.00
0.00
100.00
0.00
100.00 : 0.00
100.00 :::0::000000 0.00
100.00
0.00
100.00 = 0.00
PropEn x2x
100.00
0.00
100.00 : 0.00
100.00 1: 0.00
100.00 1: 0.00
100.00
0.00
100.00 :: 0.00
PropEn xy2xy
100.00 0.00
100.00 + 0.00
100.00 +: 0.00
100.00 +: 0.00
100.00 + 0.00
100.00 + 0.00
Explicit
100.00 0.00
100.00 : 0.00
100.00 1 0.00
100.00 1: 0.00
100.00 0.00
100.00 :: 0.00
pinwheel
ratio imp
PropEn mix x2x
86.73
10.56
84.02 13.94
80.94 10.80
87.53 10.41
91.35 + 7.75
86.41 5.68
PropEn mix xy2xy
71.56
13.09
66.32
9.5
70.39 15.13
56.41 15.88
51.57 20.53
61.54 1: 12.33
PropEn
89.29
10.09
85.97
12.89
91.03 9.54
92.67 6.69
87.75
PropEn xy2xy
71.67 12.14
65.13 17.10
69.67 9.04
52.38
47.86 16.98
63.40
17.54
Explicit
52.92 5.66
49.55 5.14
50.45
51.75 4.70
4.64
50.35
loglike
PropEn mix
-31.48 2.70
-31.44 2.62
31.77
0.96
-68.34 3.31
67.43
2.70
-9.000 2.71
PropEn mix xy2xy
33.22 2.13
335.33 1.26
35.06 3.84
-33.83 8.83
86.89
11.47
-80.11
4.17
PropEn x2x
-31.47 1.36
-31.12
2:20
-31.37
3.44
.6.888
2.88
66.66 1.96
-68.12
2.92
PropEn xy2xy
-31.96
137.10
4.35
-35.50
2.29
-57773
8.75
58.06
13.58
-79.83 = 7.58
Explicit
-0040 1.93
-39.38 2.25
-40.4 1.80
90.32 5.00
88.90 3.16
-87.91 4.63
uniquess
PropEn mix x2x
92.76
7.93
94.67 4.65
97.89 1.71
80.59
19
78.85 7.48
82.56 10.31
PropEn mix xy2xy
94.08
10.69
96.84 2.88
97.37
5.63
98.90
2.02
96.01
3.87
90.51 10.54
PropEn x2x
66.96
9.44
68.42
15.57
71.58
15.16
49.23
15.84
47.99
12.97
50.71 8.39
PropEn xy2xy
84.70 7.20
72.37
17.29
78.86
8.48
69.96 9.21
60.68
17.14
64.80 10.58
Explicit
100.00 0.00
100.00 4: 0.00
100.00 + 0.00
100.00
0.00
100.00 +: 0.00
100.00 + 0.00
novelty
PropEn mix x2x
100.00 0.00
100.00 + 0.00
100.00 + 0.00
100.00 + 0.00
100.00 4: 0.00
100.00 + 0.00
PropEn mix xy2xy
100.00 = 0.00
100.00
0.00
100.00
0.00
100.00 1:: 0.00
100.00 0.00
100.00 = 0.00
PropEn x2x
100.00 0.00
100.00 = 0.00
100, 00 0.00
100.00 0.00
100.00
0.00
100.00 0.00
PropEn xy2xy
100.00 0.00
100.00 00000000000 0.00
100.00
0.00
100.00 0.00
100.00
0.00
100.00 = 0.00
Explicit
100.00 0.00
100.00 0.00
100.00 0.00
100.00 - 0.00
100.00 1:000:::00 0.00
100.00 + 0.00

---

N=50
N=100
d=10
d:30
d=100
d=10
d:50
d=100
ratlo
PropEn mix
11.66
80.03
9.30
+ 7.41
92.56 4.27
4.05
94.02 6:21
PropEn mix sy2xy
70.00
14.05
82.71
8.44
$1.38 4: 5.52
90.38 6.09
822.02
9.07
59.74 7.56
PropEn x2x
81.87 #: 12.65
65.96
10.85
78.95 4: 6.15
92.82 3.78
93.08 d: 6.71
93.41 5.71
PropEn 2y
66.45
11.23
78.20
6.28
78.95 +: 13.42
88.28 5.31
80.22 = 7.79
82.05 9.85
Explicit
40.28 = 10.86
51.20 + 6.27
48.80 + 6.70
50.12 2: 6.42
48.02
4.30
50:12 = 3.69
Inglike
PropEn mix x2x
442.61 3.91
-41.18 + 2.06
441.42 1.99
-5.70 +: 2.80
-7.99 + 3.89
-80.2 020.82255 + 3.12
PropEn mix 2yy
-33.81
3..1
-42.28 t: 2.05
42.25
2.37
-87.73 d: 3.09
-9.188 4: 3,40
-88.79 +: 4.11
PropEn x2x
40.63 + 3.59
39.08 + 2.96
39.54
0.79
50.22 + 3.54
79.61 + 4.29
-80.88 + 3.74
PropEn xybxy
43.62 + 4.15
-40.37
+ 2.51
-00.44
4.33
-2.33
3.51
-84.99
4 6.19
-84.04 + 4.75
Explicit
-45 -5,00 2.50
-44.11 +: 2.45
-44.64 = 1.71
-99.35 +: 3.01
-99.63 4: 3.84
-98,90 +:: 3.63
unigness
PropEin mix x2x
100.00 + 0.00
100.00
0.00
100.00 = 0.00
100.00
0.00
100.00 + 0.00
100.00 1 0.00
PropEn mix sy2xy
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.00 d: 0.00
PropEn x2x
100.00 00 + 0.00
100.00
0.00
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.00 0.00
PropEn byb
100. 00 +: 0.00
100.00
0.00
100.00 1.0.00010 + 0.00
100.00
0.00
100.00 +: 0.00
100.00
0.00
Explicil
100.00 :: 0.00
100.00 0.00
100. 00 + 0.00
100.00
0.00
100.00 +: 0.00
100.00
0.00
novelity
PropEn mix x2x
100.00 1: 0.00
100.00 0.00
100.00 100.00010 +: 0.00
100.00
0.00
100.00 +: 0.00
100.00 0.00
PropEn mix Syy
100.00 000.000:2 + 0.00
100.00 0.00
1000 +: 0.00
100.00
0.00
100.00 + 0.00
100.00 0.00
PropEn x2s
100.00 = 0.00
100.00 0.00
100.00 + 0.00
100.00
0.00
100.00 +: 0.00
100.00 0.00
PropEn xylxy
100.00 + 0.00
100.00 0.00
100.00 + 0.00
100.00 0.00
100.00 0.00
100.00
Explicit
100.00 + 100.00
100.00
100.00
0.00
100.00
pinwheel
ratio
PropEn mix x2x
76.13 8.52
74.69 9.07
70.18 +: 14.31
81.98 4.55
85.58
8.33
81.79 7.09
PropEn mis ky2ky
63.52
8.45
77.89 + 6.86
70.39 + 14.60
70.70 8.74
63.53 = 12.08
76.15 8.02
PropEn x2x
80.77 9.76
77.78 + 7.34
84.21 + 9.85
86.15 4: 5.57
82.05
5.73
84.05 7.00
PropEn xylxy
61:79 5.53
60.53 :: 14.07
67.42 + 13.04
61.54 14.35
50.43 1: 7.01
57.81 14.70
Explicit
51.46 11.62
46.62 + 6.34
49.04 + 7.47
48.48
4.65
49.65
4.33
50.58 3.82
Inglike
PropEn mix x2x
13..12 +: 2.53
38.42 è: 3.01
39.73 3.12
99.000 # 5.16
82.53 4 2.71
82.38 + 0.64
PropEn máx 2y2
-39.38 4: 2,92
-37.39 t: 2.91
-39.10 e 3.74
-87.02
8.20
88.08 4 6.19
-84.10 +: 6.25
PropEn x2s
-37.56 1: 1,90
-37.00
4: 4.90
-36.36 = 6.56
-80.00 :: 6.26
- 76.46
1: 2.88
-76.90 1: 3.46
PropEn sy2sy
.0.92 1: 3.45
-41.60
1.60
+ 3.88
-39.22 = 4.34
-6.66.7
:: 6.00
89.45
8.21
-91.71 71 + 13:46
Explicit
40.440 +: 1.93
-39.58
+: 2.24
-00.5 = 1.81
90.00.3 =: 5.03
88.01
+ 3.16
-87.91 + 4.62
unigness
PropEn mix x2x
100.00 +: 0.00
100.00
0.00
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.00 1: 0.00
PropEn mix sy2sy
100.00 +: 0.00
100.00
0.00
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.00 d: 0.00
Propln x2x
100. D0 +: 0.00
100.00
0.00
100.00 + 0.00
99.23
2.43
100.00 + 0.00
100.00 de 0.00
PropEn xyb
100.00 00 +: 0.00
100.00
0.00
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.00
0.00
Explicit
100.00 00.00 1: 0.00
100.00 0.00
100. 00.00 4 0.00
100.00
0.00
100.00 1: 0.00
100.00
0.00
novelity
PropEnn mix
100.00 + 0.00
100.00 0.00
100.00 + 0.00
100.00
0.00
100.00 + 0.00
100.000 0.00
PropEn mix sy2xy
100.00 +
100.00 0.00
100.00 + 0.00
100.00 0.00
100.00 0.00
100.00
PropEn
100.00
100.00
100.00
100.00
100.00
PropEn xybxy
100.00
100.00
Explicit
100.00 +
100.00 0.00
100.00 + 0.00
100.00
100.00 0.00
100.00
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No]" provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: For all claims made in the introduction, we provide theoretical analysis in section and experimental validation in section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes],
23

---

Justification: We include paragraph on limitations of the current method at the end of section 4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All proofs are included in the supplementary material and corresponding references can be found in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we provide tabular results with mean and standard deviations for all experiments.
24

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We provide link to open source repository for the code, and details for reproducing experiments are in Appendic B.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc//uulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include all details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
We provide confidence intervals and standard deviations in all tables and figures report.
[Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All details can be found in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We conform with all guidlines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper proposes methods that help optimizing seed designs. It is representation-aanosii and such can help in advancing many scientific domains.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and funetioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
27

---

Justification:
he paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes Yes]
Justification:
We use the NeuralFoil software and propery cite in the manuscript.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide documented annonimized demo code for running part of the experiments in our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
28

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer:[NA)
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "xQLLcqKMVY", "reviewer": "Reviewer_WPJM", "summary": "The paper presents a generative framework for property enhancement. The proposed framework consists of only a generative model, and it's missing a discriminator that is usually found in other frameworks for guided design. This is achieved by training the generative model on a \"matched\" dataset that consists of paired examples $(x, x')$ where $x'$ is an \"enhanced\" version of $x$ from its immediate proximity (\"enhanced\" meaning it has a higher value of the property that is sought to be maximized in the guided design process, and \"proximity\" depends on the problem domain).\n\nA mathematical analysis of the proposed method shows that the learnt generative function converges to the gradient of the function computing the property and that examples sampled by the function are as likely as the training set. These analysis motivate the iterative application of the generative function to converge to a stationary point with enhanced property.\n\nThe authors claim that this framework is particularly well suited for applications with scarce data, and apply it to a toy problem and airfoil optimization, as well as protein optimization. In the latter application the method shows superior performance compared to other methods, as measured by wetlab experiments.", "questions": "1. Section 2.3 lines 126-127 \"As a property to optimize, we choose the log-likelihood of the data as estimated by a KDE with Gaussian kernel\" – it seems that optimizing for likely examples would both optimize the property as well as the likelihood of the examples. But theorem 1 is about increasing the property value and theorem 2 is about sampling likely examples. So I would have thought it more natural to choose an illustrative example that decouples the property to be optimized from the likelihood of the examples.\n\n1. I couldn't find the numbers reported in Table 3 in referenced papers [38, 39]. If it is a reproduction by the authors, then I would like to see that clearly stated in the main text or the table caption.\n\n1. The assumption of a \"sufficiently overparametrized model\" (line 472) seems a pretty strong one to me. How realistic is this assumption in the presented settings, and what would be the consequence for Theorem 1, and PropEn applicability in general, if the assumption does not hold?\n\nVarious nits, questions, comments:\n1. line 33: it should say \"even for deep neural networks\" (remove \"with\")\n1. caption Figure: missing space \"value.Bottom\"\n1. line 39: missing a word at \"[6] the\"\n1. line 48: \"have only be used provide\"\n1. line 57: \"PropEn\", not \"Propen\"\n1. line 113: \"from the its\"\n1. line 125: please add a reference to the \"well-konwn 2d pinwheel dataset\" (I don't know which dataset is meant exactly); similarly, the \"8-Gaussian\" dataset (line 157) is also missing a reference\n1. line 154: why is the regularized variant called \"mixup/mix\"? I find this confusing, as there is a well-known mixup technique in ML that does not seem related (https://arxiv.org/abs/1710.09412)\n1. line 165: how are metrics \"uniqueness\" and \"novelty\" defined?\n1. line 185: reference for \"NACA airfoils\" missing\n1. Section 3.1.2: how were the initial airfoil designs chosen?\n1. lines 216-218 and Figure 4: when varying one threshold, what is the value of the other threshold? a 2D density plot might point a more complete picture of the measured improvement when varying both thresholds.\n1. lines 219-222 and Figure 4: we can see that in a single example the $C_l/C_d$ ratio increased monotonically; but it's quite a stretch from that to claim that \"all designs ... are deemed plausible\" and that there is (always) a \"consistent enhancement\"; Figure 7 in the appendix (which is not mentioned in the main paper) does not add substantially to these claims\n1. line 225: it would be nice to reference figures/tables to show that \"mixup variant of PropEn may require longer training\"\n1. line 234: \"referred\", not \"refereed\"\n1. Figure 5 (left): why do the points have different sizes?\n1. Figure 8: what is the take-away from this figure?", "limitations": "The \"NeurIPS Paper Checklist\" in the appendix still needs to be filled in (currently it only shows the instructions).", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a simple method with mathematical justifications why it should work.\n\n1. Additionally to the toy examples, the method is applied to airfoil enhancement and protein optimization. The predictions of the latter are confirmed by wetlab experiments, and the reported results are better than state-of-the art methods.", "weaknesses": "1. The paper is missing several important implementation details. I would be especially interested in knowing the details of the training datasets and the constructed matched datasets, which I could not find anywhere in the paper or the appendix. Section B.1 in the appendix should be expanded to explain in detail what models were used. For example, line 571 states that a \"ResNet with 3 blocks\" was used to generate one-hot encoded AhO aligned sequences. But it's not clear to my why a 2D convolutional architecture would be used for such a problem, how exactly the data was represented, and what motivates the use of that precise model.\n\n1. The airfoil optimization is missing a strong baseline. From the presented results I find it hard to judge whether the method indeed works well across domains. I'm not familiar with the protein optimization experiments, so I cannot judge the power of the method by the presented results, especially since important implementation details are missing (see above weakness). Since all the findings in Section 3.2 are based on very few datapoints (per optimization target) and there is a wide intra-method variability, I would like to see some statistical analysis for presented data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Details on Training and Matched Datasets]**\nWe add the following table in the supplement:\n\n| Dataset         | # Pairs | # Unique Samples in Train | Y Range (Control) | Y Range (Treatment) |\n|-----------------|---------|---------------------------|-------------------|---------------------|\n| 8 Gaussians     | 1,746   | 96                        | [0.14, 0.86]      | [0, 1]              |\n| Airfoils        | 8,125   | 200                       | [56.3, 82.8]      | [65.2, 90.6]        |\n| Antibodies (T1) | 1,362   | 268                       | [4, 8.9]          | [6.5, 9.3]          |\n\n\n**[Details on Models and the choice of ResNet]**\n\nWe missed mentioning that the ResNet blocks in our network are non-convolutional. PropEn for antibodies uses fully connected layers with residual connections, one-hot encoding for input sequences, and layer normalization and GELU activation. This choice is tailored to the specific demands of antibody sequence analysis.\n\n**[Airfoil Optimization Baseline]**\n\nWe added a diffusion baseline for the airfoil experiment following the implementation from [1]. The model is a 3-layer MLP with 128 units per layer, sinusoidal time and input embeddings, and GELU activations. We conducted a grid search over batch size, hidden layer size, learning rate, and time steps and repeated the experiment five times for the best parameters.\nResults for $T \\in \\{ 5, 15, 50\\}\\$ show that the diffusion model optimizes almost all hold-out designs, however with minor improvements in the lift-drag ratio. For the final manuscript, we'll explore more advanced diffusion models, particularly from reference [2].\n\n1. [tiny-diffusion](https://github.com/tanelp/tiny-diffusion)\n2. [Diffusion-based 2DAirfoil Generation](https://github.com/tonyzyl/Diffusion-based-2DAirfoil-Generation/blob/main/models/airfoil_MLP.py)\n\n**[Statistical Analysis for Results in Section 3.2]**\n\nThe expense of wet-lab experiments constrains us, limiting the number of tested examples. The overall cost for the results presented was approximately $20,000. State-of-the-art methods like Walk-Jump and Lambo include results from a single target and seed, whereas we expand to 4 targets and 8 seeds.\n\nAs requested, we add statistical significance tests using Fisher's Exact test and Chi-Square tests.\n\n**Statistical Significance Tests for Table 3: Binding rate per method;**\n\n| Propen vs               | Fisher Odds Ratio | Fisher P-Val | Chi-Square stst| Chi-Square P-Val |\n|--------------------|-------------------|----------------|----------------------|---------------------|\n| Walk-Jump          | 10.4000           | **0.0**        | 26.9855              | **0.0**              |\n| Lambo (Guided)     | 22.2316           | **0.00**         | 41.6322              | **0.0**              |\n| Diffusion          | 2.7077            | 0.1022         | 1.6451               | 0.1996              |\n| Diffusion (Guided) | 1.3538            | 0.4295         | 0.0304               | 0.8616              |\n\n\n\n**Statistical Significance Tests for Table 4: Number of designs improving over seed per method;**\n\n| PropEn vs       | Fisher Odds Ratio | Fisher P-Val | Chi-Square stat | Chi-Square P-Val|\n|--------------------|-------------------|----------------|----------------------|---------------------|\n| Walk-Jump          | 10.4918           |  **0.0**        | 26.6097              | **0.0**             |\n| Lambo (Guided)     | 3.2350            | **0.0098**         | 5.1369               | **0.0234**              |\n| Diffusion          | 2.8478            | **0.0156**         | 4.4274               | **0.0354**             |\n| Diffusion (Guided) | 1.7947            | 0.0586         | 2.4436               | 0.1180              |\n\n\n**[Toy Exp;  choice of property]**\n\nWe propose a setup where the property and likelihood are disentangled, retaining the shape of 8 Gaussians but computing a property increasing in value in an anticlock-wise direction. Please see the attached PDF for results.\n\n**[Numbers in Table 3]**\n\nWe only reference the source publications and we independently run each of them our wet-lab experiments. This is why the numbers do not appear in their corresponding publications. \n\n**[Overparameterized Model Assumption]**\nThis is a general, common assumption in deep learning models. We will add a discussion on the overparameterized model; essentially, it is a standard assumption that has been theoretically [1, 2] and empirically explored in many prior works on deep learning. Overparameterization refers to the practice of using models with more parameters than the number of data points or the complexity strictly required to fit the training data. This assumption might seem counterintuitive initially, as it suggests using models that are larger than necessary. However, several compelling arguments justify this approach, highlighting its benefits for model performance, generalization, and learning dynamics.\n\n[1] Bengio, Y. et al. \"Representation learning: A review and new perspectives.\" IEEE transactions on pattern analysis and machine intelligence\".\n[2] Allen-Zhu et al  \"A convergence theory for deep learning via over-parameterization.\" ICML 2019.\n\n**[Response to minor comments]**\n\n[Mixup/Mix]\nThis version of PropEn, similar to the original Mixup you referenced, interpolates between examples. If this is confusing, we can consider renaming it.\n\n[Uniqueness and Novelty]\n- Uniqueness:  #unique designs divided by #gen designs.\n- Novelty: #designs proposed by the method that don't appear in the training data, divided by the #gen designs.\n\n[How Were the Initial Airfoil Designs Chosen?]\nRandom holdout dataset.\n\n[Figure 4]\nWhen varying \\(\\Delta_x\\), \\(\\Delta_y\\) is set to 1, and vice versa.\n\n[Figure 5]\nThe size corresponds to the number of designs for that seed.\n\n[Figure 8]\nExplicit guidance tends to fall off the manifold when many optimization steps are taken or seeds are OOD.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkrLYK9JLW", "reviewer": "Reviewer_zYK4", "summary": "This work proposes the method PropEn, which is inspired by the concept of matching techniques in econometrics. Using PropEn (specifically in scenarios with a lack of large datasets), the authors can expand the dataset, which will inherently help in design improvement, etc. To do this, they train a network to learn a mapping from an initial sample to another sample with an improved target attribute selected during the dataset curation phase. This model can further be sampled auto-regressively until it converges to a final candidate design.", "questions": "See above in weakness", "limitations": "No limitations are mentioned by the authors.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well written, and the motivation, methods, and results are clearly presented and easy to understand. \n- The proposed method is simple, generalizable, and appears effective. \n- The experiments and corresponding results are well-defined and support the claims made in the paper.", "weaknesses": "- Additional experiments that would add substantial value to the work would be to extend the data matching method to other models, rather than a simple encoder-decoder model with a reconstruction loss. As an example, provide experiments for diffusion models where the conditioning signal is the x_0 seed (or x_i in autoregressive sampling), and the generated design is x_{i+1}, which would solidify your claims about the generalizability of the data matching method.\n- Have the authors considered active learning approaches? This work is very close to active learning ideas, applied in the context of low data regimes.\n- The aspect of evaluating the target properties on the neighboring samples is not clearly explained. \n- the examples shown are simple and not really huge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Extending Matching to Other Models]**\nPlease see our general response. \n\n**[Comparison to Active Learning]**\nPlease see our general response. \n\n**[Evaluating Target Properties on Neighbouring Samples**]\n\nCould you please elaborate on what you mean by \"evaluating the properties on neighbouring samples\"? We are not entirely sure about the question, but we hope this explanation helps: The matching is based on observed or measured values of properties for each example; we do not use any predictor to obtain them. For the toy example, the property was computed using a KDE estimator. For the airfoils, lift and drag coefficients were obtained by Neural Foil, and for the antibodies, binding affinity was measured through wet lab experiments.\n\n**[Complexity and Diversity of Examples]**\n\nWe made an effort to demonstrate that this method works across different domains, neural network architectures, and tasks. Typically, machine learning papers follow a standard benchmark within a single application domain. Our experiments cover toy, engineering, and biology applications, showcasing a broader diversity. Additionally, we include results from wet-lab experiments, which cost $20,000 and take several months to complete. We hope the reviewer will reconsider and examine our manuscript with attention to these aspects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8TxrFFfh0N", "reviewer": "Reviewer_gEZM", "summary": "The paper address design optimization, which is the process of optimizing over a \"design\" parameter space to optimize over one or more observable outcomes in many scientific and engineering problems. The proposed framework \"PropEn\" uses a three step process, first identifying a \"matched dataset\" that pairs every sample with another with a better outcome; this is followed by training a model that takes a sample and predicts, essentially creating an autoregressive sampler. The technique is evaluated on multiple science and engineering benchmarks.", "questions": "Please see above", "limitations": "Yes, it appears so.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The matched objective is an interesting application of regressive sampling, and a different view on augmenting a small dataset, which is a common issue in many science problems. \n* The paper goes into theoretical justifications as to why a matched reconstruction objective is useful, suggesting that it approximates the gradient of an explicitly trained property predictor. \n* Its also shown that the generated samples have a high likelihood close to the training samples, indicating that they are indeed sampling from the training distribution.", "weaknesses": "* PropEn is an interesting idea for design optimization but it needs to be more thoroughly benchmarked against existing methods. For example, there is a host of methods that do Bayesian optimization, and variants thereof, that can work with as few as 10 samples (albeit in lower dimensions) which the paper does not consider. I think it sheds light into the behavior of PropEn in lower dimensions. More recent methods that use DNN based UQ estimates are able to go beyond 5-10D datasets effectively as well. \n* At a high level, matching and matched reconstruction appear to be very closely related to diffusion models. In my understanding, the diffusion process can be framed as matched reconstruction, where the property to maximize is the likelihood of the sample. Framing matched reconstruction through the lens of diffusion might not only strengthen the paper's formulation but also provide other theoretical insights. \n* Its unclear to me why neighborhoods in the design space are constructed using L2-balls, why is this the best way to identify similarity in x? How sensitive is the optimization to the size of the neighborhood? Or in another sense, a related question is -- how sensitive is matched reconstruction to the choice of sampling? for e.g. if you start with an LHC sampling (or any other variants) vs other types. \n* On small amounts of data, what kinds of regularizations keep the main model from overfitting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Benchmarking Against Bayesian Optimization (BO)]**\nPlease see our general response. \n\n**[Connection to Diffusion Models]**\nPlease see our general response. \n\n**[Neighborhoods in L2 Ball]**\n\nThe assumption regarding neighborhoods in L2 balls was made solely for theoretical purposes, to make the explanation more general and accessible. Neighborhoods in L2 balls naturally connect to training with Mean Squared Error (MSE) loss. However, the methodology is versatile and can be adapted to train with different losses; it can be shown that the same theory holds for Binary Cross-Entropy (BCE) loss as well. We are happy to expand on the proof in further discussion if the reviewer finds this interesting.\n\nSimilarity in \\(x\\) should be defined based on the application, which is why we showcase different datasets. For antibodies, we used Levenshtein distance as an example. We believe this choice gives freedom to include more inductive biases from domain experts.\n\n**[Sensitivity of Matched Reconstruction to Sampling Choice]**\n\nThere is no sampling choice made in PropEn; we keep all the possible pairs from the training data. This approach ensures that our method is comprehensive and does not rely on arbitrary sampling decisions.\n\n**[Regularization to Prevent Overfitting]**\n\nWe have identified two ways to avoid overfitting:\n\n1. Mixup technique: Including a reconstruction loss to the original sample in the training loss (the mixup variant in experiments) helps prevent overfitting. Due to the one-to-many matching scheme, we cannot perfectly reconstruct a single example, but rather learn how to interpolate between samples.\n\n2. Property Value Inclusion: Incorporating the value of the property in reconstruction controls the gradient steps (the xy2xy variant).\n\nThe theoretical background on why this works is included in the supplementary material, Section A.4, \"Understanding the Matched Reconstruction Objective.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N9wJ1Grlmx", "reviewer": "Reviewer_KGXt", "summary": "The paper develops a simple but novel deep learning framework to implicitly optimize a seed input design to iteratively improve upon a particular property $$ g(\\cdot) $$, until it reaches a fixed-point. In this way, the paper develops a novel framework for design optimization and provides a theoretical derivation to showcase that this implicit setup leads to an optimal learned neural network function $$ f^*(\\cdot) $$ whose iterative refinement of a seed design , approximates the direction of gradient of the property function of interest. The framework has been evaluated on three diverse design tasks and compared with several state-of-the-art design methods.", "questions": "1. How are $ \\Delta_x $ and $\\Delta_y  $ set / tuned?\n\n2. How does property under consideration $ g(\\cdot) $ influence the calibration / selection of $ \\Delta_y $?\n\n3. How can the current model be augmented to function with multiple properties that might need to be simultaneously improved?\n\n4. How are seed designs selected for initial inputs? i.e, what is considered a minimum viable seed design to be improved upon / refined until convergence? How does the model react to bad (i.e., flawed / noisy) designs?", "limitations": "A limitation of the current work is its inability to incorporate more than a single property of interest. However, the authors have explicitly mentioned this limitation and the reviewer believes that despite this, the proposed method is still valuable and can serve as a good foundation for future research in design optimization.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is well motivated and novel for the problem of design optimization with neural networks without adversarial classification (which are inherently hard to train).\n\n\n- The various experiments on toy as well as realistic design tasks like airfoil design optimization, antibody binding affinity design optimization demonstrate that the proposed method based on a simple `pair-wise matching` criterion is able to achieve better results than state-of-the-art models previously used for the same design task.\n\n\n- The insight and the proof demonstrating that the constructed optimal function $f^*(\\cdot)$ approximates the direction of the gradient of a property function $g(\\cdot) $ even though it is not explicitly trained to do so, is interesting and will benefit the space of design optimization.", "weaknesses": "- Further analysis about the effect of $ \\Delta_x $ and $\\Delta_y  $, their selection methodology and their sensitivity in various design contexts is necessary as the matching step critically depends on these two parameters.\n\n\n- The method has been demonstrated only in the context of designs that improve a single property $ g(\\cdot) $. However in reality designs usually need to consider multiple properties simultaneously. This brings into question the applicability of the current method to practical problems. \n\n\n- A rigorous analysis of the \"goodness\" of the seed design and the effect of said goodness on the convergence speed / accuracy is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Setting $\\Delta_x$ and $\\Delta_y$]**\n\nThe choice of parameters $\\Delta_x$ and $\\Delta_y$ should be informed by the specific application. For example, in antibody design, domain experts recommend not using thresholds above a Levenshtein distance of 8, as such differences are considered biologically irrelevant. Similarly, for $\\Delta_y$, knowing that noise in binding measurements can be up to 0.3, we chose 0.5 to ensure proper matching.\n\nWhen faced with a new dataset of \\(n\\) unique data points, a good initial approach is to use the standard deviation of pairwise distances for \\(x\\), and the mean or median for property \\(y\\):\n\n- $\n\\Delta_x \\lt \\sigma_d = \\sqrt{\\frac{2}{n(n-1)} \\sum_{1 \\leq i < j \\leq n} \\left( d(x_i, x_j) - \\frac{2}{n(n-1)} \\sum_{1 \\leq k < l \\leq n} d(x_k, x_l) \\right)^2}\n$\n\n- $\n\\Delta_y \\lt \\frac{2}{n(n-1)} \\sum_{1 \\leq i < j \\leq n} d(x_i, x_j)\n$\n\n\n**[How g(.) influences the calibration/selection of d_y]**\nWhen the $g(.)$ is non-smooth, or we have a very few datapoints, this certainly influences the choice of $\\Delta_y$, in this case we opt for as small steps as possible since this will create many pairs which should give sense of the direction of the gradient around the sparse regions. \nIn the attached PDF, we provide an additional ablation study on how threshold choices affect the number of training pairs, showing that sufficiently large thresholds include all unique training points. For performance impact, please refer to the ablation study in Figure 4.\n\n**[Multiple-Property Optimization]**\n\nWe agree that multiple-property optimization is crucial in real-world applications, as we have experienced ourselves. In such scenarios, we compute a multivariate rank over all properties of interest and optimize for that single score with PropEn. The multivariate score can be computed using the Hypervolume [1] or the CDF indicator, as suggested by [2].  Both methods work effectively, and we provide results on in-silico multi-property antibody optimization in the attached PDF. We select four developability properties: number of liabilities, hyrophobicity, positive and negative charge. We compute a multi-property ranking over these 4 properties, resulting in a single, scalar score we will use in PropEn. We then train PropEn models to optimize for each of these properties individually, and an *mPropEn* which uses the multivariate ranking score. In the figure in the attached pdf, we observe that (1) for two orthogonal targets, mPropEn suggests designs on the Pareto Front, while the single-objective properties tend to suggest designs at the tails of the front as expected, and (2) taking all four objectives into account, the top 20% of designs at the Pareto front come from the variants of mPropEn. This validates our idea of combining PropEn with multi-objective indicators, and we plan to expand on other datasets and experiments in further analysis.\n\n[1] \nZitzler, Eckart, et al. \"Performance assessment of multiobjective optimizers: An analysis and review.\" IEEE Transactions on evolutionary computation 7.2 (2003): 117-132.\n\n[2] Park, Ji Won, et al. \"BOtied: Multi-objective Bayesian optimization with tied multivariate ranks.\"ICML 2024.\n\n**[Seed Designs Selection and Flawed Designs]**\n\nGenerally, seed designs come from the training data. In cases where they do not, our model's tendency to optimize locally ensures that out-of-distribution (OOD) seeds are mapped to the closest neighborhood in the training set, as shown in Figure 2 in the submission and figure 1 and 2 in the attached pdf. In contrast, other methods, such as explicit guidance, often fall off the manifold and suggest designs in regions unsupported by the training data.\n\nWe have found that starting with seeds at the distribution's edge, with the highest possible values, helps achieve significant improvements, as these seeds have the best chance of extrapolating. The attached PDF provides examples of seeds where PropEn achieved higher values than anything observed in the training distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their time and thoughtful comments. Here we will summarize our response addressing common/essential concerns and then follow with point by point responses. In what follows we refer to the submited manuscript as ‘submission’ and the pdf accompanying our rebuttal as the ‘attached pdf’.

**[clarification]** We would first like to reiterate on the contributions of our work which we believe the reviewers might have missed or overlooked. We propose a general method, that is domain and architecture agnostic and can be straightforwardly applied to variety of tasks. In the submission we show experiments across various domains such as engineering and biology. We made effort to provide theoretical foundation for PropEn by relating it to implicitly approximating the gradient for the property of interest, and finally, we include an in-vitro validation experiment to compare to SOTA methods, that is, to the best of our knowledge, the most thorough experiment (see below) of this kind appearing in any publication (ML or otherwise). 

**[value and significance of antibody optimization results]** We hope the reviewers and AC do see the value of including a diverse, expensive wet-lab experimetnts evaluation. 
* It is very challenging, and therefore, exceedingly uncommon for machine learning papers to include in-vitro results -- much less ones that devote experimental budget to competing methods, that utilize expensive and highly accurate assays (SPR), and that include 3 targets and 8 seeds (previous works considered 1 seed). This is a contribution in itself because it shows the real applicability and advantage of PropEn. 
* We are comparing PropEn head-to-head with exceptionally strong baselines: WalkJump received the best paper award in ICLR 2024 and Lambo was a spotlight in NeurIPS 2023.  
* To strenghten our claims, we did a statistical significance test for the numbers in Table 3 and 4. Detailed results are included in our response to reviewer WPJM.

**[additional toy experiment]** as requested by reviewer WPJM we include a new toy example where the property is dissentangled from the likelihood of the data. We repeated the analysis as in the original submission, and all results can be found in the attached pdf. The conclusions are consistent with the disussion in the manuscript.

**[stronger baseline for the airfoil example]** We now add a diffusion baseline for the airfoil optimization application. The results in the attached pdf show that these models do improve the shape in many cases, however the improvement is not substantial. We hope this answers the concerns raised by revievewrs WPJM and zYK4.

**[multi-property optimization]** As requested by reviewer KGXt, we now include experiments on multi-property optimization by leveraging Hypervolume or CDF scores to compute and then optimize multivariate ranks. Our experiments show this strategy is effective and straightforward to use. Please see the attached pdf for results.

**[comparison to Bayesian Optimization/Active Learning (BO/AL)]**
This point was raised by two reviewers gEZM and zYK4. 
While both BO and PropEn aim at optimization in small sample sizes, the two frameworks solve different problems. The goal of PropEn is to generate designs, whereas in BO/AL the goal is to choose the most promising designs that should be labeled in order to imporve a predictors performance or find the best candidate, i.e. the focus is selection. In the context of optimizing designs, one would have a suite of (1) generative models, (2) property predictors and (3) BO/AL module that will do the final selection across pool of candidates. PropEn falls in (1), the category of generative models section that will contribute to the library of potential candidates. As a side note, Lambo, a method we compared to, uses a BO inspired acquisition function to guide the search for better designs, and we do compare to it (favorably), but, we must highlight the difference, **PropEn and Lambo are generative models not BO/AL methods. We hope this clarifies the differences between the two frameworks and highlights their complementarity.

**[matching and diffusion models]**
Another remark raised by two reviewers gEZM and zYK4.
We found it surprising that the reviewers find a connection between PropEn and diffusion models, as these are very different frameworks. We do not see how matching can be straightforwardly incorporated into the training procedure of a diffusion model, where samples are successively noised until they resemble a simple base (usually Gaussian) distribution, followed by a reversed process that learns to denoise to the original state. Substituting the base distribution by a data distribution corresponds to a complete reformulation of the denoising diffusion framework and requires dealing with the lack of an easy-to-compute density.   


We appreciate the reviewers insightful comments and suggestions, which have significantly improved our manuscript. We have carefully addressed each point in our responses and conducted additional experiments and analyses that we hope will make you reconsider and view our submission ready for acceptance. If there are any further questions, we are more than willing to address them.

Meta-review: The paper introduces PropEn, a generative framework for design optimization that enhances properties of interest without requiring a discriminator, typically found in adversarial models. PropEn works by training on a "matched" dataset, where each example is paired with an improved version from its vicinity, and the generative model learns to approximate the gradient of the property function. The framework is claimed to be particularly effective in data-scarce scenarios and is validated through applications in airfoil design and protein optimization, with the latter showing superior performance in wetlab experiments.

The reviews highlight PropEn's simplicity and solid theoretical foundation as key strengths, particularly praising its applicability to real-world problems like protein optimization. However, they point out several weaknesses, including a lack of detailed implementation information, insufficient baselines in the airfoil task, and limited statistical analysis of the results. The rebuttal has provided a satisfying answer to the major concerns and I will recommend acceptance and ask the authors to incorporate the feedback.
Decision: Accept (poster)