Title: Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient
Venue: NeurIPS 2024
Accepted: yes
Keywords: matching

property enhancement

gradient approximation

design optimization

shape optimization

antibodies
Abstract: Across scientific domains, generating new models or optimizing existing ones while meeting specific criteria is crucial. Traditional machine learning frameworks for guided design use a generative model and a surrogate model (discriminator), requiring large datasets. However, real-world scientific applications often have limited data and complex landscapes, making data-hungry models inefficient or impractical. We propose a new framework, PropEn, inspired by ``matching'', which enables implicit guidance without training a discriminator. By matching each sample with a similar one that has a better property value, we create a larger training dataset that inherently indicates the direction of improvement. Matching, combined with an encoder-decoder architecture, forms a domain-agnostic generative framework for property enhancement. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution, allowing efficient design optimization. Extensive evaluations in toy problems and scientific applications, such as therapeutic protein design and airfoil optimization, demonstrate PropEn's advantages over common baselines. Notably, the protein design results are validated with wet lab experiments, confirming the competitiveness and effectiveness of our approach. Our code is available at https://github.com/prescient-design/propen.

Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient

Abstract

Across scientific domains, generating new models or optimizing existing ones while meeting specific criteria is crucial. Traditional machine learning frameworks for guided design use a generative model and a surrogate model (discriminator), requiring large datasets. However, real-world scientific applications often have limited data and complex landscapes, making data-hungry models inefficient or impractical. We propose a new framework, PropEn , inspired by 'matching', which enables implicit guidance without training a discriminator. By matching each sample with a similar one that has a better property value, we create a larger training dataset that inherently indicates the direction of improvement. Matching, combined with an encoder-decoder architecture, forms a domain-agnostic generative framework for property enhancement. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution, allowing efficient design optimization. Extensive evaluations in toy problems and scientific applications, such as therapeutic protein design and airfoil optimization, demonstrate PropEn's advantages over common baselines. Notably, the protein design results are validated with wet lab experiments, confirming the competitiveness and effectiveness of our approach. Our code is available at https://github.com/prescient-design/propen.

1 Introduction

Navigating the complex world of design is a challenge in many fields, from engineering [35] to material science [33, 40] and life sciences [6]. In life sciences, the goal may be to refine molecular structures for drug discovery [6], focusing on properties like binding affinity or stability. In engineering, optimizing the shapes of aircraft wings or windmill blades to achieve desired aerodynamic traits like lift and drag forces is crucial [35]. The common thread in these fields is the design cycle : experts start with an initial design and aim to improve a specific property. Guided by intuition and expertise, they make adjustments and evaluate the changes. If the property improves, they continue this iterative optimization process. This cycle is repeated multiple times, making it time-consuming and resource-intensive. ML holds promise to reduce these costs, speed up design cycles, and create better-performing designs [3, 31, 44, 22].

Yet, progress in ML methods for design is hindered by practical challenges. The first challenge is limited data availability . Since gathering label measurements is resource intensive [46, 28, 50, 19], designers are more often than not constrained to very small-scale datasets. In addition, there are often non-smooth functional dependencies between the features and outcome , complicating approximation, even for deep neural networks [30, 45]. Traditional methods use two part frameworks, requiring discriminators to guide the property enhancement for examples produced by a generative model. Such discriminators should be able to reliably predict the property of interest given some training data or its latent representations. Because of the dependency on training a discriminator for guidance,

Implicit guidance

Step1:Matchdataset

Step 2:TrainPropEn

Legend

samples

_

neural net

embedded

PropEn latent space

samples

train dataset

matched dataset

Explicit guidance

>O

property

latent space

Step1:Traina generativemodel

Step 2: Train a property predictor Step 3: Guide optimization in latent space

we denote these methods as explicit guidance . While Genetic Algorithms were once prevalent [14], contemporary models like auto-encoders [47], GANs [49], and diffusion models now dominate both research and practice [31] the role of generative models. Despite their flexibility, such models face challenges typical to deep learning: they are 'data-hungry' and unreliable when encountering out-of-distribution examples [29, 21, 7, 37].

Motivated by these challenges, we propose a new approach inspired by the concept of 'matching'. Matching techniques in econometrics are used to address the challenge of selection bias and confounding when estimating causal effects in observational studies [2, 38, 39, 43]. These techniques aim to create comparable groups of units by matching treated and control observations based on observable characteristics. The basic idea behind matching is to identify untreated units that are similar to treated units in terms of observed covariates, effectively creating a counterfactual comparison group for each treated unit. Matching techniques in ML as in econometrics have only been used to provide more robust, reliable causal-effect estimation [24, 48, 9].

This work argues that, in lack of large datasets, matching allows for implicit guidance, completely sidestepping the need for training a discriminator (differentiable surrogate model). We match each sample with a similar one that has a superior value for the property of interest. By doing so, we obtain a much larger training dataset, inherently embedding the direction of property enhancement. We name our method PropEn and we illustrate it in Figure 1. By leveraging this expanded dataset within a standard encoder-decoder framework, we circumvent the need for a separate discriminator model. We show that PropEn is domain agnostic, can be applied to any data modality continuous or discrete. Additionally, PropEn alleviates some common problems with explicit guidance such as falling off the data manifold or requiring complex engineering.

Overall, our contributions are as follows:

We propose 'matching' (inspired by causal effect estimation) to expand small training datasets (subsection 2.1);

We provide a theoretical analysis on how training on a matched dataset implicitly learns an approximation of the gradient for a property of interest (subsection 2.2);

We provide guarantees that the proposed designs are as likely as the training distribution, avoiding common pitfalls where unreliable discriminators lead to unrealistic, pathological designs (subsection 2.3);

We demonstrate the effectiveness and advantages of implicit guidance through extensive experiments in both toy and real-world scientific problems, using both numerical and wet lab validation (section 3).

2 Property Enhancer (PropEn)

Given a set of initial examples, our objective is to propose a new design which is similar to the initial set, but, exceeds it in some property value of interest.

Problem setup. Concretely, we start with a dataset D = { x i , y i } n i =1 consisting of n observed examples x i ∈ R m drawn from a distribution p together with their corresponding properties y i = g ( x i ) ∈ R . Our objective is to determine ways to improve the property of a test example. Concretely, at test time, we are given a point x 0 and aim to identify some new point x new ∼ p close to x 0 such that g ( x new ) > g ( x 0 ) . In effect, our problem combines constrained optimization (maximize g ( x new ) while staying close to x 0 ) with sampling from a distribution (point x new should be likely according to p ).

Hereafter, we will refer to the initial example we wish to optimize as seed design and the model's proposal as candidate design. Our method, PropEn , entails three steps: (i) matching a dataset, (ii) approximating the gradient by training a model with a matched reconstruction objective and (iii) sampling with implicit guidance.

2.1 Match the dataset

We view the group of samples with superior property values as the treated group and their lower-value counter part as the control group. This motivates us to construct a 'matched dataset' for every ( x, y ) within D :

where ∆ x and ∆ y are predefined positive thresholds.

The matched dataset gives us a new and extended collection M whose size N = O ( n 2 ) ≫ n can significantly exceed that of the training set, depending on the choice of matching thresholds.

2.2 Approximate the gradient

After matching the dataset, we train a deep encoder-decoder network f θ over M by minimizing the matched reconstruction objective:

where ℓ is an appropriate loss for the data in question, such as an mean-squared error (MSE) or cross-entropy loss.

Before illustrating the properties of our method empirically, we perform a theoretical analysis. We show that minimizing the matched reconstruction objective yields a model that approximates the direction of the gradient of g ( · ) , even if no property predictor has been explicitly trained:

Theorem 1. Let f ∗ be the optimal solution of the matched reconstruction objective with a sufficiently small ∆ x . For any point x in the matched dataset for which p is uniform within a ball of radius ∆ x , we have f ∗ ( x ) → c ∇ g ( x ) for some positive constant c .

The detailed proof is provided in subsection A.1.

Remark 1. The proof of 1 is founded on the assumption that distribution is uniformly distributed within a ball of radius ∆ x around point x . This assumption is made to maintain the generality of the theorem without specific information about the sampling distribution, assuming uniformity avoids

Design

1.0

NLL

0.8

propertyvalue

IID Seed

0.6

37

5

10

15

20

0

100

0.4

Design

-150

0.2

NLL

200

250

0.0

(a)optimization trajectory

(b)optimization trajectory

-350

OODSeed

(IID seeds)

(OOD seeds)

10

15

20

25

optimization step

(c) NLL during optimization

introducing any biases that could arise from other distributional assumptions, such as symmetry, finite variance etc.

Remark 2. It can be shown that with with a matched reconstruction objective we learn a direction that is a -colinear with the gradient of g , avoiding the isotropy assumption. This leads to additional analysis on understanding the implications of the choices while matching, all included in Appendix A.

2.3 Optimize designs with implicit guidance

Training on a matched dataset allows for auto-regressive sampling. Starting with a design seed x 0 , for t = 1 , 2 , . . . , we can generate x t = f θ ( x t -1 ) until convergence, f θ ( x t ) = x t , s.t. g ( x t ) > g ( x t -1 ) . At test time, we feed a seed design x 0 to PropEn, and read out an optimized design x 1 from its output. We then proceed to iteratively re-feed the current design to PropEn until f θ ( x t ) = x t , which is analogous to arriving at a stationary point with ∇ g ( x t ) = 0 and we have exhausted the direction of property enhancement given the training data. Exploiting the implicit guidance from matching results in a trajectory of multiple optimized candidate designs.

We next show that optimized samples are almost as likely as our training set according to the data distribution p . This serves as a guarantee that the generated designs lie within distribution, as desired:

Theorem 2. Consider a model f ∗ trained to minimize the matched reconstruction objective. The probability of f ∗ ( x ) is at least

where ˆ µ x is the empirical measure on the dataset, H p ( x ) is the Hessian of p at x and σ 2 ( M x ) = E x ′ ∼ ˆ µ x [ ∥ x ′ -E x ′′ ∼ ˆ µ x [ x ′′ ] ∥ 2 2 ] is the variance induced by the matching process.

The detailed proof is provided in subsection A.3.

We use a synthetic example to illustrate optimizing designs with PropEn. We choose a 2d pinwheel dataset. As a property to optimize, we choose the log-likelihood of the data as estimated by a KDE with Gaussian kernel with σ = 0 . 01 . Figure 2 depicts in gray the training points, with the color intensity representing the value of the property-hence a higher/darker value is better in this example. After training PropEn, we take held out points (pink squares) and use them as seed designs. With orange x-markers, we illustrate PropEn candidates, with the color intensity increasing at each step t . We notice that PropEn moves towards the regions of the training data with highest property value, consistently improving at each step (right-most panel). Additionally, we also use out-of-distribution seeds, and we demonstrate in the middle panel that PropEn chooses to optimize them by proposing designs from the closest regions in the training data.

3 Experimental Results

We empirically evaluate PropEn on synthetic and real-world experiments to answer the following main questions: (i) Can PropEn be applied across various domains and datasets? (ii) Does PropEn provide reliable guidance, especially in situations with limited data and when dealing with out-ofdistribution examples? (iii) How effective is PropEn in recommending optimal designs? Can it suggest candidates with property values exceeding those in the training set? (iv) How does PropEn's performance vary with different data characteristics (e.g., dimensionality, sample size, heterogeneity) and hyperparameters (such as ∆ x , ∆ y , and regularization terms)? Our code is available at https: //github.com/prescient-design/propen https://github.com/prescient-design/propen.

Datasets. We consider three different data types: synthetic 2d toy datasets and their higher dimension transformations, NACA airfoil samples, and therapeutic antibody proteins. An overview of the data is given in Table 1. We present our results in two settings, in silico where we rely on experimental validation using computer simulations and solvers, and in vitro experiments where candidate designs were tested in a wet lab. Each of the experiments is evaluated under the baselines and metrics suitable for the domain.

PropEn variants. We investigate the utilization of matching and reconstruction within the PropEn framework. Two key considerations emerge: first, whether to reconstruct solely the input features (x2x) or both the input features and the property (xy2xy); second, the proximity to the initial sample, regulated by incorporating a straightforward reconstruction regularizer into the training loss ℓ ( f θ ( x ) , x ) . This regularized variant will be referred to as mixup/mix .

3.1 In silico experiments

3.1.1 Toy data

We choose two well-known multi-modal densities: pinwheel and 8-Gaussians. These are 2d datasets, but, in order to make the task more challenging, we expand the dimensionality to d ∈ { 10 , 50 , 100 } by randomly isometrically embedding the data within a higher dimensional space. Our findings are summarized in Figure 3 and we include the tabular results in item B.7. We empirically validate the four variants of PropEn and we compare against explicit guidance method: for consistency, we chose an auto-encoder of the same architecture as PropEn augmented with a discriminator for guidance in the latent space. We denote this baseline Explicit . We compare the methods by ratio of improvement , the proportion of holdout samples for which PropEn or baselines demonstrate enhanced property values. To assess the quality of the generated samples we report uniqueness and novelty in tables. We use a likelihood model derived from a Kernel Density Estimation (KDE) fit on the training data. The negative log-likelihood scores under this model serve as an indicator of in-distribution performance. Higher values for all metrics indicate better performance.

Results. Several insights can be gleaned from these experiments. When analyzing the results based on the number of samples, a clear trend emerges: as the number of training samples increases, PropEn consistently outperforms explicit guidance across all metrics, except for average improvement, where all methods exhibit similar behavior. The choice of the preferred metric may vary depending on the specific application; however, it is noteworthy that while explicit AE guidance improves approximately 50% of the designs for all datasets, PropEn demonstrates the potential to enhance up to 85% of the designs. Importantly, this improvement trend remains consistent regardless of the

0.07 1

d = 50

model

d = 50

- d = 100

d = 100

0.07 4

OS = p

model

d = 50

d = 100

d = 100

0.06

0.05

PropEn xy2yx

PropEn x2x

PropEn mix xy2yx

PropEn mix x2x

0.06

0.05

PropEn mix xy2yx

PropEn xy2yx

PropEn x2x

PropEn mix x2x

Density

0.04

Explicit

0.04

Explicit

0.03

0.03

0.02

0.02

0.01

0.01

0.00

0

ratio of improvement

50

100

60

-50

NLL

40

30

ratio of improvement

50

100

-50

NLL

40

30

0.00

0

ratio of improvement

50

100

60

50

NLL

40

-30

ratio of improvement

50

100

-50

40

NLL

-30

dimensionality of the data. Furthermore, an intriguing observation pertains to the performance of different variations of PropEn. It is noted that as the sample size increases, PropEn xy2xy does not exhibit an advantage over PropEn x2x. Moreover, the impact of iterative sampling with PropEn is notable. With each step, the property improves until it reaches a plateau after multiple iterations, albeit with a simultaneous drop in the uniqueness of the solution to around 80%. Nevertheless, iterative optimization can be continued until convergence, with all designs saved along the trajectory for later filtering according to user needs.

3.1.2 Engineering: airfoil optimization

In an engineering context, shape optimization entails altering the shape of an object to enhance its efficiency. NACA airfoils (National Advisory Committee for Aeronautics) [4], rooted in aerodynamics and parameterized by numerical values, serve as a well-documented benchmark due to their versatility. These airfoils span diverse aerodynamic characteristics, from high lift to low drag, making them ideal for exploring different optimization objectives. Given their integral role in aircraft wings, optimizing airfoil shapes can significantly impact aerodynamic performance by improving lift, drag, and other properties essential for aerospace engineering.

Data & experimental design. We generate NACA 4-digit series airfoil coordinates by choosing these parameters: M (maximum camber percentage), P (location of maximum camber percentage), and T (maximum thickness percentage). Each airfoil is represented by 200 coordinates, resulting in a 400 vector representation when flattened. Our objective is to optimize the lift-to-drag ratio ( C l /C d ratio) for each shape. We calculate lift and drag using NeuralFoil [41], a precise deep-learning emulator of XFoil [11].

Note that the lift-to-drag ratio is pivotal in aircraft design, reflecting the wing's lift generation efficiency relative to drag production. A high value signifies superior lift production with minimal drag, translating to enhanced fuel efficiency, extended flight ranges, and overall improved performance. This ratio is paramount in aerodynamic design and optimization, facilitating aircraft to travel farther and more efficiently through the air. Traditionally, engineers have relied on genetic algorithms guided by Gaussian Process models (kriging) [22], however, in recent years the community has moved towards ML-based methods which consist of a generative model that can be GAN-based [51, 49] or a variation of a V AE [26, 52, 31, 47]. Similarly, for the surrogate, guidance model, GPs and numerical solvers have been replaced by deep models [5, 41]. For our experiments we follow this standard setup: we choose a VAE-like baseline as it is the most similar architectural choice to PropEn, and for guidance we use a MLP. All networks (encoder, decoder and surrogate) are fully connected 3 layer MLPs with 50 units, ReLU activations per layer and a latent space of dimension 50. We randomly select 0.1% as holdout dataset for seeds, and use the rest for training.

Results. Our numerical findings are summarized in ?? . Similar to the toy dataset, the designs produced by PropEn variants demonstrate enhanced properties compared to those guided explicitly. Delving into further analysis with ablation studies, depicted in Figure 4(b) and (c), we observe that increasing the matching thresholds in PropEn correlates with higher rates of improvement. Remarkably, all designs within a PropEn trajectory are deemed plausible, as depicted in the accompanying figure. Moreover, a consistent enhancement in the lift-to-drag ratio C l /C d is noted along the optimization trajectory until convergence. This consistent trend underscores the effectiveness of PropEn in progressively refining airfoil designs to bolster their aerodynamic performance.

Interestingly, we find that in larger training datasets the threshold for property improvement ( ∆ y ) may not be necessary for optimization, as the PropEn x2x demonstrates satisfactory performance.

84

improvement

100

80

82

98

70

80

imp

CI/Cd of shape

78

96

ratio

60

76

average

94

Cl/Cd

50

74

92

40

threshold

72

70

90

30

y_th

x_th

88

20

68

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.6

0.7

0.8

0.9

1.0

0

2

4

6

8

10

12

14

thresholdvalue

thresholdvalue

iteration

We also notice that the mix variant of PropEn may require longer training. This observation is consistent with the notion that mix, by introducing a regularization term in the training loss, may improve at a slower rate compared to other PropEn variants. This slower improvement can be attributed to the regularization term's tendency to pull generated designs closer to the initial seed, thereby limiting the extent of exploration in the design space.

3.2 In vitro experiment: therapeutic protein optimization

The design of antibodies good functional and developability properties is essential for developing effective treatments for diseases ranging from cancer to autoimmune disorders. We here focus on the task of optimizing the binding affinity of a starting antibody (the seed) while staying close to it in terms of edit distance. The task is referred to as affinity maturation in the drug design literature and constitutes an essential and challenging step in any antibody design campaign.

Antibody binding affinity refers to the strength of the interaction between an antibody molecule and its target antigen. High binding affinity is crucial in antibody-based therapeutics as it determines the antibody's ability to recognize and bind to its target with high specificity and efficiency. We follow the standard practice of quantifying binding affinity by the negative log ratio of the association and dissociation constants (pKD), which represents the concentration of antigen required to dissociate half of the bound antibody molecules. Higher pKD indicates a tighter and more stable interaction, leading to improved therapeutic outcomes such as enhanced neutralization of pathogens or targeted delivery of drugs to specific cells.

Data & experimental design. The data collection process involved conducting low-throughput Surface Plasmon Resonance (SPR) experiments aimed at measuring with high accuracy the binding affinity of antibodies targeting three different target antigens: the human epidermal growth factor receptor 2 (HER2) and two additional targets that we denote as T1 and T2. For each of those targets, one or more seed designs were selected by domain experts. In the case of HER2, we used the cancer drug Herceptin as seed. We ensured the correctness of the SPR measurements by validating the fit of the SPR kinetic curves according to standard practices.

As the targets differ in the properties of their binding sites, we trained a PropEn model per each target (but for all seeds for that target jointly). For this application, we opted for the PropEn x2x mix variant. The reconstruction of the original sequence (mix) complies with antibody engineering wisdom that a candidate design should not deviate from a seed by more than small number of mutations. Similar to [34], we used a one-hot encoded representation of antibodies aligned according to the AHo numbering scheme [20] determined by ANARCI [12]. The encoder-decoder architecture is based on a ResNet [18] with 3 blocks each. We compare PropEn with four strong baselines: two state-of-the-art methods for guided and unguided antibody design namely walk-jump sampler [13] and lambo [17]; as well as two variants of a diffusion model trained on AHo antibody sequences differing on their use of guidance. The first one (labeled as diffusion ) is based on a variational diffusion model [25] trained on a latent space obtained by projecting AHo 1-hot representation using an encoder-decoder type of architecture similar to PropEn 's architecture; encoder-decoder model is trained simultaneously with the diffusion model. The second one (labeled as diffusion(guided) ) is a variant of the first one with

0.0

0.2

0.4

0.6

0.8

1.0

binging rate (per seed)

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

binding affinity improvement (per seed)

better than seed

PropEn

walk-jump

diffusion

diffusion (guided)

lambo (guided)

((a))

PropEn

walk-jump

lambo (guided)

diffusion

diffusion (guided)

3

2

1

0

1

binding affinity improvement

((b))

added guidance based on the iterative latent variable refinement idea described in the paper by Choi et al. [8], which ensures generating samples that are close to the initial seed.

We evaluate the set of designs in terms of their binding rate (fraction of designs tested that were binders), the percentage of designs than improve the seed, and their binding affinity improvement (pKD design - pKD of seed).

Results. As seen in Tables 2 and 3, PropEn excelled in generating functional antibodies with consistently high binding rates (94.6%) and 34.5% of the tested designs showed improve binding than the seed, outpacing other models in overall performance. To account for the trade-off between binding rate and affinity improvement (larger affinity improvement requires making risky mutations that might end-up killing binding), we visualize the Pareto front in Figure 5(a). In the plot, we mark the performance of each method for a specific seed design by placing a marker based on the achieved binding rate (x-axis) and maximum affinity improvement (y-axis). Compared to baseline methods, PropEn struck a beneficial trade-off, on average achieving a larger affinity improvement than methods with a high binding rate.

Figure 5(b) takes a closer look at the affinity improvement on the subset of designs that bound. As observed, all models produced some binders that were better than the seed, speaking for the strength of all considered models. Interestingly, none the top three models in terms of binding affinity improvement relied on explicit guidance, which aligns with our argument about the brittleness of explicit guidance in low-data regimes. Out of the those three models, PropEn generated two designs that improved the seed by at least one pKD unit (10 times better binder) followed by the walk-jump and the unguided diffusion model, that generated one such design each.

4 Related work

As design optimization has been ubiquitous across science domains, naturally our approach relates to a variety of methods and applications. In the molecular design domain, data are bound to discrete representation which can be challenging for ML methods. A natural way to circumvent that is by optimization in a latent continuous space. Bombarelli et al. [15] presented such an approach in their seminal work on optimizing molecules for chemical properties, and it has since spawn across different domains [6]. Recently, one of the common way for obtaining embeddings for explicit guidance relies on language models [27, 32]. One of the challenges of using a latent space is the issue for blindly guiding the optimization into ambiguous regions of the manifold where no training data was available [21, 29]. Follow up works attempt to address this problem [7, 16, 37] by incorporating uncertainty estimates into black-box optimization.

Another line of work perhaps more close to our approach is the notion of neural translation, where the goal is to go from one language to another by training on aligned datasets. [23, 10] have build on this idea to improve properties for small molecules translating one graph or sequence to another one with better properties. These works propose tailored approaches for domain specific applications. With PropEn we take a step further and propose a domain-agnostic framework that is empirically validated across different domains (uniquely including wetlab experiments). We also derive novel theoretical guarantees that illustrate the relation of the generated samples with the property gradient, as well as provide guarantees that our designs fall within distribution.

Previous works have also considered learning an optimizer for some function based on observed samples [42, 1, 36]. This is usually achieved by either (i) rendering the optimizer differentiable and training some of its hyperparameters; or (ii) by unfolding the iterative optimizer and treating each iteration as a trainable layer. Our approach is different, thanks to the matched reconstruction objective that lets us implicitly approximate the gradient of a function of interest.

5 Conclusion

Strengths. We introduced PropEn, a new method for implicit guidance in design optimization that approximates the gradient for a property of interest. We achieve this by leveraging matched datasets, which increase the size of the training data and inherently include the direction of property enhancement. Our findings highlight the versatility and effectiveness of PropEn in optimizing designs in engineering and drug discovery domains. We include wet lab in-vitro results for comparison with state-of-the-art baselines in therapeutic protein design. By utilizing thresholds for shape dissimilarity and property improvement, PropEn efficiently navigates the design space, generating diverse and high-performance configurations. We believe our method offers a simple yet effective recipe for design optimization that can be applied across various scientific domains.

Limitations. The matching step adds some computation overhead with complexity depending on the choice of distance metric. Since PropEn is targeting low data-regime applications, scalability is out of scope for the current work. However, we are considering on-the-fly distance evaluation or parallelisation across multiple nodes. The choice of distance metric for matching to a certain extent, can be considered a limitation because it requires understanding of the context for the application. However, this choice is also what allows for incorporating domain knowledge and constraints, which can be meaningful and necessary in the domain of interest (edit distance for antibodies, deviations only in the camber of the airfoil etc).

Future work. Immediate extensions of PropEn are applications to other molecular modalities, such as small molecules, material discovery, and optimization of adeno-associated virus vectors for gene-therapy. Additionally, we are keen to explore how different similarity metrics incorporate various inductive biases that can be leveraged for property optimization. Ongoing worthwhile efforts include developing a multi-property PropEn framework to address the optimization of properties simultaneously, offering a more comprehensive approach to the design process.

References

Akshay Agrawal, Shane Barratt, Stephen Boyd, and Bartolomeo Stellato. Learning convex optimization control policies. In Learning for Dynamics and Control , pages 361-373. PMLR,

2020.

Robert P Althauser and Donald Rubin. The computerized construction of a matched sample. American Journal of Sociology , 76(2):325-346, 1970.

Prasanna V Balachandran, Benjamin Kowalski, Alp Sehirlioglu, and Turab Lookman. Experimental search for high-temperature ferroelectric perovskites guided by two-step machine learning. Nature communications , 9(1):1668, 2018.

Roger E Bilstein. Orders of Magnitude: A History of the NACA and NASA, 1915-1990 , volume 4406. National Aeronautics and Space Administration, Office of Management . . . , 1989.

Mohamed Amine Bouhlel, Sicheng He, and Joaquim RRA Martins. Scalable gradient-enhanced artificial neural networks for airfoil shape design in the subsonic and transonic regimes. Structural and Multidisciplinary Optimization , 61(4):1363-1376, 2020.

John Bradshaw, Brooks Paige, Matt J Kusner, Marwin Segler, and José Miguel HernándezLobato. A model to search for synthesizable molecules. Advances in Neural Information Processing Systems , 32, 2019.

David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning , pages 773-782. PMLR, 2019.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 , 2021.

Oscar Clivio, Fabian Falck, Brieuc Lehmann, George Deligiannidis, and Chris Holmes. Neural score matching for high-dimensional causal inference. In International Conference on Artificial Intelligence and Statistics , pages 7076-7110. PMLR, 2022.

Farhan Damani, Vishnu Sresht, and Stephen Ra. Black box recursive translations for molecular optimization. arXiv preprint arXiv:1912.10156 , 2019.

Mark Drela. Xfoil: An analysis and design system for low reynolds number airfoils. In Low Reynolds Number Aerodynamics: Proceedings of the Conference Notre Dame, Indiana, USA, 5-7 June 1989 , pages 1-12. Springer, 1989.

James Dunbar and Charlotte M. Deane. ANARCI: antigen receptor numbering and receptor classification. Bioinformatics , 32(2):298-300, 09 2015.

Nathan C. Frey, Dan Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien LafranceVanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, Andreas Loukas, Vladimir Gligorijevic, and Saeed Saremi. Protein discovery with discrete walk-jump sampling. In The Twelfth International Conference on Learning Representations , 2024.

Mitsuo Gen and Runwei Cheng. Genetic algorithms and engineering optimization , volume 7. John Wiley & Sons, 1999.

Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science , 4(2):268-276, 2018.

Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science , 11(2):577-586, 2020.

Nate Gruver, Samuel Don Stanton, Nathan C. Frey, Tim G. J. Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jordan Hoffmann, Yohai Bar-Sinai, Lisa M Lee, Jovana Andrejevic, Shruti Mishra, Shmuel M Rubinstein, and Chris H Rycroft. Machine learning in a data-limited regime: Augmenting experiments with synthetic data uncovers order in crumpled sheets. Science advances , 5(4):eaau6792, 2019.

Annemarie Honegger and Andreas PluÈckthun. Yet another numbering scheme for immunoglobulin variable domains: an automatic modeling and analysis tool. Journal of molecular biology , 309(3):657-670, 2001.

David Janz, Jos van der Westhuizen, and José Miguel Hernández-Lobato. Actively learning what makes a discrete sequence valid. arXiv preprint arXiv:1708.04465 , 2017.

Jingchao Jiang. A survey of machine learning in additive manufacturing technologies. International Journal of Computer Integrated Manufacturing , 36(9):1258-1280, 2023.

Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to-graph translation for molecular optimization. arXiv preprint arXiv:1812.01070 , 2018.

Nathan Kallus. Deepmatch: Balancing deep covariate representations for causal inference using adversarial training. In International Conference on Machine Learning , pages 5067-5077. PMLR, 2020.

Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems , 34:21696-21707, 2021.

Jiaqing Kou, Laura Botero-Bolívar, Román Ballano, Oscar Marino, Leandro De Santana, Eusebio Valero, and Esteban Ferrer. Aeroacoustic airfoil shape optimization enhanced by autoencoders. Expert Systems with Applications , 217:119513, 2023.

Ben Krause, Subu Subramanian, Tom Yuan, Marisa Yang, Aaron Sato, and Nikhil Naik. Improving antibody affinity using laboratory data with language model guided design. bioRxiv , pages 2023-09, 2023.

Rafał Kurczab, Sabina Smusz, and Andrzej J Bojarski. The influence of negative training set size on machine learning-based virtual screening. Journal of cheminformatics , 6:1-9, 2014.

Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. In International conference on machine learning , pages 1945-1954. PMLR, 2017.

Karolina Kwapien, Eva Nittinger, Jiazhen He, Christian Margreitter, Alexey Voronov, and Christian Tyrchan. Implications of additivity and nonadditivity for machine learning and deep learning models in drug design. ACS omega , 7(30):26573-26581, 2022.

Jichao Li, Xiaosong Du, and Joaquim RRA Martins. Machine learning in aerodynamic shape optimization. Progress in Aerospace Sciences , 134:100849, 2022.

Lin Li, Esther Gupta, John Spaeth, Leslie Shing, Rafael Jaimes, Emily Engelhart, Randolph Lopez, Rajmonda S Caceres, Tristan Bepler, and Matthew E Walsh. Machine learning optimization of candidate antibody yields highly diverse sub-nanomolar affinity antibody libraries. Nature Communications , 14(1):3454, 2023.

Turab Lookman, Stephan Eidenbenz, Frank Alexander, and Cris Barnes. Materials discovery and design: By means of data science and optimal learning , volume 280. Springer, 2018.

Karolis Martinkus, Jan Ludwiczak, WEI-CHING LIANG, Julien Lafrance-Vanasse, Isidro Hotzel, Arvind Rajpal, Yan Wu, Kyunghyun Cho, Richard Bonneau, Vladimir Gligorijevic, and Andreas Loukas. Abdiffuser: full-atom generation of in-vitro functioning antibodies. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Joaquim RRA Martins. Aerodynamic design optimization: Challenges and perspectives. Computers & Fluids , 239:105391, 2022.

Vishal Monga, Yuelong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine , 38(2):18-44, 2021.

Pascal Notin, José Miguel Hernández-Lobato, and Yarin Gal. Improving black-box optimization in vae latent space using decoder uncertainty. Advances in Neural Information Processing Systems , 34:802-814, 2021.

Donald B Rubin. Matching to remove bias in observational studies. Biometrics , pages 159-183, 1973.

Donald B Rubin. The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Statistics in medicine , 26(1):20-36, 2007.

Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques. Recent advances and applications of machine learning in solid-state materials science. npj computational materials , 5(1):83, 2019.

Peter Sharpe. NeuralFoil: An airfoil aerodynamics analysis tool using physics-informed machine learning. https://github.com/peterdsharpe/NeuralFoil , 2023.

Nir Shlezinger, Yonina C Eldar, and Stephen P Boyd. Model-based deep learning: On the intersection of deep learning and optimization. IEEE Access , 10:115384-115398, 2022.

Elizabeth A Stuart. Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics , 25(1):1, 2010.

Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learning in drug discovery and development. Nature reviews Drug discovery , 18(6):463-477, 2019.

Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of chemical information and modeling , 62(23):5938-5951, 2022.

Derek van Tilborg, Helena Brinkmann, Emanuele Criscuolo, Luke Rossen, Rıza Özçelik, and Francesca Grisoni. Deep learning for low-data drug discovery: hurdles and opportunities. Current Opinion in Structural Biology , 86:102818, 2024.

Jing Wang, Cheng He, Runze Li, Haixin Chen, Chen Zhai, and Miao Zhang. Flow field prediction of supercritical airfoils via variational autoencoder based deep learning framework. Physics of Fluids , 33(8), 2021.

Tianyu Wang, Marco Morucci, M Usaid Awan, Yameng Liu, Sudeepa Roy, Cynthia Rudin, and Alexander Volfovsky. Flame: A fast large-scale almost matching exactly approach to causal inference. Journal of Machine Learning Research , 22(31):1-41, 2021.

Yuyang Wang, Kenji Shimada, and Amir Barati Farimani. Airfoil gan: encoding and synthesizing airfoils for aerodynamic shape optimization. Journal of Computational Design and Engineering , 10(4):1350-1362, 2023.

Zhenpeng Yao, Yanwei Lum, Andrew Johnston, Luis Martin Mejia-Mendoza, Xin Zhou, Yonggang Wen, Alán Aspuru-Guzik, Edward H Sargent, and Zhi Wei Seh. Machine learning for a sustainable energy future. Nature Reviews Materials , 8(3):202-215, 2023.

Jianmiao Yi and Feng Deng. Cooperation of thin-airfoil theory and deep learning for a compact airfoil shape parameterization. Aerospace , 10(7):650, 2023.

Kazuo Yonekura, Kazunari Wada, and Katsuyuki Suzuki. Generating various airfoil shapes with required lift coefficient using conditional variational autoencoders. arXiv preprint arXiv:2106.09901 , 2021.

Appendix / supplemental material

A Deferred proofs

A.1 Proof of Theorem 1

Proof. The matched reconstruction objective for the MSE loss can be expressed as

Assuming a model that is sufficient overparametrized, we can w.l.o.g. suppose that there is some θ x which minimizes every x and thus swap the sum and argmin. We are left with the objective of minimizing the inner sum:

Noting that the the squared loss is minimized by the expected value and taking the data limit we get:

where constant C is needed because the expected value should only be computed on the non-zero terms. To proceed, we consider a rotation matrix R x for which the following holds:

and introduce the reparametrization z = R x δ x . The last vector is also uniformly distributed

due to the uniform distribution on a ball being rotation invariant. Note that above we write R x U (0 , ∆ x ) m to mean the push-forward of the distribution through the inverse rotation.

We can now rewrite Equation 4 as follows:

Above, to go from line 7 to line 8 we Taylor expand g around x :

with | ϵ ( g, x, ∆ x ) | = O (∆ 2 x ) L and L being the Lipschitz constant of ∇ g .

The conditional in Equation 8 is equal to

or equivalently

where we have denoted the first coordinate of z as z 1 .

Thus, setting c = -ϵ ( g,x, ∆ x ) ∥∇ g ( x ) ∥ 2 , Equation 8 can be re-written as

We further note that E ( z 1 | z 1 > c ) = (∆ x --ϵ ( g,x, ∆ x ) ∥∇ g ( x ) ∥ 2 ) / 2 due to z 1 being distributed uniformly in U (0 , ∆ x ) . We have thus shown that

and the constant in front of the gradient is positive for ∆ x > | ϵ ( g,x, ∆ x ) | ∥∇ g ( x ) ∥ 2 . The latter condition is met whenever ∆ x = O ( ∥∇ g ( x ) ∥ 2 L ) .

A.2 Understanding the relation between the learned direction and the property gradient

Our approach entails constructing a conditional ' matching distribution ':

and then training a model to optimize the following regularized matched reconstruction objective:

Note that by ˆ p ( x ) we denote the empirical density supported on the training set ˆ p ( x ) = 1 n ∑ n i =1 δ ( x i -x ) (and analogously for ˆ µ x ).

Theorem 3. Let f ∗ be the optimal solution of the matched reconstruction objective matched reconstruction objective. For any point x , the global minimizer is given by

Further, for a λ 1 -Lipschitz and λ 2 -smooth function g , the vector f ∗ ( x ) -x is a -colinear with the gradient of g :

whenever ∆ x < 2 δ y -αλ 1 ∥ E x ′ ∼ ˆ µx [ x ′ ] -(1 -β ) x ∥ 2 λ 2 .

Proof. The global minimizer of the matched reconstruction objective for the MSE loss is given by

which directly follows by taking the gradient of the mean-squared error loss and setting it to zero. We next consider a C 2 function g and Taylor expand it around x :

where the norm of the approximation error is at most | ϵ ( x, x ′ ) | ≤ λ 2 ∥ x -x ′ ∥ 2 2 / 2 ≤ λ 2 ∆ x / 2 . Taking the expectation w.r.t. ˆ µ x yields

where in the last step we substituted the expectation by f ′ ( x ) . We notice that, as long as E x ′ ∼ ˆ µ x [ g ( x ′ ) -ϵ ( x, x ′ )] -g ( x ) > 0 , the learned direction is pointing towards a similar direction as the gradient:

We expand the term within the condition in the following:

to determine that the following sufficient condition for ∡ ( ∇ g ( x ) ⊤ , f ∗ ( x ) -x ) to be below 90 degrees:

a sufficient condition for the normalized inner product to be above α is

One may also set ∥ f ∗ ( x ) -x ∥ 2 = E x ′ ∼ ˆ µx [ x ′ ] -(1 -β ) x 1+ β in the equation above to obtain the condition

as claimed.

A.3 Proof of Theorem 2

Proof. Set x ′ = f ∗ ( x ) . Taking a Taylor expansion of p around x ′ , we deduce that for every x ′′ ∈ D the following holds:

with ∥ H p ( x ′ ) ∥ 2 being the Hessian of p at x ′ . Taking the expectation w.r.t. ˆ µ x yields

or equivalently

If we further assume that the model is trained to minimize the matched reconstruction objective with an MSE loss, we obtain the claimed result:

- ∥

p

(

x

)

x

′′

∼

ˆ

µ

x

[

p

(

x

)]

∥

2

2

,

where σ 2 ( M x ) = E x ′ ∼ ˆ µ x [ ∥ x ′ -E x ′′ ∼ ˆ µ x [ x ′′ ] ∥ 2 2 ] is the variance induced by the matching process.

Observe that, since the variance is upper bounded by σ 2 ( M x ) ≤ 4∆ x , one may provide higher likelihood samples by restricting the matching process to consider closer pairs. Further, as perhaps expected, the likelihood is higher for smoother densities.

H

p

(

x

′

)

σ

2

(

M

x

)

More generally, since

′

E

′′

A.4 Understanding the matched reconstruction objective

The following analysis analyses the key characteristics of the proposed method. Though our analysis does not take into account the training process with stochastic gradient descent and the potentially useful inductive biases of the neural network f θ , it provides useful insights that qualitatively align with the practical model behavior.

Property 1. The learned direction follows the property gradient. We first show that by optimizing the matched reconstruction objective, the model will learn to adjust an input point by approximately following the gradient ∇ g ( x ) of the property function.

For ease of notation, we denote by M x the subset of our paired dataset with all pairs that start from point x .

Theorem 4. Let f ∗ β be the optimal solution of the matched reconstruction objective. For a λ 1 -Lipschitz and λ 2 -smooth function g , the vector v x = f ∗ β ( x ) -x is a -colinear with the gradient of g :

The theorem reveals the role of the introduced thresholds in controlling how close we follow the property gradient: the larger the required property change δ y and the smaller the allowed input deviation ∆ x the closer v x tracks the gradient of g . Wethus observe a trade-off between approximation and dataset size: selecting less conservative thresholds will lead to a larger |M| at the expense of a rougher approximation.

Property 2. Regularization controls the step size. Hyperparameter β enables us to control the magnitude of the allowed input change (the step size in optimization parlance), with larger β encouraging smaller changes:

Therefore, setting β to a larger value can be handy for conservative design problems that prioritize distance constraints or when the property function is known to be complex.

B Additional results for experiments

B.1 Details on Matched datasets

B.2 Toy experiment - 8 Gaussians with anti-clockwise increasing property value

We include a toy example where the property is disentangled from the likelihood of the data. The results in Figure B.2 and Figure B.2 are consistent with the discussion in the main text.

B.3 Setting ∆ x and ∆ y

The choice of parameters ∆ x and ∆ y should be informed by the specific application. For example, in antibody design, domain experts recommend not using thresholds above a Levenshtein distance of 8, as such differences are considered biologically irrelevant. Similarly, for ∆ y , knowing that noise in binding measurements can be up to 0.3, we chose 0.5 to ensure proper matching. When faced

Explicit

PropEn

1.0

0.8

0.8

8°0

0.6

0.6

0.6

Q

0.4

0.4

0.4

Z'0

Z'0

0.2

Z-

1

0

1

2

0'0

-2

-1

0

1

2

0'0

Z-

-1

0

1

2

0'0

ratio imp

75

50

loglike

08-

60

d=2

ratio imp

100

50

d=10

loglike

60

80

d=o

ratio imp

100

75

50

loglike

09-

80

25

25

0

propen mixup xy2xy

x2x

propen x2x

+

propen xy2xy

100

propen mixup xy2xy

xZx

propen x2x

propen xy2xy

explicit

0

propen mixup xy2xy

xZx

propen xy2xy

explicit

-100

H

propen mixup xy2xy

x2x

propen x2x

propen xy2xy

explicit

propen mixup xy2xy

x2x

propen xy2xy

explicit

100

propen mixup xy2xy

x2x

propen x2x

propen xy2xy

explicit

propen mixup

propen mixup

propen mixup

propen mixup

propen mixup

propen mixup

ratioofimprovement

NLL

ratio ofimprovement

NLL

ratio ofimprovement

NLL

with a new dataset of n unique data points, a good initial approach is to use the standard deviation of pairwise distances for x , and the mean or median for property y :

How g ( · ) influences the calibration/selection of ∆ y When the g ( . ) is non-smooth, or we have a very few datapoints, this certainly influences the choice of ∆ y , in this case we opt for as small steps as possible since this will create many pairs which should give sense of the direction of the gradient around the sparse regions.

In the Figure 8, we provide an additional ablation study on how threshold choices affect the number of training pairs, showing that sufficiently large thresholds include all unique training points. For performance impact, please refer to the ablation study in Figure 4.

Number of Pairs vs , Values

Number of Unique vs , Values

2000

000000

80

1500

anbyun po Jaqun

70

60

1000

50

500

40

30

0

1.0

1.5

2.0

2.5

, Threshold Values

3.0

3.5

4.0

4.5

5.0

1.0

1.5

2.0

2.5

, Threshold Values

3.0

3.5

4.0

4.5

5.0

Number of Pairs vs , Values

Number of Unique vs , Values

60

230

220

58

210

56

200

54

190

52

0.4

0.6

0.8

, Threshold Values

1.0

T

1.4

1.6

1.8

0.4

0.6

0.8

, Threshold Values

1.0

1.2

1.4

1.6

1.8

B.4 Airfoils

B.4.1 Additional baseline for airfoils

B.5 Example of PropEn designs for antibodies

亿

EE

36

40

46

50

55

59

61

63

65

1

82A.82C.84.86.88.90

Z6

6

96.98.100.100B.100D.100F.102.104.106.108

SHVGYFDVWGQGTLV

H3

G

S

D

KD

Y

G

G

GF

N I K D

Y

H

W

R

上

A

GFNIKD

IH

WGQGTL

B.6 Details on experimental setups

Toy datasets

ablation studies: N ∈ { 100 , 200 } , d ∈ { 2 , 10 , 50 , 100 }

matching thresholds: ∆ x = ∆ y = 1 ,

number of epochs: 500, batch size: 64

Criterion :

MSELoss()

Encoder :

Decoder : Analogous to the encoder, starting with the compressed representation (15 features) and progressively reconstructing the original input size (2 features).

Toy Example Discriminator:

Decoder : Analogous to the encoder, starting with the compressed representation (50 features) and progressively reconstructing the original input size (400 features).

AirFoil Discriminator:

Hyper-parameter choice. For optimizing the parameters of the baselines in the toy and engineering experiments, we conducted a grid search over the learning rate ([1e-2, 1e-5]), weight decay ([1e-2, 1e-5]), number of epochs ([300, 1000, 5000]), batch size (32, 64, 128), and number of neurons per layer ([30, 50, 100]).

Therapeutic Proteins

batch size: 32, training epochs: 300

matching thresholds: ∆ x = ∈ { 10 , 15 } mutational, edit distance, ∆ y = 0 . 5

baselines: For WJS, we used a 1D Conv architecture (please see A.1 in [38]), and for Lambo, a bert-small transformer (please see C.1 in [39]).

Decoder : Analogous to the encoder, starting with the compressed representation (256 features) and progressively reconstructing the original input size (6556).

Comparison to Bayesian Optimization. While both BO and PropEn aim at optimization in small sample sizes, the two frameworks solve different problems. The goal of PropEn is to generate designs, whereas in BO/AL the goal is to choose the most promising designs that should be labeled in order to imporve a predictors performance or find the best candidate, i.e. the focus is selection. In the context of optimizing designs, one would have a suite of (1) generative models, (2) property predictors and (3) BO/AL module that will do the final selection across pool of candidates. PropEn falls in (1), the category of generative models section that will contribute to the library of potential candidates. As a side note, Lambo, a method we compared to, uses a BO inspired acquisition function to guide the search for better designs, and we do compare to it (favorably), but, we must highlight the difference, **PropEn and Lambo are generative models not BO/AL methods. We hope this clarifies the differences between the two frameworks and highlights their complementarity.

B.7 Ablation study for toy experiments

Additional metrics included:

Uniqueness: number of unique designs divided by number of generated designs.

Novelty: number of designs proposed by the method that don't appear in the training data, divided by the number of generated designs.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] ' is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] ' provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive' or 'we were unable to find the license for the dataset we used'). In general, answering '[No] ' or '[NA] ' is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist' ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: For all claims made in the introduction, we provide theoretical analysis in section 2 and experimental validation in section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] ,

Justification: We include a paragraph on limitations of the current method at the end of section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All proofs are included in the supplementary material and corresponding references can be found in the main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we provide tabular results with mean and standard deviations for all experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide link to open source repository for the code, and all details for reproducing experiments are in Appendic B.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include all details in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer:

[Yes] We provide confidence intervals and standard deviations in all tables and figures we report.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All details can be found in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with all guidlines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our paper proposes methods that help optimizing seed designs. It is representation-agnostic and as such it can help in advancing many scientific domains.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: he paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer:[Yes]

Justification: We use the NeuralFoil software and propery cite it in the manuscript.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide documented annonimized demo code for running part of the experiments in our paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a simple but novel deep learning framework to implicitly optimize a seed input design to iteratively improve upon a particular property $$ g(\\cdot) $$, until it reaches a fixed-point. In this way, the paper develops a novel framework for design optimization and provides a theoretical derivation to showcase that this implicit setup leads to an optimal learned neural network function $$ f^*(\\cdot) $$ whose iterative refinement of a seed design , approximates the direction of gradient of the property function of interest. The framework has been evaluated on three diverse design tasks and compared with several state-of-the-art design methods.\n\nSTRENGTHS:\n- The proposed method is well motivated and novel for the problem of design optimization with neural networks without adversarial classification (which are inherently hard to train).\n\n\n- The various experiments on toy as well as realistic design tasks like airfoil design optimization, antibody binding affinity design optimization demonstrate that the proposed method based on a simple `pair-wise matching` criterion is able to achieve better results than state-of-the-art models previously used for the same design task.\n\n\n- The insight and the proof demonstrating that the constructed optimal function $f^*(\\cdot)$ approximates the direction of the gradient of a property function $g(\\cdot) $ even though it is not explicitly trained to do so, is interesting and will benefit the space of design optimization.\n\nWEAKNESSES:\n- Further analysis about the effect of $ \\Delta_x $ and $\\Delta_y  $, their selection methodology and their sensitivity in various design contexts is necessary as the matching step critically depends on these two parameters.\n\n\n- The method has been demonstrated only in the context of designs that improve a single property $ g(\\cdot) $. However in reality designs usually need to consider multiple properties simultaneously. This brings into question the applicability of the current method to practical problems. \n\n\n- A rigorous analysis of the \"goodness\" of the seed design and the effect of said goodness on the convergence speed / accuracy is necessary.\n\nQUESTIONS:\n1. How are $ \\Delta_x $ and $\\Delta_y  $ set / tuned?\n\n2. How does property under consideration $ g(\\cdot) $ influence the calibration / selection of $ \\Delta_y $?\n\n3. How can the current model be augmented to function with multiple properties that might need to be simultaneously improved?\n\n4. How are seed designs selected for initial inputs? i.e, what is considered a minimum viable seed design to be improved upon / refined until convergence? How does the model react to bad (i.e., flawed / noisy) designs?\n\nLIMITATIONS:\nA limitation of the current work is its inability to incorporate more than a single property of interest. However, the authors have explicitly mentioned this limitation and the reviewer believes that despite this, the proposed method is still valuable and can serve as a good foundation for future research in design optimization.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper address design optimization, which is the process of optimizing over a \"design\" parameter space to optimize over one or more observable outcomes in many scientific and engineering problems. The proposed framework \"PropEn\" uses a three step process, first identifying a \"matched dataset\" that pairs every sample with another with a better outcome; this is followed by training a model that takes a sample and predicts, essentially creating an autoregressive sampler. The technique is evaluated on multiple science and engineering benchmarks.\n\nSTRENGTHS:\n* The matched objective is an interesting application of regressive sampling, and a different view on augmenting a small dataset, which is a common issue in many science problems. \n* The paper goes into theoretical justifications as to why a matched reconstruction objective is useful, suggesting that it approximates the gradient of an explicitly trained property predictor. \n* Its also shown that the generated samples have a high likelihood close to the training samples, indicating that they are indeed sampling from the training distribution.\n\nWEAKNESSES:\n* PropEn is an interesting idea for design optimization but it needs to be more thoroughly benchmarked against existing methods. For example, there is a host of methods that do Bayesian optimization, and variants thereof, that can work with as few as 10 samples (albeit in lower dimensions) which the paper does not consider. I think it sheds light into the behavior of PropEn in lower dimensions. More recent methods that use DNN based UQ estimates are able to go beyond 5-10D datasets effectively as well. \n* At a high level, matching and matched reconstruction appear to be very closely related to diffusion models. In my understanding, the diffusion process can be framed as matched reconstruction, where the property to maximize is the likelihood of the sample. Framing matched reconstruction through the lens of diffusion might not only strengthen the paper's formulation but also provide other theoretical insights. \n* Its unclear to me why neighborhoods in the design space are constructed using L2-balls, why is this the best way to identify similarity in x? How sensitive is the optimization to the size of the neighborhood? Or in another sense, a related question is -- how sensitive is matched reconstruction to the choice of sampling? for e.g. if you start with an LHC sampling (or any other variants) vs other types. \n* On small amounts of data, what kinds of regularizations keep the main model from overfitting?\n\nQUESTIONS:\nPlease see above\n\nLIMITATIONS:\nYes, it appears so.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This work proposes the method PropEn, which is inspired by the concept of matching techniques in econometrics. Using PropEn (specifically in scenarios with a lack of large datasets), the authors can expand the dataset, which will inherently help in design improvement, etc. To do this, they train a network to learn a mapping from an initial sample to another sample with an improved target attribute selected during the dataset curation phase. This model can further be sampled auto-regressively until it converges to a final candidate design.

STRENGTHS:
- The paper is very well written, and the motivation, methods, and results are clearly presented and easy to understand. 
- The proposed method is simple, generalizable, and appears effective. 
- The experiments and corresponding results are well-defined and support the claims made in the paper.

WEAKNESSES:
- Additional experiments that would add substantial value to the work would be to extend the data matching method to other models, rather than a simple encoder-decoder model with a reconstruction loss. As an example, provide experiments for diffusion models where the conditioning signal is the x_0 seed (or x_i in autoregressive sampling), and the generated design is x_{i+1}, which would solidify your claims about the generalizability of the data matching method.
- Have the authors considered active learning approaches? This work is very close to active learning ideas, applied in the context of low data regimes.
- The aspect of evaluating the target properties on the neighboring samples is not clearly explained. 
- the examples shown are simple and not really huge.

QUESTIONS:
See above in weakness

LIMITATIONS:
No limitations are mentioned by the authors.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a generative framework for property enhancement. The proposed framework consists of only a generative model, and it's missing a discriminator that is usually found in other frameworks for guided design. This is achieved by training the generative model on a \"matched\" dataset that consists of paired examples $(x, x')$ where $x'$ is an \"enhanced\" version of $x$ from its immediate proximity (\"enhanced\" meaning it has a higher value of the property that is sought to be maximized in the guided design process, and \"proximity\" depends on the problem domain).\n\nA mathematical analysis of the proposed method shows that the learnt generative function converges to the gradient of the function computing the property and that examples sampled by the function are as likely as the training set. These analysis motivate the iterative application of the generative function to converge to a stationary point with enhanced property.\n\nThe authors claim that this framework is particularly well suited for applications with scarce data, and apply it to a toy problem and airfoil optimization, as well as protein optimization. In the latter application the method shows superior performance compared to other methods, as measured by wetlab experiments.\n\nSTRENGTHS:\n1. The paper presents a simple method with mathematical justifications why it should work.\n\n1. Additionally to the toy examples, the method is applied to airfoil enhancement and protein optimization. The predictions of the latter are confirmed by wetlab experiments, and the reported results are better than state-of-the art methods.\n\nWEAKNESSES:\n1. The paper is missing several important implementation details. I would be especially interested in knowing the details of the training datasets and the constructed matched datasets, which I could not find anywhere in the paper or the appendix. Section B.1 in the appendix should be expanded to explain in detail what models were used. For example, line 571 states that a \"ResNet with 3 blocks\" was used to generate one-hot encoded AhO aligned sequences. But it's not clear to my why a 2D convolutional architecture would be used for such a problem, how exactly the data was represented, and what motivates the use of that precise model.\n\n1. The airfoil optimization is missing a strong baseline. From the presented results I find it hard to judge whether the method indeed works well across domains. I'm not familiar with the protein optimization experiments, so I cannot judge the power of the method by the presented results, especially since important implementation details are missing (see above weakness). Since all the findings in Section 3.2 are based on very few datapoints (per optimization target) and there is a wide intra-method variability, I would like to see some statistical analysis for presented data.\n\nQUESTIONS:\n1. Section 2.3 lines 126-127 \"As a property to optimize, we choose the log-likelihood of the data as estimated by a KDE with Gaussian kernel\" – it seems that optimizing for likely examples would both optimize the property as well as the likelihood of the examples. But theorem 1 is about increasing the property value and theorem 2 is about sampling likely examples. So I would have thought it more natural to choose an illustrative example that decouples the property to be optimized from the likelihood of the examples.\n\n1. I couldn't find the numbers reported in Table 3 in referenced papers [38, 39]. If it is a reproduction by the authors, then I would like to see that clearly stated in the main text or the table caption.\n\n1. The assumption of a \"sufficiently overparametrized model\" (line 472) seems a pretty strong one to me. How realistic is this assumption in the presented settings, and what would be the consequence for Theorem 1, and PropEn applicability in general, if the assumption does not hold?\n\nVarious nits, questions, comments:\n1. line 33: it should say \"even for deep neural networks\" (remove \"with\")\n1. caption Figure: missing space \"value.Bottom\"\n1. line 39: missing a word at \"[6] the\"\n1. line 48: \"have only be used provide\"\n1. line 57: \"PropEn\", not \"Propen\"\n1. line 113: \"from the its\"\n1. line 125: please add a reference to the \"well-konwn 2d pinwheel dataset\" (I don't know which dataset is meant exactly); similarly, the \"8-Gaussian\" dataset (line 157) is also missing a reference\n1. line 154: why is the regularized variant called \"mixup/mix\"? I find this confusing, as there is a well-known mixup technique in ML that does not seem related (https://arxiv.org/abs/1710.09412)\n1. line 165: how are metrics \"uniqueness\" and \"novelty\" defined?\n1. line 185: reference for \"NACA airfoils\" missing\n1. Section 3.1.2: how were the initial airfoil designs chosen?\n1. lines 216-218 and Figure 4: when varying one threshold, what is the value of the other threshold? a 2D density plot might point a more complete picture of the measured improvement when varying both thresholds.\n1. lines 219-222 and Figure 4: we can see that in a single example the $C_l/C_d$ ratio increased monotonically; but it's quite a stretch from that to claim that \"all designs ... are deemed plausible\" and that there is (always) a \"consistent enhancement\"; Figure 7 in the appendix (which is not mentioned in the main paper) does not add substantially to these claims\n1. line 225: it would be nice to reference figures/tables to show that \"mixup variant of PropEn may require longer training\"\n1. line 234: \"referred\", not \"refereed\"\n1. Figure 5 (left): why do the points have different sizes?\n1. Figure 8: what is the take-away from this figure?\n\nLIMITATIONS:\nThe \"NeurIPS Paper Checklist\" in the appendix still needs to be filled in (currently it only shows the instructions).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-23"}