Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient

Authors: nataša  tagasovska
vladimir  gligorijević
kyunghyun  cho
andreas  loukas
prescient / mldd
genentech  research
early  development
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:49:04

Abstract: Across scientific domains, generating new models or optimizing existing ones while meeting specific criteria is crucial. Traditional machine learning frameworks for guided design use a generative model and a surrogate model (discriminator), requiring large datasets. However, real-world scientific applications often have limited data and complex landscapes, making data-hungry models inefficient or impractical. We propose a new framework, PropEn, inspired by ``matching'', which enables implicit guidance without training a discriminator. By matching each sample with a similar one that has a better property value, we create a larger training dataset that inherently indicates the direction of improvement. Matching, combined with an encoder-decoder architecture, forms a domain-agnostic generative framework for property enhancement. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution, allowing efficient design optimization. Extensive evaluations in toy problems and scientific applications, such as therapeutic protein design and airfoil optimization, demonstrate PropEn's advantages over common baselines. Notably, the protein design results are validated with wet lab experiments, confirming the competitiveness and effectiveness of our approach. Our code is available at https://github.com/prescient-design/propen.

Main paper: 
 introduction: 
 Navigating the complex world of design is a challenge in many fields, from engineering [35] to material science [33, 40] and life sciences [6] . In life sciences, the goal may be to refine molecular structures for drug discovery [6] , focusing on properties like binding affinity or stability. In engineering, optimizing the shapes of aircraft wings or windmill blades to achieve desired aerodynamic traits like lift and drag forces is crucial [35] . The common thread in these fields is the design cycle: experts start with an initial design and aim to improve a specific property. Guided by intuition and expertise, they make adjustments and evaluate the changes. If the property improves, they continue this iterative optimization process. This cycle is repeated multiple times, making it time-consuming and resource-intensive. ML holds promise to reduce these costs, speed up design cycles, and create better-performing designs [3, 31, 44, 22 ]. Yet, progress in ML methods for design is hindered by practical challenges. The first challenge is limited data availability. Since gathering label measurements is resource intensive [46, 28, 50, 19] , designers are more often than not constrained to very small-scale datasets. In addition, there are often non-smooth functional dependencies between the features and outcome, complicating approximation, even for deep neural networks [30, 45] . Traditional methods use two part frameworks, requiring discriminators to guide the property enhancement for examples produced by a generative model. Such discriminators should be able to reliably predict the property of interest given some training data or its latent representations. Because of the dependency on training a discriminator for guidance, Figure 1 : Conceptual summary of implicit and explicit guidance. The task is increasing the size of the objects. Top -in implicit guidance, first we match the training dataset, by pairing each sample with the closest one w.r.t. its shape, which has a better property (size). Then, we train a encoder-decoder framework which due to the construction of the dataset learns a lower dimensional manifold where the embeddings are ordered by the property value. Bottom -in explicit guidance, we train two separate models: a generator and a discriminator that guides the optimization in latent space.we denote these methods as explicit guidance. While Genetic Algorithms were once prevalent [14] , contemporary models like auto-encoders [47] , GANs [49] , and diffusion models now dominate both research and practice [31] the role of generative models. Despite their flexibility, such models face challenges typical to deep learning: they are "data-hungry" and unreliable when encountering out-of-distribution examples [29, 21, 7, 37] .Motivated by these challenges, we propose a new approach inspired by the concept of "matching". Matching techniques in econometrics are used to address the challenge of selection bias and confounding when estimating causal effects in observational studies [2, 38, 39, 43] . These techniques aim to create comparable groups of units by matching treated and control observations based on observable characteristics. The basic idea behind matching is to identify untreated units that are similar to treated units in terms of observed covariates, effectively creating a counterfactual comparison group for each treated unit. Matching techniques in ML as in econometrics have only been used to provide more robust, reliable causal-effect estimation [24, 48, 9] . This work argues that, in lack of large datasets, matching allows for implicit guidance, completely sidestepping the need for training a discriminator (differentiable surrogate model). We match each sample with a similar one that has a superior value for the property of interest. By doing so, we obtain a much larger training dataset, inherently embedding the direction of property enhancement. We name our method PropEn and we illustrate it in Figure 1 . By leveraging this expanded dataset within a standard encoder-decoder framework, we circumvent the need for a separate discriminator model. We show that PropEn is domain agnostic, can be applied to any data modality continuous or discrete. Additionally, PropEn alleviates some common problems with explicit guidance such as falling off the data manifold or requiring complex engineering.Overall, our contributions are as follows:• We propose "matching" (inspired by causal effect estimation) to expand small training datasets (subsection 2.1); • We provide a theoretical analysis on how training on a matched dataset implicitly learns an approximation of the gradient for a property of interest (subsection 2.2);• We provide guarantees that the proposed designs are as likely as the training distribution, avoiding common pitfalls where unreliable discriminators lead to unrealistic, pathological designs (subsection 2.3); • We demonstrate the effectiveness and advantages of implicit guidance through extensive experiments in both toy and real-world scientific problems, using both numerical and wet lab validation (section 3).2 Property Enhancer (PropEn)Given a set of initial examples, our objective is to propose a new design which is similar to the initial set, but, exceeds it in some property value of interest.Problem setup. Concretely, we start with a dataset D = {x i , y i } n i=1 consisting of n observed examples x i ∈ R m drawn from a distribution p together with their corresponding properties y i = g(x i ) ∈ R. Our objective is to determine ways to improve the property of a test example. Concretely, at test time, we are given a point x 0 and aim to identify some new point x new ∼ p close to x 0 such that g(x new ) > g(x 0 ). In effect, our problem combines constrained optimization (maximize g(x new ) while staying close to x 0 ) with sampling from a distribution (point x new should be likely according to p).Hereafter, we will refer to the initial example we wish to optimize as seed design and the model's proposal as candidate design. Our method, PropEn, entails three steps: (i) matching a dataset, (ii) approximating the gradient by training a model with a matched reconstruction objective and (iii) sampling with implicit guidance. 
match the dataset: 
 We view the group of samples with superior property values as the treated group and their lower-value counter part as the control group. This motivates us to construct a "matched dataset" for every (x, y) within D:EQUATIONwhere ∆ x and ∆ y are predefined positive thresholds.The matched dataset gives us a new and extended collection M whose size N = O(n 2 ) ≫ n can significantly exceed that of the training set, depending on the choice of matching thresholds.After matching the dataset, we train a deep encoder-decoder network f θ over M by minimizing the matched reconstruction objective:ℓ(f θ ; M) = 1 |M| (x,x ′ )∈M ℓ(f θ (x), x ′ ), (matched reconstruction objective)where ℓ is an appropriate loss for the data in question, such as an mean-squared error (MSE) or cross-entropy loss.Before illustrating the properties of our method empirically, we perform a theoretical analysis. We show that minimizing the matched reconstruction objective yields a model that approximates the direction of the gradient of g(•), even if no property predictor has been explicitly trained: Theorem 1. Let f * be the optimal solution of the matched reconstruction objective with a sufficiently small ∆ x . For any point x in the matched dataset for which p is uniform within a ball of radius ∆ x , we have f * (x) → c∇g(x) for some positive constant c.The detailed proof is provided in subsection A.1. Remark 1. The proof of 1 is founded on the assumption that distribution is uniformly distributed within a ball of radius ∆ x around point x. This assumption is made to maintain the generality of the theorem without specific information about the sampling distribution, assuming uniformity avoids introducing any biases that could arise from other distributional assumptions, such as symmetry, finite variance etc. Remark 2. It can be shown that with with a matched reconstruction objective we learn a direction that is a-colinear with the gradient of g, avoiding the isotropy assumption. This leads to additional analysis on understanding the implications of the choices while matching, all included in Appendix A.Training on a matched dataset allows for auto-regressive sampling. Starting with a design seed x 0 , for t = 1, 2, . . . , we can generate x t = f θ (x t-1 ) until convergence, f θ (x t ) = x t , s.t. g(x t ) > g(x t-1 ).At test time, we feed a seed design x 0 to PropEn, and read out an optimized design x 1 from its output. We then proceed to iteratively re-feed the current design to PropEn until f θ (x t ) = x t , which is analogous to arriving at a stationary point with ∇g(x t ) = 0 and we have exhausted the direction of property enhancement given the training data. Exploiting the implicit guidance from matching results in a trajectory of multiple optimized candidate designs.We next show that optimized samples are almost as likely as our training set according to the data distribution p. This serves as a guarantee that the generated designs lie within distribution, as desired: Theorem 2. Consider a model f * trained to minimize the matched reconstruction objective. The probability of f * (x) is at leastp(f * (x)) ≥ E x ′ ∼μx [p(x ′ )] - ∥H p (f (x))∥ 2 σ 2 (M x ) 2 ,where μx is the empirical measure on the dataset, H p (x) is the Hessian of p at x and σ2 (M x ) = E x ′ ∼μx [∥x ′ -E x ′′ ∼μx [x ′′ ]∥ 2 2 ]is the variance induced by the matching process.The detailed proof is provided in subsection A.3. We use a synthetic example to illustrate optimizing designs with PropEn. We choose a 2d pinwheel dataset. As a property to optimize, we choose the log-likelihood of the data as estimated by a KDE with Gaussian kernel with σ = 0.01. Figure 2 depicts in gray the training points, with the color intensity representing the value of the property-hence a higher/darker value is better in this example.After training PropEn, we take held out points (pink squares) and use them as seed designs. With orange x-markers, we illustrate PropEn candidates, with the color intensity increasing at each step t.We notice that PropEn moves towards the regions of the training data with highest property value, consistently improving at each step (right-most panel). Additionally, we also use out-of-distribution seeds, and we demonstrate in the middle panel that PropEn chooses to optimize them by proposing designs from the closest regions in the training data. Datasets. We consider three different data types: synthetic 2d toy datasets and their higher dimension transformations, NACA airfoil samples, and therapeutic antibody proteins. An overview of the data is given in Table 1 . We present our results in two settings, in silico where we rely on experimental validation using computer simulations and solvers, and in vitro experiments where candidate designs were tested in a wet lab. Each of the experiments is evaluated under the baselines and metrics suitable for the domain.PropEn variants. We investigate the utilization of matching and reconstruction within the PropEn framework. Two key considerations emerge: first, whether to reconstruct solely the input features (x2x) or both the input features and the property (xy2xy); second, the proximity to the initial sample, regulated by incorporating a straightforward reconstruction regularizer into the training loss ℓ(f θ (x), x). This regularized variant will be referred to as mixup/mix.In silico experiments 
toy data: 
 We choose two well-known multi-modal densities: pinwheel and 8-Gaussians. These are 2d datasets, but, in order to make the task more challenging, we expand the dimensionality to d ∈ {10, 50, 100} by randomly isometrically embedding the data within a higher dimensional space. Our findings are summarized in Figure 3 and we include the tabular results in item B.7. We empirically validate the four variants of PropEn and we compare against explicit guidance method: for consistency, we chose an auto-encoder of the same architecture as PropEn augmented with a discriminator for guidance in the latent space. We denote this baseline Explicit. We compare the methods by ratio of improvement , the proportion of holdout samples for which PropEn or baselines demonstrate enhanced property values. To assess the quality of the generated samples we report uniqueness and novelty in tables. We use a likelihood model derived from a Kernel Density Estimation (KDE) fit on the training data. The negative log-likelihood scores under this model serve as an indicator of in-distribution performance.Higher values for all metrics indicate better performance.Results. Several insights can be gleaned from these experiments. When analyzing the results based on the number of samples, a clear trend emerges: as the number of training samples increases, PropEn consistently outperforms explicit guidance across all metrics, except for average improvement, where all methods exhibit similar behavior. The choice of the preferred metric may vary depending on the specific application; however, it is noteworthy that while explicit AE guidance improves approximately 50% of the designs for all datasets, PropEn demonstrates the potential to enhance up to 85% of the designs. Importantly, this improvement trend remains consistent regardless of the dimensionality of the data. Furthermore, an intriguing observation pertains to the performance of different variations of PropEn. It is noted that as the sample size increases, PropEn xy2xy does not exhibit an advantage over PropEn x2x. Moreover, the impact of iterative sampling with PropEn is notable. With each step, the property improves until it reaches a plateau after multiple iterations, albeit with a simultaneous drop in the uniqueness of the solution to around 80%. Nevertheless, iterative optimization can be continued until convergence, with all designs saved along the trajectory for later filtering according to user needs.In an engineering context, shape optimization entails altering the shape of an object to enhance its efficiency. NACA airfoils (National Advisory Committee for Aeronautics) [4] , rooted in aerodynamics and parameterized by numerical values, serve as a well-documented benchmark due to their versatility. These airfoils span diverse aerodynamic characteristics, from high lift to low drag, making them ideal for exploring different optimization objectives. Given their integral role in aircraft wings, optimizing airfoil shapes can significantly impact aerodynamic performance by improving lift, drag, and other properties essential for aerospace engineering.Data & experimental design. We generate NACA 4-digit series airfoil coordinates by choosing these parameters: M (maximum camber percentage), P (location of maximum camber percentage), and T (maximum thickness percentage). Each airfoil is represented by 200 coordinates, resulting in a 400 vector representation when flattened. Our objective is to optimize the lift-to-drag ratio (C l /C d ratio) for each shape. We calculate lift and drag using NeuralFoil [41] , a precise deep-learning emulator of XFoil [11] .Note that the lift-to-drag ratio is pivotal in aircraft design, reflecting the wing's lift generation efficiency relative to drag production. A high value signifies superior lift production with minimal drag, translating to enhanced fuel efficiency, extended flight ranges, and overall improved performance. This ratio is paramount in aerodynamic design and optimization, facilitating aircraft to travel farther and more efficiently through the air. Traditionally, engineers have relied on genetic algorithms guided by Gaussian Process models (kriging) [22] , however, in recent years the community has moved towards ML-based methods which consist of a generative model that can be GAN-based [51, 49] or a variation of a VAE [26, 52, 31, 47] . Similarly, for the surrogate, guidance model, GPs and numerical solvers have been replaced by deep models [5, 41] . For our experiments we follow this standard setup: we choose a VAE-like baseline as it is the most similar architectural choice to PropEn, and for guidance we use a MLP. All networks (encoder, decoder and surrogate) are fully connected 3 layer MLPs with 50 units, ReLU activations per layer and a latent space of dimension 50. We randomly select 0.1% as holdout dataset for seeds, and use the rest for training.Results. Our numerical findings are summarized in ??. Similar to the toy dataset, the designs produced by PropEn variants demonstrate enhanced properties compared to those guided explicitly.Delving into further analysis with ablation studies, depicted in Figure 4 (b) and (c), we observe that increasing the matching thresholds in PropEn correlates with higher rates of improvement. Remarkably, all designs within a PropEn trajectory are deemed plausible, as depicted in the accompanying figure. Moreover, a consistent enhancement in the lift-to-drag ratio C l /C d is noted along the optimization trajectory until convergence. This consistent trend underscores the effectiveness of PropEn in progressively refining airfoil designs to bolster their aerodynamic performance.Interestingly, we find that in larger training datasets the threshold for property improvement (∆ y ) may not be necessary for optimization, as the PropEn x2x demonstrates satisfactory performance. We also notice that the mix variant of PropEn may require longer training. This observation is consistent with the notion that mix, by introducing a regularization term in the training loss, may improve at a slower rate compared to other PropEn variants. This slower improvement can be attributed to the regularization term's tendency to pull generated designs closer to the initial seed, thereby limiting the extent of exploration in the design space.The design of antibodies good functional and developability properties is essential for developing effective treatments for diseases ranging from cancer to autoimmune disorders. We here focus on the task of optimizing the binding affinity of a starting antibody (the seed) while staying close to it in terms of edit distance. The task is referred to as affinity maturation in the drug design literature and constitutes an essential and challenging step in any antibody design campaign.Antibody binding affinity refers to the strength of the interaction between an antibody molecule and its target antigen. High binding affinity is crucial in antibody-based therapeutics as it determines the antibody's ability to recognize and bind to its target with high specificity and efficiency. We follow the standard practice of quantifying binding affinity by the negative log ratio of the association and dissociation constants (pKD), which represents the concentration of antigen required to dissociate half of the bound antibody molecules. Higher pKD indicates a tighter and more stable interaction, leading to improved therapeutic outcomes such as enhanced neutralization of pathogens or targeted delivery of drugs to specific cells.Data & experimental design. The data collection process involved conducting low-throughput Surface Plasmon Resonance (SPR) experiments aimed at measuring with high accuracy the binding affinity of antibodies targeting three different target antigens: the human epidermal growth factor receptor 2 (HER2) and two additional targets that we denote as T1 and T2. For each of those targets, one or more seed designs were selected by domain experts. In the case of HER2, we used the cancer drug Herceptin as seed. We ensured the correctness of the SPR measurements by validating the fit of the SPR kinetic curves according to standard practices.As the targets differ in the properties of their binding sites, we trained a PropEn model per each target (but for all seeds for that target jointly). For this application, we opted for the PropEn x2x mix variant. The reconstruction of the original sequence (mix) complies with antibody engineering wisdom that a candidate design should not deviate from a seed by more than small number of mutations.Similar to [34] , we used a one-hot encoded representation of antibodies aligned according to the AHo numbering scheme [20] determined by ANARCI [12] . The encoder-decoder architecture is based on a ResNet [18] with 3 blocks each. We compare PropEn with four strong baselines: two state-of-the-art methods for guided and unguided antibody design namely walk-jump sampler [13] and lambo [17] ; as well as two variants of a diffusion model trained on AHo antibody sequences differing on their use of guidance. The first one (labeled as diffusion) is based on a variational diffusion model [25] trained on a latent space obtained by projecting AHo 1-hot representation using an encoder-decoder type of architecture similar to PropEn's architecture; encoder-decoder model is trained simultaneously with the diffusion model. The second one (labeled as diffusion(guided)) is a variant of the first one with added guidance based on the iterative latent variable refinement idea described in the paper by Choi et al. [8] , which ensures generating samples that are close to the initial seed.We evaluate the set of designs in terms of their binding rate (fraction of designs tested that were binders), the percentage of designs than improve the seed, and their binding affinity improvement (pKD design -pKD of seed). Results. As seen in Tables 2 and 3 , PropEn excelled in generating functional antibodies with consistently high binding rates (94.6%) and 34.5% of the tested designs showed improve binding than the seed, outpacing other models in overall performance. To account for the trade-off between binding rate and affinity improvement (larger affinity improvement requires making risky mutations that might end-up killing binding), we visualize the Pareto front in Figure 5 (a). In the plot, we mark the performance of each method for a specific seed design by placing a marker based on the achieved binding rate (x-axis) and maximum affinity improvement (y-axis). Compared to baseline methods, PropEn struck a beneficial trade-off, on average achieving a larger affinity improvement than methods with a high binding rate. As observed, all models produced some binders that were better than the seed, speaking for the strength of all considered models. Interestingly, none the top three models in terms of binding affinity improvement relied on explicit guidance, which aligns with our argument about the brittleness of explicit guidance in low-data regimes. Out of the those three models, PropEn generated two designs that improved the seed by at least one pKD unit (10 times better binder) followed by the walk-jump and the unguided diffusion model, that generated one such design each. 
related work: 
 As design optimization has been ubiquitous across science domains, naturally our approach relates to a variety of methods and applications. In the molecular design domain, data are bound to discrete representation which can be challenging for ML methods. A natural way to circumvent that is by optimization in a latent continuous space. Bombarelli et al. [15] presented such an approach in their seminal work on optimizing molecules for chemical properties, and it has since spawn across different domains [6] . Recently, one of the common way for obtaining embeddings for explicit guidance relies on language models [27, 32] . One of the challenges of using a latent space is the issue for blindly guiding the optimization into ambiguous regions of the manifold where no training data was available [21, 29] . Follow up works attempt to address this problem [7, 16, 37] by incorporating uncertainty estimates into black-box optimization.Another line of work perhaps more close to our approach is the notion of neural translation, where the goal is to go from one language to another by training on aligned datasets. [23, 10] have build on this idea to improve properties for small molecules translating one graph or sequence to another one with better properties. These works propose tailored approaches for domain specific applications.With PropEn we take a step further and propose a domain-agnostic framework that is empirically validated across different domains (uniquely including wetlab experiments). We also derive novel theoretical guarantees that illustrate the relation of the generated samples with the property gradient, as well as provide guarantees that our designs fall within distribution.Previous works have also considered learning an optimizer for some function based on observed samples [42, 1, 36] . This is usually achieved by either (i) rendering the optimizer differentiable and training some of its hyperparameters; or (ii) by unfolding the iterative optimizer and treating each iteration as a trainable layer. Our approach is different, thanks to the matched reconstruction objective that lets us implicitly approximate the gradient of a function of interest. 
conclusion: 
 Strengths. We introduced PropEn, a new method for implicit guidance in design optimization that approximates the gradient for a property of interest. We achieve this by leveraging matched datasets, which increase the size of the training data and inherently include the direction of property enhancement. Our findings highlight the versatility and effectiveness of PropEn in optimizing designs in engineering and drug discovery domains. We include wet lab in-vitro results for comparison with state-of-the-art baselines in therapeutic protein design. By utilizing thresholds for shape dissimilarity and property improvement, PropEn efficiently navigates the design space, generating diverse and high-performance configurations. We believe our method offers a simple yet effective recipe for design optimization that can be applied across various scientific domains.Limitations. The matching step adds some computation overhead with complexity depending on the choice of distance metric. Since PropEn is targeting low data-regime applications, scalability is out of scope for the current work. However, we are considering on-the-fly distance evaluation or parallelisation across multiple nodes. The choice of distance metric for matching to a certain extent, can be considered a limitation because it requires understanding of the context for the application. However, this choice is also what allows for incorporating domain knowledge and constraints, which can be meaningful and necessary in the domain of interest (edit distance for antibodies, deviations only in the camber of the airfoil etc).Future work. Immediate extensions of PropEn are applications to other molecular modalities, such as small molecules, material discovery, and optimization of adeno-associated virus vectors for gene-therapy. Additionally, we are keen to explore how different similarity metrics incorporate various inductive biases that can be leveraged for property optimization. Ongoing worthwhile efforts include developing a multi-property PropEn framework to address the optimization of properties simultaneously, offering a more comprehensive approach to the design process.Appendix / supplemental material A Deferred proofs A.1 Proof of Theorem 1Proof. The matched reconstruction objective for the MSE loss can be expressed asEQUATIONAssuming a model that is sufficient overparametrized, we can w.l.o.g. suppose that there is some θ x which minimizes every x and thus swap the sum and argmin. We are left with the objective of minimizing the inner sum:EQUATIONNoting that the the squared loss is minimized by the expected value and taking the data limit we get:EQUATIONwhere constant C is needed because the expected value should only be computed on the non-zero terms. To proceed, we consider a rotation matrix R x for which the following holds:EQUATIONand introduce the reparametrization z = R x δ x . The last vector is also uniformly distributedz ∼ R x U(0, ∆ x ) m ∼ U(0, ∆ x ) mdue to the uniform distribution on a ball being rotation invariant. Note that above we write R x U(0, ∆ x ) m to mean the push-forward of the distribution through the inverse rotation.We can now rewrite Equation 4 as follows:EQUATIONAbove, to go from line 7 to line 8 we Taylor expand g around x:EQUATIONx )L and L being the Lipschitz constant of ∇g. The conditional in Equation 8 is equal toEQUATIONor equivalentlyEQUATION)where we have denoted the first coordinate of z as z 1 .Thus, setting c = -ϵ(g,x,∆x) ∥∇g(x)∥2 , Equation 8 can be re-written asEQUATIONWe further note that E(z 1 |z 1 > c) = (∆ x --ϵ(g,x,∆x) ∥∇g(x)∥2 )/2 due to z 1 being distributed uniformly in U (0, ∆ x ). We have thus shown thatEQUATIONand the constant in front of the gradient is positive for ∆ x > |ϵ(g,x,∆x)| ∥∇g(x)∥2 . The latter condition is met whenever∆ x = O( ∥∇g(x)∥2 L ). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our approach entails constructing a conditional 'matching distribution':EQUATIONand then training a model to optimize the following regularized matched reconstruction objective:ℓ(f, p) = E x∼ p[E x ′ ∼μx [ℓ(x ′ , f (x)) + β ℓ(x, f (x))]].Note that by p(x) we denote the empirical density supported on the training set p(x) = 1 n n i=1 δ(x ix) (and analogously for μx ). Theorem 3. Let f * be the optimal solution of the matched reconstruction objective matched reconstruction objective. For any point x, the global minimizer is given byEQUATIONFurther, for a λ 1 -Lipschitz and λ 2 -smooth function g, the vector f * (x) -x is a-colinear with the gradient of g:a ≤ ∇g(x) ⊤ (f * (x) -x) ∥∇g(x)∥ 2 ∥f * (x) -x∥ 2 whenever ∆ x < 2 δy-αλ1∥E x ′ ∼ μx [x ′ ]-(1-β)x∥2 λ2.Proof. The global minimizer of the matched reconstruction objective for the MSE loss is given byEQUATIONwhich directly follows by taking the gradient of the mean-squared error loss and setting it to zero.We next consider a C 2 function g and Taylor expand it around x:g(x ′ ) = g(x) + ∇g(x) ⊤ (x ′ -x) + ϵ(x, x ′ ),where the norm of the approximation error is at most |ϵ(x, x ′ )| ≤ λ 2 ∥x -x ′ ∥ 2 2 /2 ≤ λ 2 ∆ x /2.Taking the expectation w.r.t. μx yieldsE x ′ ∼μx [g(x ′ )] -g(x) = ∇g(x) ⊤ (E x ′ ∼μx [x ′ ] -x) + E x ′ ∼μx [ϵ(x, x ′ )] ⇔ E x ′ ∼μx [g(x ′ ) -ϵ(x, x ′ )] -g(x) = ∇g(x) ⊤ (f * (x) -x)(1 + β),where in the last step we substituted the expectation by f ′ (x). We notice that, as long as E x ′ ∼μx [g(x ′ ) -ϵ(x, x ′ )] -g(x) > 0, the learned direction is pointing towards a similar direction as the gradient:EQUATIONWe expand the term within the condition in the following:E x ′ ∼μx [g(x ′ ) -ϵ(x, x ′ )] -g(x) ≥ inf x ′ E x ′ ∼μx [g(x ′ ) -|ϵ(x, x ′ )|] -g(x)≥ δ y -λ 2 ∆ x 2 to determine that the following sufficient condition for ∡(∇g(x) ⊤ , f * (x)-x) to be below 90 degrees:∆ x < 2δ y λ 2 .More generally, sinceEQUATION(1 + β)λ 1 ∥f * (x) -x∥ 2 a sufficient condition for the normalized inner product to be above α isa ≤ ∇g(x) ⊤ (f * (x) -x) ∥∇g(x) ⊤ ∥∥f * (x) -x∥ ⇐ ∆ x < 2 δ y -α(1 + β)λ 1 ∥f * (x) -x∥ 2 λ 2One may also set ∥f * (x) -x∥ 2 = E x ′ ∼ μx [x ′ ]-(1-β)x 1+βin the equation above to obtain the condition∆ x < 2 δ y -αλ 1 ∥E x ′ ∼μx [x ′ ] -(1 -β)x∥ 2 λ 2 .as claimed.Proof. Set x ′ = f * (x). Taking a Taylor expansion of p around x ′ , we deduce that for every x ′′ ∈ D the following holds:p(x i ) ≤ p(x ′ ) + ∇p(x ′ ) ⊤ (x ′′ -x) + ∥H p (x ′ )∥ 2 ∥x ′′ -x∥ 2 2 2 ,with ∥H p (x ′ )∥ 2 being the Hessian of p at x ′ . Taking the expectation w.r.t. μx yieldsE x ′′ ∼μx [p(x ′′ )] ≤ p(x ′ ) + ∇p(x ′ ) ⊤ (E x ′′ ∼μx [x ′′ ] -x ′ ) + ∥H p (x ′ )∥ 2 E x ′′ ∼μx [∥x ′′ -x ′ ∥ 2 2 ] 2 or equivalently p(x ′ ) ≥ E x ′′ ∼μx [p(x ′′ )] -∇p(x ′ ) ⊤ (E x ′′ ∼μx [x ′′ ] -x ′ ) - ∥H p (x ′ )∥ 2 E x ′′ ∼μx [∥x ′′ -x ′ ∥ 2 2 ] 2If we further assume that the model is trained to minimize the matched reconstruction objective with an MSE loss, we obtain the claimed result:p(x ′ ) ≥ E x ′′ ∼μx [p(x ′′ )] - ∥H p (x ′ )∥ 2 σ 2 (M x ) 2 ,whereσ 2 (M x ) = E x ′ ∼μx [∥x ′ -E x ′′ ∼μx [x ′′ ]∥ 2 2 ]is the variance induced by the matching process.Observe that, since the variance is upper bounded by σ 2 (M x ) ≤ 4∆ x , one may provide higher likelihood samples by restricting the matching process to consider closer pairs. Further, as perhaps expected, the likelihood is higher for smoother densities.The following analysis analyses the key characteristics of the proposed method. Though our analysis does not take into account the training process with stochastic gradient descent and the potentially useful inductive biases of the neural network f θ , it provides useful insights that qualitatively align with the practical model behavior.Property 1. The learned direction follows the property gradient. We first show that by optimizing the matched reconstruction objective, the model will learn to adjust an input point by approximately following the gradient ∇g(x) of the property function.For ease of notation, we denote by M x the subset of our paired dataset with all pairs that start from point x. Theorem 4. Let f * β be the optimal solution of the matched reconstruction objective. For a λ 1 -Lipschitz and λ 2 -smooth function g, the vector v x = f * β (x) -x is a-colinear with the gradient of g:a ≤ ∇g(x) ⊤ v x ∥∇g(x)∥ 2 ∥v x ∥ 2 whenever ∆ x < 2 λ2 δ y -αλ 1 (1 + β)∥f * β (x) -x∥ 2 .The theorem reveals the role of the introduced thresholds in controlling how close we follow the property gradient: the larger the required property change δ y and the smaller the allowed input deviation ∆ x the closer v x tracks the gradient of g. We thus observe a trade-off between approximation and dataset size: selecting less conservative thresholds will lead to a larger |M| at the expense of a rougher approximation. .Therefore, setting β to a larger value can be handy for conservative design problems that prioritize distance constraints or when the property function is known to be complex. The choice of parameters ∆ x and ∆ y should be informed by the specific application. For example, in antibody design, domain experts recommend not using thresholds above a Levenshtein distance of 8, as such differences are considered biologically irrelevant. Similarly, for ∆ y , knowing that noise in binding measurements can be up to 0.3, we chose 0.5 to ensure proper matching. When faced with a new dataset of n unique data points, a good initial approach is to use the standard deviation of pairwise distances for x, and the mean or median for property y: How g(•) influences the calibration/selection of ∆ y When the g(.) is non-smooth, or we have a very few datapoints, this certainly influences the choice of ∆ y , in this case we opt for as small steps as possible since this will create many pairs which should give sense of the direction of the gradient around the sparse regions.∆ x σ d = 2 n(n -1) 1≤i<j≤n   d(x i , x j ) - 2 n(n -1) 1≤k<l≤n d(x k , x l )   2 ∆ y ≤ 2 n(n -1) 1≤i<j≤n d(x i , x j )In the Figure 8 , we provide an additional ablation study on how threshold choices affect the number of training pairs, showing that sufficiently large thresholds include all unique training points. For performance impact, please refer to the ablation study in Figure 4 . Hyper-parameter choice. For optimizing the parameters of the baselines in the toy and engineering experiments, we conducted a grid search over the learning rate ([1e-2, 1e-5]), weight decay ([1e-2, 1e-5]), number of epochs ([300, 1000, 5000]), batch size (32, 64, 128) , and number of neurons per layer ( [30, 50, 100] ).• batch size: 32, training epochs: 300 • matching thresholds: ∆ x =∈ {10, 15} mutational, edit distance, ∆ y = 0.5• baselines: For WJS, we used a 1D Conv architecture (please see A.1 in [38] ), and for Lambo, a bert-small transformer (please see C.1 in [39] ).Encoder Decoder: Analogous to the encoder, starting with the compressed representation (256 features) and progressively reconstructing the original input size (6556).Comparison to Bayesian Optimization. While both BO and PropEn aim at optimization in small sample sizes, the two frameworks solve different problems. The goal of PropEn is to generate designs, whereas in BO/AL the goal is to choose the most promising designs that should be labeled in order to imporve a predictors performance or find the best candidate, i.e. the focus is selection. In the context of optimizing designs, one would have a suite of (1) generative models, (2) property predictors and (3) BO/AL module that will do the final selection across pool of candidates. PropEn falls in (1), the category of generative models section that will contribute to the library of potential candidates. As a side note, Lambo, a method we compared to, uses a BO inspired acquisition function to guide the search for better designs, and we do compare to it (favorably), but, we must highlight the difference, **PropEn and Lambo are generative models not BO/AL methods. We hope this clarifies the differences between the two frameworks and highlights their complementarity.Additional metrics included:1. Uniqueness: number of unique designs divided by number of generated designs. 2. Novelty: number of designs proposed by the method that don't appear in the training data, divided by the number of generated designs. Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 novelty PropEn mix x2x 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00PropEn mix xy2xy 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 novelty PropEn mix x2x 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00pinwheelPropEn mix xy2xy 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 PropEn mix xy2xy 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 novelty PropEn mix x2x 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00PropEn mix xy2xy 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00pinwheelPropEn mix xy2xy 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 99.23 ± 2.43 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 novelty PropEn mix x2x 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00PropEn mix xy2xy 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 Explicit 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper introduces PropEn, a generative framework for design optimization that enhances properties of interest without requiring a discriminator, typically found in adversarial models. PropEn works by training on a "matched" dataset, where each example is paired with an improved version from its vicinity, and the generative model learns to approximate the gradient of the property function. The framework is claimed to be particularly effective in data-scarce scenarios and is validated through applications in airfoil design and protein optimization, with the latter showing superior performance in wetlab experiments.

The reviews highlight PropEn's simplicity and solid theoretical foundation as key strengths, particularly praising its applicability to real-world problems like protein optimization. However, they point out several weaknesses, including a lack of detailed implementation information, insufficient baselines in the airfoil task, and limited statistical analysis of the results. The rebuttal has provided a satisfying answer to the major concerns and I will recommend acceptance and ask the authors to incorporate the feedback.

Reviews: review: paper_summary: paper_summary: The paper presents a generative framework for property enhancement. The proposed framework consists of only a generative model, and it's missing a discriminator that is usually found in other frameworks for guided design. This is achieved by training the generative model on a "matched" dataset that consists of paired examples $(x, x')$ where $x'$ is an "enhanced" version of $x$ from its immediate proximity ("enhanced" meaning it has a higher value of the property that is sought to be maximized in the guided design process, and "proximity" depends on the problem domain).

A mathematical analysis of the proposed method shows that the learnt generative function converges to the gradient of the function computing the property and that examples sampled by the function are as likely as the training set. These analysis motivate the iterative application of the generative function to converge to a stationary point with enhanced property.

The authors claim that this framework is particularly well suited for applications with scarce data, and apply it to a toy problem and airfoil optimization, as well as protein optimization. In the latter application the method shows superior performance compared to other methods, as measured by wetlab experiments.
strength_weakness: strength_weakness: 1. The paper is missing several important implementation details. I would be especially interested in knowing the details of the training datasets and the constructed matched datasets, which I could not find anywhere in the paper or the appendix. Section B.1 in the appendix should be expanded to explain in detail what models were used. For example, line 571 states that a "ResNet with 3 blocks" was used to generate one-hot encoded AhO aligned sequences. But it's not clear to my why a 2D convolutional architecture would be used for such a problem, how exactly the data was represented, and what motivates the use of that precise model.

1. The airfoil optimization is missing a strong baseline. From the presented results I find it hard to judge whether the method indeed works well across domains. I'm not familiar with the protein optimization experiments, so I cannot judge the power of the method by the presented results, especially since important implementation details are missing (see above weakness). Since all the findings in Section 3.2 are based on very few datapoints (per optimization target) and there is a wide intra-method variability, I would like to see some statistical analysis for presented data.
questions: questions: 1. Section 2.3 lines 126-127 "As a property to optimize, we choose the log-likelihood of the data as estimated by a KDE with Gaussian kernel" – it seems that optimizing for likely examples would both optimize the property as well as the likelihood of the examples. But theorem 1 is about increasing the property value and theorem 2 is about sampling likely examples. So I would have thought it more natural to choose an illustrative example that decouples the property to be optimized from the likelihood of the examples.

1. I couldn't find the numbers reported in Table 3 in referenced papers [38, 39]. If it is a reproduction by the authors, then I would like to see that clearly stated in the main text or the table caption.

1. The assumption of a "sufficiently overparametrized model" (line 472) seems a pretty strong one to me. How realistic is this assumption in the presented settings, and what would be the consequence for Theorem 1, and PropEn applicability in general, if the assumption does not hold?

Various nits, questions, comments:
1. line 33: it should say "even for deep neural networks" (remove "with")
1. caption Figure: missing space "value.Bottom"
1. line 39: missing a word at "[6] the"
1. line 48: "have only be used provide"
1. line 57: "PropEn", not "Propen"
1. line 113: "from the its"
1. line 125: please add a reference to the "well-konwn 2d pinwheel dataset" (I don't know which dataset is meant exactly); similarly, the "8-Gaussian" dataset (line 157) is also missing a reference
1. line 154: why is the regularized variant called "mixup/mix"? I find this confusing, as there is a well-known mixup technique in ML that does not seem related (https://arxiv.org/abs/1710.09412)
1. line 165: how are metrics "uniqueness" and "novelty" defined?
1. line 185: reference for "NACA airfoils" missing
1. Section 3.1.2: how were the initial airfoil designs chosen?
1. lines 216-218 and Figure 4: when varying one threshold, what is the value of the other threshold? a 2D density plot might point a more complete picture of the measured improvement when varying both thresholds.
1. lines 219-222 and Figure 4: we can see that in a single example the $C_l/C_d$ ratio increased monotonically; but it's quite a stretch from that to claim that "all designs ... are deemed plausible" and that there is (always) a "consistent enhancement"; Figure 7 in the appendix (which is not mentioned in the main paper) does not add substantially to these claims
1. line 225: it would be nice to reference figures/tables to show that "mixup variant of PropEn may require longer training"
1. line 234: "referred", not "refereed"
1. Figure 5 (left): why do the points have different sizes?
1. Figure 8: what is the take-away from this figure?
limitations: limitations: The "NeurIPS Paper Checklist" in the appendix still needs to be filled in (currently it only shows the instructions).
paper_summary: paper_summary: This work proposes the method PropEn, which is inspired by the concept of matching techniques in econometrics. Using PropEn (specifically in scenarios with a lack of large datasets), the authors can expand the dataset, which will inherently help in design improvement, etc. To do this, they train a network to learn a mapping from an initial sample to another sample with an improved target attribute selected during the dataset curation phase. This model can further be sampled auto-regressively until it converges to a final candidate design.
strength_weakness: strength_weakness: - Additional experiments that would add substantial value to the work would be to extend the data matching method to other models, rather than a simple encoder-decoder model with a reconstruction loss. As an example, provide experiments for diffusion models where the conditioning signal is the x_0 seed (or x_i in autoregressive sampling), and the generated design is x_{i+1}, which would solidify your claims about the generalizability of the data matching method.
- Have the authors considered active learning approaches? This work is very close to active learning ideas, applied in the context of low data regimes.
- The aspect of evaluating the target properties on the neighboring samples is not clearly explained. 
- the examples shown are simple and not really huge.
questions: questions: See above in weakness
limitations: limitations: No limitations are mentioned by the authors.
paper_summary: paper_summary: The paper address design optimization, which is the process of optimizing over a "design" parameter space to optimize over one or more observable outcomes in many scientific and engineering problems. The proposed framework "PropEn" uses a three step process, first identifying a "matched dataset" that pairs every sample with another with a better outcome; this is followed by training a model that takes a sample and predicts, essentially creating an autoregressive sampler. The technique is evaluated on multiple science and engineering benchmarks.
strength_weakness: strength_weakness: * PropEn is an interesting idea for design optimization but it needs to be more thoroughly benchmarked against existing methods. For example, there is a host of methods that do Bayesian optimization, and variants thereof, that can work with as few as 10 samples (albeit in lower dimensions) which the paper does not consider. I think it sheds light into the behavior of PropEn in lower dimensions. More recent methods that use DNN based UQ estimates are able to go beyond 5-10D datasets effectively as well. 
* At a high level, matching and matched reconstruction appear to be very closely related to diffusion models. In my understanding, the diffusion process can be framed as matched reconstruction, where the property to maximize is the likelihood of the sample. Framing matched reconstruction through the lens of diffusion might not only strengthen the paper's formulation but also provide other theoretical insights. 
* Its unclear to me why neighborhoods in the design space are constructed using L2-balls, why is this the best way to identify similarity in x? How sensitive is the optimization to the size of the neighborhood? Or in another sense, a related question is -- how sensitive is matched reconstruction to the choice of sampling? for e.g. if you start with an LHC sampling (or any other variants) vs other types. 
* On small amounts of data, what kinds of regularizations keep the main model from overfitting?
questions: questions: Please see above
limitations: limitations: Yes, it appears so.
paper_summary: paper_summary: The paper develops a simple but novel deep learning framework to implicitly optimize a seed input design to iteratively improve upon a particular property $$ g(\cdot) $$, until it reaches a fixed-point. In this way, the paper develops a novel framework for design optimization and provides a theoretical derivation to showcase that this implicit setup leads to an optimal learned neural network function $$ f^*(\cdot) $$ whose iterative refinement of a seed design , approximates the direction of gradient of the property function of interest. The framework has been evaluated on three diverse design tasks and compared with several state-of-the-art design methods.
strength_weakness: strength_weakness: - Further analysis about the effect of $ \Delta_x $ and $\Delta_y  $, their selection methodology and their sensitivity in various design contexts is necessary as the matching step critically depends on these two parameters.


- The method has been demonstrated only in the context of designs that improve a single property $ g(\cdot) $. However in reality designs usually need to consider multiple properties simultaneously. This brings into question the applicability of the current method to practical problems. 


- A rigorous analysis of the "goodness" of the seed design and the effect of said goodness on the convergence speed / accuracy is necessary.
questions: questions: 1. How are $ \Delta_x $ and $\Delta_y  $ set / tuned?

2. How does property under consideration $ g(\cdot) $ influence the calibration / selection of $ \Delta_y $?

3. How can the current model be augmented to function with multiple properties that might need to be simultaneously improved?

4. How are seed designs selected for initial inputs? i.e, what is considered a minimum viable seed design to be improved upon / refined until convergence? How does the model react to bad (i.e., flawed / noisy) designs?
limitations: limitations: A limitation of the current work is its inability to incorporate more than a single property of interest. However, the authors have explicitly mentioned this limitation and the reviewer believes that despite this, the proposed method is still valuable and can serve as a good foundation for future research in design optimization.

Comments: title: Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
comment: Thank you for reconsidering your score recommendation. In addition to the comparison with Lambo, which uses BO to guide design optimization, we could include an experiment in the supplementary of the final manuscript that evaluates PropEn in BO on standard test functions like Branin-Currin or DTLZ, where we have ground truth solutions for comparison. This would involve a different problem setup, considering multiple rounds, and would require adapting PropEn to handle design selection independently. We will also include an extended discussion on this point.

Best regards
We are pleased to hear that the reviewer agrees with our rebuttal, and we sincerely appreciate all the comments and remarks provided during the discussion period. We will incorporate the additional results and clarifications into our manuscript.

Best regards
Thank you for the quick and exhaustive reply to my latest comments.

[Mixup] I think using any other name, such as "Cross xx2xx" would be clearer. Thank you for the update.

[Details on Models] Thank you for the additional information about the used models. I think these details should also be briefly mentioned in the appendix.

**[Missing comparisons to strong baselines]** Since I am not familiar with the guided design literature and airfoil design / protein optimization, I cannot get a solid impression of the performance of the proposed algorithm with the data presented in the paper. If the method were compared directly to results from the literature, that would certainly give a better grounded impression of the performance.

But I broadly agree with the rebuttal from the authors, and since no other reviewer commented on this point, I have increased my rating accordingly.
Thank you for reviewing our rebuttal and further clarifying your remarks.

**[Mixup]** We will change the name of this variant to **Cross xx2xx**, and we are open to suggestions from the reviewer. By interpolation, we refer to the behaviour observed during training in this mode. Specifically, the network learns to reconstruct (a control sample) `x` into (a treatment sample) `x'`, `x''`, `x'''`, etc. This means the network never learns a perfect reconstruction of one specific treatment, but instead generates variants in-between the two.

**[Details on Models]** The rebuttal response was limited to 6,000 characters, so we couldn't include all the details. Please note that in Section B.1, lines 556-562, we specify 30 neurons per layer, not 100 as mentioned in the reviewer's comment. To avoid confusion, please find below the networks used for the toy and engineering experiments, which we will add to the supplementary material of the paper. For WJS, we used a 1D Conv architecture (please see A.1 in [38]), and for Lambo, a bert-small transformer (please see C.1 in [39]).

Please note that the code for all of the experiments will be released. For optimizing the parameters of the baselines in the toy and engineering experiments, we conducted a grid search over the learning rate (`[1e-2, 1e-5]`), weight decay (`[1e-2, 1e-5]`), number of epochs (`[300, 1000, 5000]`), batch size (`32, 64, 128`), and number of neurons per layer (`[30, 50, 100]`). For the antibody experiments, each baseline was implemented and trained as advised by the authors of the original publications.

**Toy example architectures:**
AE(
  (criterion): MSELoss()
  (encoder): Sequential(
    (0): Linear(in_features=2, out_features=30, bias=True)
    (1): ReLU()
    (2): Linear(in_features=30, out_features=30, bias=True)
    (3): ReLU()
    (4): Linear(in_features=30, out_features=30, bias=True)
    (5): ReLU()
    (6): Linear(in_features=30, out_features=15, bias=True)
  )
  The decoder is analogous to the encoder, starts with the compressed representation (15) and progressively reconstructs the original input size (2 features).

(Discriminator): Sequential(
    (0): Linear(in_features=2, out_features=30, bias=True)
    (1): ReLU()
    (2): Linear(in_features=30, out_features=30, bias=True)
    (3): ReLU()
    (4): Linear(in_features=30, out_features=30, bias=True)
    (5): ReLU()
    (6): Linear(in_features=30, out_features=1, bias=True)
  )
)

Diffusion:

MLP(
  (time_mlp): PositionalEmbedding(
    (layer): SinusoidalEmbedding()
  )
  (input_mlp1): PositionalEmbedding(
    (layer): SinusoidalEmbedding()
  )
  (input_mlp2): PositionalEmbedding(
    (layer): SinusoidalEmbedding()
  )
  (joint_mlp): Sequential(
    (0): Linear(in_features=2, out_features=128, bias=True)
    (1): GELU(approximate='none')
    (2): Block(
      (ff): Linear(in_features=128, out_features=128, bias=True)
      (act): GELU(approximate='none')
    )
    (3): Block(
      (ff): Linear(in_features=128, out_features=128, bias=True)
      (act): GELU(approximate='none')
    )
    (4): Block(
      (ff): Linear(in_features=128, out_features=128, bias=True)
      (act): GELU(approximate='none')
    (5): Linear(in_features=128, out_features=2, bias=True))

**[Concerning all comments in the response]** Yes, we will add all modifications from the rebuttal to the manuscript. We appreciate all remarks and believe that by incorporating these modifications, we will improve our submission.

**[Missing comparison to strong baselines]** We disagree with the reviewer regarding the lack of strong baselines: 
* In the toy example, AE with guidance is the most standard baseline from the literature. As is common in all ML publications, to ensure a fair comparison, we compare the same architectures for the implicit and explicit baselines (after hyperparameter tuning). 
* In the engineering example, tiny diffusion is the most appropriate baseline to compare to given the MLP architecture used for PropEn. The diffusion model was developed to generate hundreds of points in 2D, which is exactly the same as the airfoil setup (so the reviewer's concern about dimensionality mismatch and the inappropriateness of the baseline is not grounded). We believe this is the fairest baseline we could add for comparison. 
* In the antibody design experiment, we compare against **three** SOTA methods. 

Finally, we would like to remind the reviewer that we never claim in the manuscript or the rebuttals that PropEn is the best method across the board. We believe the true advantage of PropEn is its **applicability and efficiency across many domains**, which is what we aim to confirm in our experimental evaluation. All other baselines are application-specific, requiring numerous modifications and inductive biases included in the architecture and training. This is also why it is difficult to have the same baselines across all applications, as most methods focus on a single application, while PropEn is general.
Thank you for attentive replies to my original review.

First some minor replies:

[Mixup/Mix] Yes indeed, I still find this confusing: in the cited Mixup paper the regularization works by creating convex combinations from pairs of input data and labels. In the case of PropEn the regularized variant works by incorporating a reconstruction regularizer $ℓ(f_θ(x), x)$. It is not quite clear to me how this regularized PropEn variant "interpolates between examples", as you claim in your response.

[Details on Models] Even with the additional piece of information that the ResNet is 1D I'm not quite sure about the exact models used. For example, the toy dataset model (Section B.1, lines 556-562) mentions a 2 layer MLP with 100 units per layer, and the main text mentions (Section 3.1.1, lines 162-163) that the explicit baseline uses an "auto-encoder of the same architecture as PropEn augmented with a discriminator for guidance". I would like to see more details about the discriminator and the encoder-decoder. More broadly, it would be nice to have a discussion about the models used and how they were selected. Especially since the authors choose their own baselines, care should be taken that not only the proposed method yields good results, but also that the baselines are properly optimized wrt their architecture and hyper parameters.

[Concerning all comments in the response] Thank you for the clarifications, I assume you would add them also to the paper? (e.g. the response currently says "We add the following table in the supplement", but it doesn't mention whether the submission would be updated to any of the other replies).

Then, some more comments on the main weakness raised in my initial comment:

**[Missing comparison to strong baselines]** After reading the paper and the author's reply to my review, I'm still not convinced that the method really performs better than other available methods. The main problem in my opinion is that the authors implement their own baselines and the paper lacks details that convince that the methods are applied in such a way that they really yield best possible results.

One way to mediate this would be to take strong results from existing papers and then show that PropEn performs better on the exact problems evaluated by these papers. Another way would be to show convincingly in the paper that the methods were tuned in such a way that they yield best possible results.

As for the added Airfoil baseline, I'm not convinced that these are really strong baselines, as [Diffusion-based 2DAirfoil Generation](https://github.com/tonyzyl/Diffusion-based-2DAirfoil-Generation/blob/main/models/airfoil_MLP.py) is a Github repo with a single star, and [tiny-diffusion](https://github.com/tanelp/tiny-diffusion) is "A minimal PyTorch implementation of probabilistic diffusion models for 2D datasets", where Table 1 states that the Airfoil dataset has 400 dimensions.

Overall, I appreciate the author's efforts in their reply, but I don't think that they sufficiently addressed the main weaknesses mentioned in my initial review, so I remain with my initial rating of 4 (borderline reject).
Thank you for the response and overall comments. I do appreciate the authors on actually implementing the designs into a wet lab framework, as a test for real world applicability -- a high bar for evaluating the impact of the work. A note on BO/AL, i think the distinction the rebuttal makes is subtle and it would still be useful to compare against BO/AL on known benchmarks to show the benefits, considering the broad applicability of BO/AL methods. There have been works in the past that use BO in conjunction with a generative models, this may be an informative experiment. In light of these comments, I will raise my score to 6.
Dear Reviewer,

Thank you for your valuable feedback and support. As the author-reviewer discussion period deadline approaches (Aug 13), we kindly remind you to provide any additional feedback. We hope our responses have addressed your concerns, and if so, would greatly appreciate a score update. If there are remaining issues, please let us know, and we will promptly address them.

Best regards.
Dear Reviewer,

Thank you for your valuable feedback and support. As the author-reviewer discussion period deadline approaches (Aug 13), we kindly remind you to provide any additional feedback. We hope our responses have addressed your concerns, and if so, would greatly appreciate a score update. If there are remaining issues, please let us know, and we will promptly address them.

Best regards.
Dear Reviewer,

Thank you for your valuable feedback and support. As the author-reviewer discussion period deadline approaches (Aug 13), we kindly remind you to provide any additional feedback. We hope our responses have addressed your concerns, and if so, would greatly appreciate a score update. If there are remaining issues, please let us know, and we will promptly address them.

Best regards.
Dear Reviewer,

Thank you for taking the time to provide your valuable feedback. We are glad to hear that the clarifications and additional results we provided addressed your concerns and we are grateful for your decision to raise the score.
Best regards.
I have read the comments and am satisfied with the clarifications and additional results. I will raise my score to borderline accept