Robust Guided Diffusion for Offline Black-box Optimization

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 17:43:29

Abstract: Offline black-box optimization aims to maximize a black-box function using an offline dataset of designs and their measured properties. Two main approaches have emerged: the forward approach, which learns a mapping from input to its value, thereby acting as a proxy to guide optimization, and the inverse approach, which learns a mapping from value to input for conditional generation. (a) Although proxy-free (classifier-free) diffusion shows promise in robustly modeling the inverse mapping, it lacks explicit guidance from proxies, essential for generating high-performance samples beyond the training distribution. Therefore, we propose proxy-enhanced sampling which utilizes the explicit guidance from a trained proxy to bolster proxy-free diffusion with enhanced sampling control. (b) Yet, the trained proxy is susceptible to out-of-distribution issues. To address this, we devise the module diffusion-based proxy refinement, which seamlessly integrates insights from proxy-free diffusion back into the proxy for refinement. To sum up, we propose Robust Guided Diffusion for Offline Black-box Optimization (RGD), combining proxy and proxy-free diffusion for effective conditional generation. Empirical evaluations on design-bench underscore the efficacy of RGD. Our code is here.

Main paper: 
 introduction: 
 Creating new objects to optimize specific properties is a ubiquitous challenge that spans a multitude of fields, including material science, robotic design, and genetic engineering. Traditional methods generally require interaction with a black-box function to generate new designs, a process that could be financially burdensome and potentially perilous [1, 2] . Addressing this, recent research endeavors have pivoted toward a more relevant and practical context, termed offline black-box optimization (BBO) [3, 4] . In this context, the goal is to maximize a black-box function exclusively utilizing an offline dataset of designs and their measured properties.There are two main approaches for this task: the forward approach and the reverse approach. The forward approach entails training a deep neural network (DNN), parameterized as J ϕ (•), using the offline dataset. Once trained, the DNN acts as a proxy and provides explicit gradient guidance to enhance existing designs. However, this technique is susceptible to the out-of-distribution (OOD) issue, leading to potential overestimation of unseen designs and resulting in adversarial solutions [5] .The reverse approach aims to learn a mapping from property value to input. Inputting a high value into this mapping directly yields a high-performance design. For example, MINs [6] adopts GAN [7] to model this inverse mapping, and demonstrate some success. Recent works [4] have applied proxy-free diffusion 1 [8] , parameterized by θ, to model this mapping, which proves its efficacy over other generative models. Proxy-free diffusion employs a score predictor sθ (•, •, ω). This represents a linear combination of conditional and unconditional scores, modulated by a strength parameter ω to balance condition and diversity in the sampling process. This guidance significantly diverges from proxy (classifier) diffusion that interprets scores as classifier gradients and thus generates adversarial solutions. Such a distinction grants proxy-free diffusion its inherent robustness in generating samples. Nevertheless, proxy-free diffusion, initially designed for in-distribution generation, such as synthesizing specific image categories, faces limitations in offline BBO. Particularly, it struggles to generate high-performance samples that exceed the training distribution due to the lack of explicit guidance2 . Consider, for example, the optimization of a two-dimensional variable (x d1 , x d2 ) to maximize the negative Rosenbrock function [9] : y(x d1 , x d2 ) = -(1 -x d1 ) 2 -100(x d2 -x 2 d1 ) 2 , as depicted in Figure 1 . The objective is to steer the initial points (indicated in pink) towards the high-performance region (highlighted in yellow). While proxyfree diffusion can nudge the initial points closer to this high-performance region, the generated points (depicted in blue) fail to reach the high-performance region due to its lack of explicit proxy guidance.To address this challenge, we introduce a proxy-enhanced sampling module as illustrated in Fig-ure 2(a) . It incorporates the explicit guidance from the proxy J ϕ (x) into proxy-free diffusion to enable enhanced control over the sampling process. This module hinges on the strategic optimization of the strength parameter ω to achieve a better balance between condition and diversity, per reverse diffusion step. This incorporation not only preserves the inherent robustness of proxy-free diffusion but also leverages the explicit proxy guidance, thereby enhancing the overall conditional generation efficacy. As illustrated in Figure 1 , samples (depicted in red) generated via proxy-enhanced sampling are more effectively guided towards, and often reach, the high-performance area (in yellow). Yet, the trained proxy is susceptible to out-ofdistribution (OOD) issues. To address this, we devise a module diffusion-based proxy refinement as detailed in Figure 2(b) . This module seamlessly integrates insights from proxy-free diffusion into the proxy J ϕ (x) for refinement. Specifically, we generate a diffusion distribution p θ (y| x) on adversarial samples x, using the associated probability flow ODE 3 . This distribution is derived independently of a proxy, thereby exhibiting greater robustness than the proxy distribution on adversarial samples. Subsequently, we calculate the Kullback-Leibler divergence between the two distributions on adversarial samples, and use this divergence minimization as a regularization strategy to fortify the proxy's robustness and reliability.To sum up, we propose Robust Guided Diffusion for Offline Black-box Optimization (RGD), a novel framework that combines the advantages of proxy (explicit guidance) and proxy-free diffusion (robustness) for effective conditional generation. Our contributions are three-fold:• We propose a proxy-enhanced sampling module which incorporates proxy guidance into proxy-free diffusion to enable enhanced sampling control.• We further develop diffusion-based proxy refinement which integrates insights from proxy-free diffusion back into the proxy for refinement.• RGD delivers state-of-the-art performance on various design-bench tasks, emphasizing its efficacy.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
offline black-box optimization: 
 Offline black-box optimization (BBO) aims to maximize a black-box function with an offline dataset.Imagine a design space as X = R d , where d is the design dimension. The offline BBO [3] is:EQUATIONIn this equation, J(•) is the unknown objective function, and x ∈ X is a possible design. In this context, there is an offline dataset, D, that consists of pairs of designs and their measured properties.Specifically, each x denotes a particular design, like the size of a robot, while y indicates its related metric, such as its speed.A common approach gradient ascent fits a proxy distribution p ϕ (y|x) = N (J ϕ (x), σ ϕ (x)) to the offline dataset where ϕ denote the proxy parameters:arg min ϕ E (x,y)∈D [-log p ϕ (y|x)]. = arg min ϕ E (x,y)∈D log( √ 2πσ ϕ (x)) + (y -J ϕ (x)) 2 2σ 2 ϕ (x).(EQUATIONFor the sake of consistency with terminology used in the forthcoming subsection on guided diffusion, we will refer to p ϕ (•|•) as the proxy distribution and J ϕ (•) as the proxy. Subsequently, this approach performs gradient ascent with J ϕ (x), leading to high-performance designs x * :EQUATIONconverging to x M after M steps. However, this method suffers from the out-of-distribution issue where the proxy predicts values that are notably higher than the actual values.Diffusion models, a type of latent variable models, progressively introduce Gaussian noise to data in the forward process, while the reverse process aims to iteratively remove this noise through a learned score estimator. In this work, we utilize continuous time diffusion models governed by a stochastic differential equation (SDE), as presented in [10] . The forward SDE is formulated as:EQUATIONwhere f (•, t) : R d → R d represents the drift coefficient, g(•) : R → R denotes the diffusion coefficient and w is the standard Wiener process. This SDE transforms data distribution into noise distribution. The reverse SDE is:EQUATIONwith ∇ x log p(x) representing the score of the marginal distribution at time t, and w symbolizing the reverse Wiener process. The score function ∇ x log p(x) is estimated using a time-dependent neural network s θ (x t , t), enabling us to transform noise into samples. For simplicity, we will use s θ (x t ), implicitly including the time dependency t.Guided diffusion seeks to produce samples with specific desirable attributes, falling into two categories: proxy diffusion [11] and proxy-free diffusion [8] . While these were initially termed classifier diffusion and classifier-free diffusion in classification tasks, we have renamed them to proxy diffu- EQUATIONWithin this context, the strength parameter ω specifies the generation's adherence to the condition y, which is set to the maximum value y max in the offline dataset following [4] . Optimization of ω balances the condition and diversity. Lower ω values increase sample diversity at the expense of conformity to y, and higher values do the opposite.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in Appendix G.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we present our method RGD, melding the strengths of proxy and proxy-free diffu- Derive the score sθ (x t+1 , y, ω) from Eq. (6).12: Update x t+1 to x t (ω) using ω as per Eq. (7) . Optimize ω to ω following Eq. (8).Finalize the update of x t with ω via Eq. ( 9). Optimization of ω. Directly updating the design x t with proxy gradient suffers from the OOD issue and determining a proper condition y necessitates the manual adjustment of multiple hyperparameters [6] . Thus, we propose to introduce proxy guidance by only optimizing the strength parameter ω within sθ (x t , y, ω) in Eq. (6) . As discussed in Section 2.3, the parameter ω balances the condition and diversity, and an optimized ω could achieve a better balance in the sampling process, leading to more effective generation.Enhanced Sampling. With the score function, the update of a noisy sample x t+1 is computed as:EQUATIONwhere the solver is the second-order Heun solver [12] , chosen for its enhanced accuracy through a predictor-corrector method. A proxy is then trained to predict the property of noise x t at time step t, denoted as J ϕ (x t , t). By maximizing J ϕ (x t (ω), t) with respect to ω, we can incorporate the explicit proxy guidance into proxy-free diffusion to enable enhanced sampling control in the balance between condition and diversity. This maximization process is:EQUATION)where η denotes the learning rate. We leverage the automatic differentiation capabilities of Py-Torch [13] to efficiently compute the above derivatives within the context of the solver's operation.The optimized ω then updates the noisy sample x t+1 through:EQUATIONThis process iteratively denoises x t , utilizing it in successive steps to progressively approach x 0 , which represents the final high-scoring design x * .Proxy Training. Notably, J ϕ (x t , t) can be directly derived from the proxy J ϕ (x), the mean of the proxy distribution p ϕ (•|x) in Eq. ( 2). This distribution is trained exclusively at the initial time step t = 0, eliminating the need for training across time steps. To achieve this derivation, we reverse the diffusion from x t back to x 0 using the formula:EQUATIONwhere s θ (x t ) is the estimated unconditional score at time step t, and σ(t) 2 and µ(t) are the variance and mean functions of the perturbation kernel at time t, as detailed in equations (32) (33) in [10] .Consequently, we expressEQUATION)This formulation allows for the optimization of the strength parameter ω via Eq. ( 8). For simplicity, we will refer to J ϕ (•) in subsequent discussions.In the proxy-enhanced sampling module, the proxy J ϕ (•) is employed to update the parameter ω to enable enhanced control. However, J ϕ (•) may still be prone to the OOD issue, especially on adversarial samples [5] . To address this, we refine the proxy by using insights from proxy-free diffusion. The procedure of this module is specified in Algorithm 1, Lines 3-7.Diffusion Distribution. Adversarial samples are identified by gradient ascent on the proxy as per Eq. ( 3) to form the distribution q(x). Consequently, these samples are vulnerable to the proxy distribution. Conversely, the proxy-free diffusion, which functions without depending on a proxy, inherently offers greater resilience against these samples, thus producing a more robust distribution.For an adversarial sample x ∼ q(x), we compute p θ ( x), p θ ( x|y) via the probability flow ODE, and p(y) through Gaussian kernel-density estimation. The diffusion distribution regarding y is derived as:EQUATIONwhich demonstrates inherent robustness over the proxy distribution p ϕ (y| x). Yet, directly applying diffusion distribution to design optimization by gradient ascent is computationally intensive and potentially unstable due to the demands of reversing ODEs and scoring steps.Proxy Refinement. We opt for a more feasible approach: refine the proxy distribution p ϕ (y| x) = N (J ϕ ( x), σ ϕ ( x)) by minimizing its distance to the diffusion distribution p θ (y| x). The distance is quantified by the Kullback-Leibler (KL) divergence:EQUATIONWe avoid the parameterization trick for minimizing this divergence as it necessitates backpropagation through p θ (y| x), which is prohibitively expensive. Instead, for the sample x, the gradient of the KL divergence D(p ϕ ||p θ ) with respect to the proxy parameters ϕ is computed as:EQUATION)Complete derivations are in Appendix A. The KL divergence then acts as regularization in our loss L:EQUATIONwhere D is the training dataset and α is a hyperparameter. We propose to optimize α based on the validation loss via bi-level optimization as detailed in Appendix B.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We provide a link to our source code in the abstract and thoroughly describe our experimental settings in Section 4.3.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we conduct comprehensive experiments to evaluate our method's performance.Tasks. Our experiments encompass a variety of tasks, split into continuous and discrete categories.The continuous category includes four tasks: (1) Superconductor (SuperC) 4 : The objective here is to engineer a superconductor composed of 86 continuous elements. The goal is to enhance the critical temperature using 17, 010 design samples. This task is based on the dataset from [1] .(2) Ant Morphology (Ant): In this task, the focus is on developing a quadrupedal ant robot, comprising 60 continuous parts, to augment its crawling velocity. It uses 10, 004 design instances from the dataset in [3, 14] .(3) D'Kitty Morphology (D'Kitty): Similar to Ant Morphology, this task involves the design of a quadrupedal D'Kitty robot with 56 components, aiming to improve its crawling speed with 10, 004 designs, as described in [3, 15] . (4) Rosenbrock (Rosen): The aim of this task is to optimize a 60-dimension continuous vector to maximize the Rosenbrock black-box function. It uses 50000 designs from the low-scoring part [9] .For the discrete category, we explore three tasks: (1) TF Bind 8 (TF8): The goal is to identify an 8-unit DNA sequence that maximizes binding activity. This task uses 32, 898 designs and is detailed in [16] .(2) TF Bind 10 (TF10): Similar to TF8, but with a 10-unit DNA sequence and a larger pool of 50, 000 samples, as described in [16] .(3) Neural Architecture Search (NAS): This task focuses on discovering the optimal neural network architecture to improve test accuracy on the CIFAR-10 dataset, using 1, 771 designs [17] .Evaluation. In this study, we utilize the oracle evaluation from design-bench [3] . Adhering to this established protocol, we analyze the top 128 promising designs from each method. The evaluation metric employed is the 100 th percentile normalized ground-truth score, calculated using the formula y n = y-ymin ymax-ymin , where y min and y max signify the lowest and highest scores respectively in the comprehensive, yet unobserved, dataset. In addition to these scores, we provide an overview of each method's effectiveness through the mean and median rankings across all evaluated tasks. Notably, the best design discovered in the offline dataset, designated as D(best), is also included for reference.For further details on the 50 th percentile (median) scores, please refer to Appendix C.Our approach is evaluated against two primary groups of baseline methods: forward and inverse approaches. Forward approaches enhance existing designs through gradient ascent. This includes: (i)Grad: utilizes simple gradient ascent on current designs for new creations; (ii) ROMA [18] : implements smoothness regularization on proxies; (iii) COMs [5] : applies regularization to assign lower scores to adversarial designs; (iv) NEMO [19] : bridges the gap between proxy and actual functions using normalized maximum likelihood; (v) BDI [20] : utilizes both forward and inverse mappings to transfer knowledge from offline datasets to the designs; (vi) IOM [21] : ensures consistency between representations of training datasets and optimized designs.Inverse approaches focus on learning a mapping from a design's property value back to its input.High property values are input into this inverse mapping to yield enhanced designs. This includes: (i)CbAS [22] : CbAS employs a VAE model to implicitly implement the inverse mapping. It gradually tunes its distribution toward higher scores by raising the scoring threshold. This process can be interpreted as incrementally increasing the conditional score within the inverse mapping framework.(ii) Autofocused CbAS (Auto.CbAS) [23] : adopts importance sampling for retraining a regression model based on CbAS. (iii) MIN [6] : maps scores to designs via a GAN model and explore this mapping for optimal designs. (iv) BONET [24] : introduces an autoregressive model for sampling high-scoring designs. (v) DDOM [4] : utilizes proxy-free diffusion to model the inverse mapping.Traditional methods as detailed in [3] are also considered: (i) CMA-ES [25] : modifies the covariance matrix to progressively shift the distribution towards optimal designs; (ii) BO-qEI [26] : implements Bayesian optimization to maximize the proxy and utilizes the quasi-Expected-Improvement acquisition function for design suggestion, labeling designs using the proxy; (iii) REINFORCE [27] :enhances the input space distribution using the learned proxy model.In alignment with the experimental protocols established in [3, 20] , we have tailored our training methodologies for all approaches, except where specified otherwise. For methods such as BO-qEI, CMA-ES, REINFORCE, CbAS, and Auto.CbAS that do not utilize gradient ascent, we base our approach on the findings reported in [3] . We adopted T = 1000 diffusion sampling steps, set the condition y to y max , and initial strength ω as 2 in line with [4] . To ensure reliability and consistency in our comparative analysis, each experimental setting was replicated across 8 independent runs, unless stated otherwise, with the presentation of both mean values and standard errors. These experiments were conducted using a NVIDIA GeForce V100 GPU. We've detailed the computational overhead of our approach in Appendix D to provide a comprehensive view of its practicality. In Tables 1 and 2 , we showcase our experimental results for both continuous and discrete tasks.To clearly differentiate among the various approaches, distinct lines separate traditional, forward, and inverse approaches within the tables For every task, algorithms performing within a standard deviation of the highest score are emphasized by bolding following [5] .We make the following observations. (1) As highlighted in In this section, we present a series of ablation studies to scrutinize the individual contributions of distinct components in our methodology. We employ our proposed approach as a benchmark and methodically exclude key modules, such as the proxy-enhanced sampling and diffusion-based proxy refinement, to assess their influence on performance. These variants are denoted as w/o proxy-e and w/o diffusion-b r. Additionally, we explore the strategy of directly performing gradient ascent on the diffusion intermediate state, referred to as direct grad update. The results from these ablation experiments are detailed in Table 3 .Our analysis reveals that omitting either module results in a decrease in performance, thereby affirming the importance of each component. The w/o diffusion-b r variant generally surpasses w/o proxy-e, highlighting the utility of the proxy-enhanced sampling even with a basic proxy setup. Conversely, direct grad update tends to produce subpar results across tasks, likely attributable to the proxy's limitations in handling out-of-distribution samples, leading to suboptimal design optimizations. To further dive into the proxy-enhanced sampling module, we visualize the strength ratio ω/ω 0 -where ω 0 represents the initial strength-across diffusion steps t. This analysis is depicted in Figure 3 for two specific tasks: Ant and TF10. We observe a pattern of initial decrease followed by an increase in ω across both tasks. This pattern can be interpreted as follows: The decrease in ω facilitates the generation of a more diverse set of samples, enhancing exploratory capabilities. Subsequently, the increase in ω signifies a shift towards integrating high-performance features into the sample generation. Within this context, conditioning on the maximum y is not aimed at achieving the dataset's maximum but at enriching samples with high-scoring attributes. Overall, this adjustment of ω effectively balances between generating novel solutions and honing in on high-quality ones.In addition, we visualize the proxy distribution alongside the diffusion distribution for a sample x from the Ant task in Figure 4 , to substantiate the efficacy of diffusion-based proxy refinement. The proxy distribution significantly overestimates the ground truth, whereas the diffusion distribution closely aligns with it, demonstrating the robustness of diffusion distribution. For a more quantitative analysis, we compute the expectation of both distributions and compare them with the ground truth. The mean of the diffusion distribution is calculated as E p θ (y| x) [y] = E p ϕ (y| x) p θ (y| x) p ϕ (y| x) y . The MSE loss for the proxy distribution is 2.88, while for the diffusion distribution, it is 0.13 on the Ant task. Additionally, we evaluate this on the TFB10 task, where the MSE loss for the proxy distribution is 323.63 compared to 0.82 for the diffusion distribution. These results further corroborate the effectiveness of our proposed module. Furthermore, we (1) investigate the impact of replacing our trained proxy model with alternative approaches, specifically ROMA and COMs, (2) analyze the performance with an optimized condition y and(3) explore a simple annealing approach of ω. For a comprehensive discussion on these, readers are referred to Appendix E.This section investigates the sensitivity of RGD to various hyperparameters. Specifically, we analyze the effects of (1) the number of diffusion sampling steps T , (2) the condition y, and (3) the learning rate η of the proxy-enhanced sampling. These parameters are evaluated on two tasks: the continuous Ant task and the discrete TFB10 task. For a detailed discussion, see Appendix F. 
related work: 
 Offline black-box optimization. A recent surge in research has presented two predominant approaches for offline BBO. The forward approach deploys a DNN to fit the offline dataset, subsequently utilizing gradient ascent to enhance existing designs. Typically, these techniques, including COMs [5] ,ROMA [18] , NEMO [19] , BDI [20, 28] , IOM [29] and Parallel-mentoring [30] , are designed to embed prior knowledge within the surrogate model to alleviate the OOD issue. The reverse approach [6, 31] is dedicated to learning a mapping from property values back to inputs. Feeding a high value into this inverse mapping directly produces a design of elevated performance. Additionally, methods in [22, 23] progressively tailor a generative model towards the optimized design via a proxy function and BONET [24] introduces an autoregressive model trained on fixed-length trajectories to sample high-scoring designs. Recent investigations [4] have underscored the superiority of diffusion models in delineating the inverse mapping. However, research on specialized guided diffusion for offline BBO remains limited. This paper addresses this research gap.Contemporary research in guided diffusion primarily concentrates on enhancing the efficiency of its sampling process. [32] propose a method for distilling a classifier-free guided diffusion model into a more efficient single model that necessitates fewer steps in sampling. [33] introduce an operator splitting method to expedite classifier guidance by separating the update process into two key functions: the diffusion function and the conditioning function. Additionally, [34] presents an efficient and universal guidance mechanism that utilizes a readily available proxy to enable diffusion guidance across time steps. In this work, we explore the application of guided diffusion in offline BBO, with the goal of creating tailored algorithms to efficiently generate high-performance designs. 
conclusion: 
 In conclusion, we propose Robust Guided Diffusion for Offline Black-box Optimization (RGD). The proxy-enhanced sampling module adeptly integrates proxy guidance to enable enhanced sampling control, while the diffusion-based proxy refinement module leverages proxy-free diffusion insights for proxy improvement. Empirical evaluations on design-bench have showcased RGD's outstanding performance, further validated by ablation studies on the contributions of these novel components.We discuss the broader impact and limitation in Appendix G.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We detail our setting in Section 4.3 and also discuss hyperparameter sensitivity in Appendix F.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have discussed these in Section 4.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss both potential positive and negative impacts in Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release any datasets nor pre-trained models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have duly credited all utilized assets and adhered to their respective licenses and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: This section provides a derivation of the gradient of the KL divergence. Let's consider the KL divergence term, defined as:EQUATIONThe gradient with respect to the parameters ϕ is computed as follows:EQUATIONWe propose adjusting α based on the validation loss, establishing a bi-level optimization framework:EQUATIONEQUATIONWithin this context, D v represents the validation dataset sampled from the offline dataset. The inner optimization task, which seeks the optimal ϕ * (α), is efficiently approximated via gradient descent.While the main text of our paper focuses on the 100 th percentile scores, this section provides an in-depth analysis of the 50 th percentile scores. These median scores, previously explored in [3] , serve as an additional metric to assess the performance of our RGD method. The outcomes for continuous tasks are detailed in Table 5 , and those pertaining to discrete tasks, along with their respective ranking statistics, are outlined in Table 6 . An examination of Table 6 highlights the notable success of the RGD approach, as it achieves the top rank in this evaluation. This finding underscores the method's robustness and effectiveness. In this section, we analyze the computational overhead of our method. RGD consists of two core components: proxy-enhanced sampling (proxy-e sampling) and diffusion-based proxy refinement (diffusion-b proxy r). Additionally, RGD employs a trained proxy and a proxy-free diffusion model, whose computational demands are denoted as proxy training and diffusion training, respectively.Table 4 indicates that experiments can be completed within approximately one hour, demonstrating efficiency. The diffusion-based proxy refinement module is the primary contributor to the computational overhead, primarily due to the usage of a probability flow ODE for sample likelihood computation.However, as this is a one-time process for refining the proxy, its high computational cost is offset by its non-recurring nature. In contexts such as robotics or bio-chemical research, the most time-intensive part of the production cycle is usually the evaluation of the unknown objective function. Therefore, the time differences between methods for deriving high-performance designs are less critical in actual production environments, highlighting RGD's practicality where optimization performance are prioritized over computational speed. This aligns with recent literature (A.3 ComputationalComplexity in [35] and A.7.5. Computational Cost in [28] ) indicating that in black-box optimization scenarios, computational time is relatively minor compared to the time and resources dedicated to experimental validation phases. In this section, we extend our exploration to include alternative proxy refinement schemes, namely ROMA and COMs, to compare against our diffusion-based proxy refinement module. The objective is to assess the relative effectiveness of these schemes in the context of the Ant and TFB10 tasks.The comparative results are presented in Table 7 . Our investigation reveals that proxies refined through ROMA and COMs exhibit performance akin to the vanilla proxy and they fall short of achieving the enhancements seen with our diffusion-based proxy refinement. We hypothesize that the diffusion-based proxy refinement, by aligning closely with the characteristics of the diffusion model, provides a more relevant and impactful signal. This alignment improves the proxy's ability to enhance the sampling process more effectively. Additionally, we contrast our approach, which adjusts the strength parameter ω, with the MIN method that focuses on identifying an optimal condition y. The MIN strategy entails optimizing a Lagrangian objective with respect to y, a process that requires manual tuning of four hyperparameters. We adopt their methodology to determine optimal conditions y and incorporate these into the proxy-free diffusion for tasks Ant and TF10. The normalized scores for Ant and TF10 are 0.950 ± 0.017 and 0.660 ± 0.027, respectively. The outcomes fall short of those achieved by our method as detailed in Table 7 . This discrepancy likely stems from the complexity involved in optimizing y, whereas dynamically adjusting ω proves to be a more efficient strategy for enhancing sampling control.Last but not least, we explore simple annealing approaches for ω. Specifically, we test two annealing scenarios considering the default ω as 2.0: (1) a decrease from 4.0 to 0.0, and (2) an increase from 0.0 to 4.0, both modulated by a cosine function over the time step (t). We apply these strategies to the Ant Morphology and TF Bind 10 tasks, and the results are as follows: The empirical results across both strategies illustrate their inferior performance compared to our approach, thereby demonstrating the efficacy of our proposed method.RGD's performance is assessed under different settings of T , y, and η. We experiment with T values of 500, 750, 1000, 1250, and 1500, with the default being T = 1000. For the condition ratio y/y max , we test values of 0.5, 1.0, 1.5, 2.0, and 2.5, considering 1.0 as the default. Similarly, for the learning rate η, we explore values of 2.5e -3 , 5.0e -3 , 0.01, 0.02, and 0.04, with the default set to η = 0.01.Results are normalized by comparing them with the performance obtained at default values.As depicted in Figures 5, 6 , and 7, RGD demonstrates considerable resilience to hyperparameter variations. The Ant task, in particular, exhibits a more marked sensitivity, with a gradual enhancement in performance as these hyperparameters are varied. The underlying reasons for this trend include:(1) An increase in the number of diffusion steps (T ) enhances the overall quality of the generated samples. This improvement, in conjunction with more effective guidance from the trained proxy, leads to better results. (2) Elevating the condition (y) enables the diffusion model to extend its reach beyond the existing dataset, paving the way for superior design solutions. However, selecting an optimal y can be challenging and may, as observed in the TFB10 task, sometimes lead to suboptimal results. (3) A higher learning rate (η) integrates an enhanced guidance signal from the trained proxy, contributing to improved performances.In contrast, the discrete nature of the TFB10 task seems to endow it with a certain robustness to variations in these hyperparameters, highlighting a distinct behavioral pattern in response to hyperparameter adjustments. Broader impact. Our research has the potential to significantly accelerate advancements in fields such as new material development, biomedical innovation, and robotics technology. These advancements could lead to breakthroughs with substantial positive societal impacts. However, we recognize that, like any powerful tool, there are inherent risks associated with the misuse of this technology. One concerning possibility is the exploitation of our optimization techniques to design objects or entities for malicious purposes, including the creation of more efficient weaponry or harmful biological agents.Given these potential risks, it is imperative to enforce strict safeguards and regulatory measures, especially in areas where the misuse of technology could lead to significant ethical and societal harm.The responsible application and governance of such technologies are crucial to ensuring that they serve to benefit society as a whole.Limitation. We recognize that the benchmarks utilized in our study may not fully capture the complexities of more advanced applications, such as protein drug design, primarily due to our current limitations in accessing wet-lab experimental setups. Moving forward, we aim to mitigate this limitation by fostering partnerships with domain experts, which will enable us to apply our method to more challenging and diverse problems. This direction not only promises to validate the efficacy of our approach in more complex scenarios but also aligns with our commitment to pushing the boundaries of what our technology can achieve.NeurIPS Paper ChecklistJustification: To ensure reliability and consistency in our comparative analysis, each experimental setting was replicated across 8 independent runs, unless stated otherwise, with the presentation of both mean values and standard errors.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: We preserve anonymity and conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Classifier-free diffusion is for classification and adapted to proxy-free diffusion to generalize to regression.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS2024). Do not distribute.Proxy-free diffusion cannot be interpreted as a proxy and thus does not provide explicit guidance[8].Ordinary Differential EquationPreviously, the task oracle exhibited inconsistencies, producing varying outputs for identical inputs. This issue has now been rectified by the development team. 


Decision: The paper proposes RGD, a method combining proxy and proxy-free diffusion approaches for offline black-box optimization. Initial reviews were mixed. Reviewers praised the comprehensive experiments and intuitive ideas but raised concerns about technical novelty, comparisons with recent methods, and potential overfitting risks.

In their rebuttal, the authors addressed many concerns by providing additional comparisons, clarifying technical details, and conducting further experiments. Most reviewers found these responses satisfactory, with some increasing their scores, and I did take the authors' concern about reviewer AXdG into consideration. However the overall support after rebuttal still fall below the typical acceptance range. Thus I regret to recommend a reject for this submission, and encourage the authors to resubmitted a revised version to the next top conference.

Reviews: review: paper_summary: paper_summary: The paper introduces a new method, named RGD, for Offline Black-box Optimization (BBO). RGD incorporates an improved proxy to guide the previous proxy-free method (i.e. DDOM[4]). Key technical innovations includes (1) improving the robustness of the proxy function against adversarial samples by consistency regularization with the diffusion process; (2) dynamic per-sample reweighting between proxy-guided and proxy-free sampling. Compared to previous approaches, RGD demonstrates superior performance on Design-Bench [3].
strength_weakness: strength_weakness: The reviewer would prefer some clarifications on the method and the experiments

i) Algorithm 1, Line 4, how to identify the adversarial examples? From Line 187-188, it looks like gradient ascent is utilized to find the x that maximize y, it is unclear to the reviewer that how to determine if the obtained x is an adversarial example

ii) Algorithm 1, Line 7, refine the proxy function via eq 15. It would be best if the author could provide further details on how to optimize eq (15), e.g. number of validation and adversarial samples, number of iterations for the bi-level optimization discussed in Appendix B.

iii) Algorithm 1 Line 13, optimizing \omega. Again, it would be best if the author could provide extra info on how to optimize \omega. From Algorithm 1, it looks like \omega is time dependent and optimized for each time step. How many training iterations are required for each time step. The reviewer also wonder if the obtained \omega are dramatically different between different time steps. 

iv) From Line 257-258, it looks like the baselines shown in Table 1 & 2 were re-implemented. If this is the case, the authors are encouraged to include more implementation details, e.g. the model architecture for the score function, etc. This could help follow-up works to reproduce the reported results. The reviewer also wonders if the source code will be made public.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Limitations have been discussed in the appendix
paper_summary: paper_summary: The paper introduces a robust guided diffusion framework for offline black-box optimization, combining proxy and proxy-free diffusion for conditional generation. Key improvements include proxy-enhanced sampling and diffusion-based proxy refinement to address out-of-distribution issues. Experiments on the Design-Bench benchmark show the method outperforms existing techniques, validated by ablation studies.
strength_weakness: strength_weakness: - The paper lacks comparison with relevant approaches like ICT [1] and TRI-mentoring [2]. Despite referencing the latter in the related work section, it’s overlooked in the results.
- It is unclear why the results without proxy-enhanced sampling still achieve competitive outcomes, surpassing the dataset y_max. This contradicts the claims in lines 40-46. Where does the out-of-distribution (OOD) problem arise then? What is the distribution of the generated 128 candidates with and without the sampling? 
- The BDI reported results are significantly lower than in the original paper, especially for the ANT and TFBIND8 tasks. This also seems to be the case for BONET results. Did the authors change the evaluation setup?


[1]: Importance-aware Co-teaching for Offline Model-based Optimization, https://arxiv.org/abs/2309.11600

[2]: Parallel-mentoring for Offline Model-based Optimization, https://arxiv.org/abs/2309.11592
questions: questions: - How are the initial 128 designs selected? Do you generate N designs and then use the proxy to select 128?
- Are the diffusion parameters like T the same for DDOM?
- Can you show, for a task, how the values (proxy/oracle) of the generated designs progress throughout the diffusion steps?
- How were the discrete tasks handled? Were logits used, or were they kept discrete?
- Previous methods typically evaluate on the Hopper task; why was it removed and Rosenbrock is added instead?
limitations: limitations: The authors address the limitations and potential negative impacts in their paper.
paper_summary: paper_summary: The paper proposes a framework called Robust Guided Diffusion for the problem of Offline Black-box Optimization. The key idea is to formulate the solution as conditional generation of high-performance designs using a diffusion model which has explicit guidance from a proxy (surrogate) model. This proxy model is also refined/updated via a proxy-free diffusion procedure. Experimental analysis is shown on multiple tasks from design-bench benchmark.
strength_weakness: strength_weakness: - One major premise (repeated multiple times in the paper) in the paper is that proxy guidance conditional generation is more robust than updating the design with standard gradient ascent on the proxy. However, it is not immediately clear why this should be true and the justification for this key point is somewhat limited. If true, this will be much bigger insight going beyond black-box optimization. If it is only about the exploration/exploitation balance driven by w, we could also make standard gradient have this property by optimizing a upper/lower confidence bound on the objective. Please describe why this is the case either via some empirical experiment or theoretical insight. Also, in equation 11, we might evaluate the proxy far away from the training data depending on the values of s_\theta(x_t), \sigma(t), \mu(t).

- The related work coverage and corresponding experimental analysis of the paper can be improved. This problem has seen an extensive body of work recently. Please see the references below and discuss/compare them appropriately. Some of them are included in references but not compared in the experiments ([1], [2], [3]):

- [1] Yuan, Ye, et al. "Importance-aware co-teaching for offline model-based optimization." Advances in Neural Information Processing Systems 36 (2023).
- [2] Kim, Minsu, et al. "Bootstrapped training of score-conditioned generator for offline design of biological sequences." Advances in Neural Information Processing Systems 36 (2023).
- [3] Nguyen, Tung, Sudhanshu Agrawal, and Aditya Grover. "ExPT: Synthetic pretraining for few-shot experimental design." Advances in Neural Information Processing Systems 36 (2023).
- [4] Chemingui, Yassine, et al. "Offline model-based optimization via policy-guided gradient search." *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 38. No. 10. 2024.
- [5] Yao, Michael S., et al. "Generative Adversarial Bayesian Optimization for Surrogate Objectives." arXiv preprint arXiv:2402.06532 (2024).
questions: questions: Some of the tasks in design-bench benchmark has errors which makes them not so informative for evaluation. For example, the offline dataset in superconductor task has multiple copies of the same inputs but with different outputs. As a result, the random forest oracle which is fit on this offline data is not reliable. It is mentioned that "This issue has now been rectified by the development team." How is it fixed?
limitations: limitations: Please see weaknesses section.
paper_summary: paper_summary: In this paper,  the authors proposed to combine both classifier guidance and classifier-free guidance for offline black-box optimization.  In addition, the authors propose a Proxy Refinement procedure by minimizing KL divergence between the Proxy distribution and diffusion distribution regarding $y$.
strength_weakness: strength_weakness: 1.  **Technical contribution seems to be incremental**

Employing diffusion models for offline black-box optimization is not new.  The technical contribution of this paper seems to be incremental. The draft extends the paper "Diffusion Models for Black-Box Optimization" [1]. However,   detailed discussions about the relationship between the proposed method and the paper [1] are missing.

[1] Siddarth Krishnamoorthy, Satvik Mashkaria, and Aditya Grover. "Diffusion Models for Black-Box Optimization." ICML 2023. 


2.  **Part of the technical details are not clear.**

(a) In Equation (12),  the concrete computation procedure of $p_\theta (\hat{\boldsymbol{x}} | y)$ and $p_\theta (\hat{\boldsymbol{x}})$  via diffusion model is not clear. 

(b) The derivation of Equation (10) is not given.   It seems that Equation (10) is from the forward pass of the diffusion model.  However,  the forward pass  (Eq.32-32 in [10]) is regarding the distribution.  And the concrete  $\boldsymbol{x} _ t $ is constructed via the backward pass with  $s_\theta(\boldsymbol{x}_k,k)$ for $k \in T,\cdots, t+1$. 
 In addition,  how to choose $\mu(t)$ and $\sigma(t)$ in Equation (10)  is not clear.  

3.  **The additional proxy training, sample refinement procedure and proxy refinement procedure  increase the computation cost**

The additional proxy training, sample refinement procedure and proxy refinement procedure increase the computation cost. However, the time comparison with baselines is missing. 

4.  **The additional proxy training, sample refinement procedure and proxy refinement procedure bring many additional hyperparameters, which may overfit the offline BBO task**

In the offline BBO tasks,  the offline dataset is provided.  The evaluation is the black-box function value at the generated query at one time. 
The long-term convergence properties and exploration/exploitation balance are not considered.  As a result, there are risks that overfit the evaluation metric for the offline tasks.  The paper Introduces lots of additional hyperparameters, which increases the overfitting risks.
questions: questions: 1. Please provide more discussion about the technical details in Weakness 2.


2.  What is the computation time of the proposed method? Please provide time comparisons with baselines.


3. Please provide explanations about the overfiting issue.
limitations: limitations: Additional computation cost and overfitting risk may be additional limitations besides the limitations discussed.
paper_summary: paper_summary: The paper proposed RGD, a novel method for integrating classifier guidance into classifier-free guidance diffusion models for solving offline MBO problems. Experiment results and ablation studies validate that the method outperforms state-of-the-art baselines and each proposed component is resonable.
strength_weakness: strength_weakness: - For the diffusion-based proxy refinement part, it seems that there are several estimations to compute the distance between $p_{\phi}(y\vert \hat{x})$ and $p_{\theta}(y\vert \hat{x})$. Furthermore, it incurs additional hyperparameter $\alpha$, which should be carefully tuned.
questions: questions: - For the diffusion-based proxy refinement part, have the authors tried other approaches for estimating $p(y)$? I wonder if the choice of estimation affects the performance significantly. 

- To compute the regularization loss term in Eq (15), we need to collect samples from the adversarial distribution. I cannot find the detailed procedure for collecting adversarial samples ($M$, $\eta$, ...). Could authors elaborate more on that part?
limitations: limitations: There are a few minor comments on the manuscript.

- For figure 2, it seems that $\tilde{s}(x_T, y, \omega)$ should be written as $\tilde{s}(x_T, y, \hat{\omega})$. Furthermore, at first, it makes me confusion that RGD conducts classifer-guidance. However, that misleading part has been resolved after reading the manuscript.

Comments: title: Urgent: please respond to rebuttal
Thank you for your continued engagement
Clarification on Eq.(10).
Additional question about Eq.(10) in the draft
Thanks for your support
Clarification on Model Robustness and Key Components
Reply to Authors' rebuttal
Appeal for Reassessment Based on Review Feedback
Thanks for your prompt response and support.
Response
Thank you for your detailed feedback
Response to rebuttal
Please reply to the rebuttal.
Please reply to the rebuttal.
reply to the rebuttal
reply to the rebuttal
Thanks for your prompt feedback and continued support.
comment: Dear Reviewer,

As the discussion period is nearing its conclusion, we kindly ask you to engage in the discussion and provide notes on any concerns that have not yet been addressed, along with the reasons why.

AC.
Thank you for your continued engagement and for raising these important points regarding the derivation of Eq.(10).

The reference to Tweedie's formula in our previous rebuttal was intended as a high-level citation, acknowledging the foundational idea of recovering samples from imperfect data. It was not directly used to derive Eq.(10), but rather to highlight the conceptual framework. To enhance understanding, we will include a citation of the seminar work DDPM [r3], where similar concepts are more explicitly detailed.

Regarding the approximation $\epsilon \approx -\sigma(t) \cdot s_{\boldsymbol{\theta}}(x_t)$, the essence of diffusion models is to learn a model that can predict the noise vector, thereby enabling the denoising of samples from pure noise to realistic data. In our specific case, we train the diffusion model $s_{\boldsymbol{\theta}}(x_t)$ to predict $-\frac{\epsilon}{\sigma(t)}$. This is operationalized by optimizing the loss function mentioned in Line 112 of our sde.py, where $\epsilon$ is the target, $\sigma(t)$ is the std, and $a$ is $s_{\boldsymbol{\theta}}(x_t)$. In essence, starting with $x_0$, we sample a noise vector $\epsilon$ as the target, add the correponding noise to $x_0$ to generate $x_t$, and then aim to train $s_{\boldsymbol{\theta}}(x_t)$ to more closely approximate $-\frac{\epsilon}{\sigma(t)}$.

~~Have we adequately addressed your concerns?~~. Based on the latest feedback, it appears that the derivation is now clear, so we have removed this question sentence. Thank you again for your thoughtful engagement with our work.
Thanks for the authors' detailed response. 

From the authors' new response,  I find that the derivation of Eq.(10) does not come from Tweedie’s formula suggested by the authors in the previous rebuttal.  The Tweedie’s formula calculates the posterior expectation $X_0$ given $X_t = x_t$, 
i.e.  $\mathbb{E} [X_0 | X_t = x_t]$, instead of a concrete sample $x_0$. 

The derivation of Eq.(10) is actually through reparametrization of $X_t$ and relies on an approximation of the Gaussian noise $\epsilon \approx -\sigma(t) \cdot s_\theta(x_t)$. 

I got the derivation.   Thanks again for the authors' detailed response.   My concern is well addressed.
Additionally, it's worth noting that Eq.(10) in our submission aligns closely with Eq.(15) from the seminal work DDPM[r3]. In DDPM, they present the equation $x_0 \approx \frac{x_t - \sqrt{1-\alpha_t} \epsilon_{\boldsymbol{\theta}}(x_t)}{\sqrt{\alpha_t}}$, which is derived in a discrete setting. 


[r3] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851.
Thank you for your insightful question regarding Eq.(10). We will now derive Eq.(10) step by step to address any concerns.

We follow Eq (33) from [r1] where $p(x_t|x_0) = N(x_t; \mu(t) x_0, \sigma^2(t) I)$. Given this, we can sample $x_t$ from $x_0$ using: $x_t = \mu(t) x_0 + \epsilon \sigma(t)$. To recover $x_0$ from $x_t$, we need to know $\epsilon$, which approximates as $\epsilon \approx -\sigma(t) \cdot s_{\boldsymbol{\theta}}(x_t)$. Using this approximation, we derive $x_0 = \frac{x_t - \epsilon \sigma(t)}{\mu(t)} \approx \frac{x_t + s_{\boldsymbol{\theta}}(x_t) \cdot  \sigma^2(t) }{\mu(t)}$.

This approach originates from [r1], and we utilize the implementation framework detailed in another seminal work [r2]. Specifically, our code, available at https://anonymous.4open.science/r/RGD-7DBB/lib/sdes.py , implements this process as follows:
- Line 24 implements $\mu(t)$
- Line 27 implements $\sigma^2(t)$
- Line 37 describes the sampling process: $x_t = \mu(t) x_0 + \epsilon \sigma(t)$
- Line 112 optimizes: $\epsilon \approx -\sigma(t) \cdot s_{\boldsymbol{\theta}}(x_t)$, where $\epsilon$ is the target, $\sigma(t)$ is the std, and $a$ is $s_{\boldsymbol{\theta}}(x_t)$.

Apologies for any confusion. Considering that most readers and reviewers come from an offline MBO background, these advanced concepts of diffusion models can be challenging. We will add a statement following Line 178, "For a more detailed derivation, please refer to the Appendix." In the Appendix, we will include the discussion outlined above.

Have we adequately addressed your concerns? 

References:
- [r1] Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. 2021.
- [r2] Huang C W, Lim J H, Courville A C. A variational perspective on diffusion-based generative models and score matching. Advances in Neural Information Processing Systems, 2021, 34: 22863-22876.
Dear authors,

I find an issue when I try to derive Eq.(10) using Tweedie’s formula suggested by the author.

Note that $X_0 \sim p_0(X_0)$ and $X_t = X_0 + \mathcal{N}(0, \sigma(t)^2 \boldsymbol{I} )$ from forward pass of diffusion model (corresponding VE SDE Eq.(31) in [r1]) .  

From the Tweedie’s formula, we achieve the following Equation: 
$$ \mathbb{E}[X_0|X_t=x_t] =  x_t + \sigma(t)^2 \nabla _{x_t} \log p_t (x_t)  $$

It can then be approximated using the trained score function $s_\theta(x_t)$ as
$$ \mathbb{E}[X_0|X_t=x_t] \approx  x_t + \sigma(t)^2   s_\theta(x_t) $$

However,  the Eq.(10) in the submission drops the conditional expectation w.r.t. $X_0$ and directly obtain 
$$ x_0  \approx  x_t + \sigma(t)^2   s_\theta(x_t) $$

Did I miss anything to achieve Eq.(10)? Could the authors explain more about how to derive Eq.(10)?


[r1] Song et al. Score-Based Generative Modeling through Stochastic Differential Equations.  2021.
Thank you for your supportive feedback. We are pleased that our explanations have helped clarify the aspects of our work. We will ensure that these discussions are incorporated into the revised manuscript.
Thanks for the authors' detailed response.  

I now have a better understanding of how each proposed component works and how they relate to one another.   I am not certain about the experiments, but I think the proposed method makes sense.  Therefore,  I decided to increase my score.
We appreciate the reviewer’s detailed feedback and are glad that most of the concerns have been addressed. We are uncertain about your use of the term 'overfitting.' We interpret this as the possibility that our proposed designs might overfit the trained proxy. Please let us know if we have misunderstood your concern.

**The first key component** enhancing our model's robustness is the use of proxy-free diffusion guidance. In this framework, the denoising step cannot be interpreted as a gradient-based adversarial attack, since it does not rely on proxy gradients to directly modify the input design. This concept is thoroughly discussed in the seminar work [A], particularly in Equation (6).

However, proxy-free diffusion guidance alone is insufficient, as it lacks direct guidance from the proxy, limiting its ability to extrapolate effectively. This limitation is illustrated in Figure 1 of our manuscript, titled 'Motivation for Explicit Proxy Guidance.'  To address this, we introduce explicit proxy guidance as **the second key component**. This component aims to direct the sampling process toward high-property regions. Direct application of proxy gradients to the input space would result in out-of-distribution samples, potentially leading to what might be perceived as overfitting. Therefore, we apply proxy gradients to the scalar strength parameter $\omega$, which modulates both condition and diversity. This approach of optimizing scalar parameters rather than the design itself is explored in the ICML 2023 paper [B], specifically in Section 4.5 on Adaptive-$\gamma$. It demonstrates how supervision signals from the proxy can effectively update scalar hyperparameters, thereby enhancing robustness without directly modifying the input design.

In essence, our model combines (1) the robustness afforded by proxy-free diffusion—which does not rely on proxy gradients—with (2) the targeted guidance from a trained proxy, which influences only the scalar strength parameter $\omega$ to mitigate overfitting risk. **These two key components** form our **proxy-enhanced sampling module**, which is crucial for enhancing the model’s resilience against overfitting and are instrumental in achieving superior outcomes.

Additionally, in our **diffusion-based proxy refinement module**, we propose utilizing diffusion-derived distribution priors to refine the proxy, an approach not previously explored. This is also **an important component**. While prior work, such as COMs and ROMA, employed simple intuitive priors like conservative estimation and smoothness to refine proxy, our method leverages diffusion-derived distributions, providing a more relevant and impactful signal for refining the proxy, which has been proved to be more effective.  The detailed comparisons with COMs and ROMA are discussed in detail in Appendix E, 'Further Ablation Studies'.

Have we adequately addressed your concerns? We truly appreciate your comprehensive feedback.

    [A] Ho J, Salimans T. Classifier-free diffusion guidance[J]. arXiv preprint arXiv:2207.12598, 2022.
    [B] C. Chen. et al. Bidirectional Learning for Offline Model-based Biological Sequence Design. ICML 2023
Dear Reviewer,

As the discussion period is nearing its conclusion, we kindly ask you to engage in the discussion and provide notes on any concerns that have not yet been addressed, along with the reasons why.

Thank you for your attention to this matter.

AC.
Thank you for your prompt feedback.

As previously mentioned, we acknowledge the TRI-mentoring method and have discussed it in the related work section. We initially did not include a direct comparison with TRI-mentoring because (1) it employs an advanced ensemble approach, distinctly different from our methodology with a single proxy, and (2) our analysis already included a comprehensive set of 14 baselines. Following your insightful recommendation, we have now incorporated a comparative analysis with TRI-mentoring.

We also recognize and regret the inadvertent omission of the citation for the BDI results from TRI-mentoring, which has been duly corrected. We wish to clarify that **our omission was limited to the citation of results; we did not manipulate the baseline results, and thus this should not be considered as selective reporting.**  We appreciate the emphasis on transparency and fully agree with its importance. We understand that **peer review should focus on the scientific content and contributions of a manuscript**. While unintentional oversights are unfortunate, they have been rectified and should not overshadow the substantive scientific evaluations of the work. 

We hope that the changes implemented demonstrate our commitment to transparency and scientific rigor.
Thanks for the authors' detailed clarification and responses.  Most of my concerns have been addressed.  

I still have some concerns regarding the overfitting risk.   I acknowledge the different focuses of offline black-box optimization and online black-box optimization and explain why the authors preclude the exploration-exploitation considerations for long-term behavior.  However,  I am not sure what the key component among the several proposed ones is that makes the whole model more robust against the overfitting.  In addition, what is the key component to achieve a better score beyond the maximum in the dataset and outperforms baselines?
Thank you for your follow-up and revisions, and for admitting the oversight. However, transparency is most effective when it’s proactive, not reactive. The initial omission of TRI-mentoring while using their BDI results raise serious concerns about the credibility of your work. Even if corrected later, selective reporting undermines trust and cannot be overlooked. Therefore, I will maintain my decision to reject the paper. I strongly urge you to prioritize transparency in future submissions.
Given your recognition that we have addressed most of the initial concerns:
> While you've addressed most of my concerns, 

we were surprised by the decision to downgrade our submission from 'borderline reject' to 'reject,' particularly since the remaining issues were, in our assessment, relatively minor.  We are confident that the revisions we have implemented thoroughly address the issues raised. In light of these efforts, we kindly request a reevaluation of our paper.
Thank you for your willingness to consider our arguments and for your constructive suggestions. We will ensure to add these points and enhance the discussion of related work in the main paper.
Thanks for the response. I am not fully convinced with the robustness of proxy-free diffusion guidance argument but I don't want to nitpick now and would be happy to increase the score towards acceptance. Please add these points in the main paper along with proper discussion of the related work.
Thank you for your detailed feedback, which provides constructive insights for refining our paper. We will incorporate these points into the revised version.

- In the paper [A], searching the term 'adversarial' reveals critical arguments supporting the robustness of proxy-free diffusion guidance over proxy diffusion guidance. The text notes,  

> Furthermore, because classifier guidance mixes a score estimate with a classifier gradient during sampling, classifier-guided diffusion sampling can be interpreted as attempting to confuse an image classifier with a gradient-based adversarial attack.

 as mentioned in the second paragraph of the introduction. Additionally, descriptions related to Equation (6) state, 

> Eq. (6) has no classifier gradient present, so taking a step in the $\epsilon$ direction cannot be interpreted as a gradient-based adversarial attack on an image classifier.

These points suggest that proxy-free diffusion guidance surpasses proxy diffusion guidance in robustness. 

Acknowledging that generating images for a specific class is simpler than guiding towards the optima of a function, the latter scenario necessitates more robust guidance mechanisms. This is where proxy-free guidance excels, as directly using proxy diffusion guidance may lead to out-of-distribution issues, whereas proxy-free diffusion guidance provides more robust guidance. We will incorporate these discussions in Line 121 of our manuscript where we introduce guided diffusion.

- Thank you for your comment. Yes, in our ablation study, we kept all elements except the proxy refinement part constant across the three methods. We will emphasize this in Line 482 of our manuscript where we perform further ablation studies.

Have we adequately addressed your concerns?
Thanks for taking the time to respond to my questions. Please see points related to your response below:

- Regarding `This significant insight aligns with findings from [A], which argues that proxy-free diffusion guidance surpasses proxy diffusion guidance in robustness.  [A] Ho, J. and Salimans, T. Classifier-free diffusion guidance.`

  Unless I am missing something, the main premise in [A] is to "attain a trade-off between sample quality and diversity **similar** to that obtained using classifier guidance". I am not sure how this reference paper [A] findings conveys that ` proxy-free diffusion guidance surpasses proxy diffusion guidance in robustness.`  Moreover, guiding diffusion models to generate images for a specific class is a relatively easier/different problem that guiding towards the optima of a function. In the former, we are interested  in sampling any point from a class conditional distribution whereas in the latter, we explicitly want to find the optima (rare sample in the distribution). 

- Thanks for the ablation comparing diffusion-based proxy refinement module with COMs/ROMa strategy. Since the ablation show final evaluation performance on the tasks, this ablation is useful only if the proxy refinement part of the three methods is changed and everything else is kept the same. For example, this requires fixing either gradient ascent or proxy enhanced sampling for search/generating candidates for evaluation. Is this the case? 

- Thanks for including the discussion about new related work and fixing the error in superconductor task.
Thank you for your detailed feedback. Regarding the use of BDI results from the TRI-mentoring paper, we have now included proper citations and incorporated TRI-mentoring into our benchmarks for a comprehensive comparison. We initially viewed TRI-mentoring primarily as an advanced ensemble method, which seemed distinct from our model’s approach, and thus only include it in the related work section. However, recognizing the importance of clarity and completeness, we have addressed this in our revised submission.

As for the reduced network architecture, we followed the specifications outlined in the published NeurIPS2023 paper TRI-mentoring and use the reported results in this NeurIPS2023 paper. Our approach was consistent with the standard three-layer MLP used across all methods in this comparison, ensuring a fair and uniform basis for evaluation.  We hope these clarifications address your concerns. Could you please let us know if there are any further issues or suggestions you have?
Thank you for your detailed responses. While you've addressed most of my concerns, two key issues are troubling: (1) The use of BDI results from the TRI-mentoring paper without proper citation, coupled with the initial omission of TRI-mentoring from the benchmarks, and (2) The results from a reduced network architecture, which leads to an unfair comparison that may inadvertently favor your method.
These issues affect the credibility of the work, prompting me to revise my score to a rejection. I hope these concerns can be addressed in any future submissions to ensure a fair and accurate presentation.
Dear Reviewer, 

Please reply to the rebuttal. 

AC.
Dear Reviewer, 

Please reply to the rebuttal. 

AC.
Dear Reviewer, 

Please reply to the rebuttal. 

AC.
Dear Reviewer, 

Please reply to the rebuttal. 

AC.
Thank you for your prompt feedback and continued support.

To further assess the robustness of our method against the choice of p(y), in addition to the Ant task, we have conducted experiments on the TFB8 and TFB10 tasks. The performance was 0.974 with the original method and 0.975 with GMM on the TFB8 task, and it was 0.694 with the original method and 0.692 with GMM on the TFB10 task. These consistent results reinforce the robustness of our estimator choice.

We will ensure that all discussions are meticulously incorporated into the final manuscript, as suggested.
Thank you for your detailed feedback and I keep my positive rating. There are some minor comments.

> For the diffusion-based proxy refinement part, have the authors tried other approaches for estimating p(y)

As the authors say, ablation studies on the choice of estimating p(y) enhance the robustness of the proposed method. I also recommend that authors conduct the ablation study across at least two tasks for the claim.