Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization

Authors: taeyoung  yun
sujin  yun
jaewoo  lee
jinkyoo  park
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:46:07

Abstract: Optimizing complex and high-dimensional black-box functions is ubiquitous in science and engineering fields. Unfortunately, the online evaluation of these functions is restricted due to time and safety constraints in most cases. In offline model-based optimization (MBO), we aim to find a design that maximizes the target function using only a pre-existing offline dataset. While prior methods consider forward or inverse approaches to address the problem, these approaches are limited by conservatism and the difficulty of learning highly multi-modal mappings. Recently, there has been an emerging paradigm of learning to improve solutions with synthetic trajectories constructed from the offline dataset. In this paper, we introduce a novel conditional generative modeling approach to produce trajectories toward high-scoring regions. First, we construct synthetic trajectories toward high-scoring regions using the dataset while injecting locality bias for consistent improvement directions. Then, we train a conditional diffusion model to generate trajectories conditioned on their scores. Lastly, we sample multiple trajectories from the trained model with guidance to explore high-scoring regions beyond the dataset and select high-fidelity designs among generated trajectories with the proxy function. Extensive experiment results demonstrate that our method outperforms competitive baselines on Design-Bench and its practical variants. The code is publicly available in \url{https://github.com/dbsxodud-11/GTG}.

Main paper: 
 introduction: 
 Optimizing complex and high-dimensional black-box functions is ubiquitous in science and engineering fields, including biological sequence design [1] , materials discovery [2] , and mechanical design [3, 4] . Traditional methods like Bayesian optimization have been developed to solve the problem by iteratively querying a black-box function. However, the online evaluation of the black-box function is restricted in most real-world situations due to time and safety constraints.Fortunately, we often have access to a previously collected offline dataset. This problem setting is referred to as offline model-based optimization (MBO), and our objective is to find a design that maximizes a target function using solely an offline dataset [5] . As no online evaluation is available, a key challenge of MBO is the out-of-distribution (OOD) issue arising from limited data coverage. Suppose we train a proxy that predicts function values given input designs and naively apply a gradient-based optimizer based on the proxy to identify the optimal design. It would fall into sub-optimal results due to inaccurate predictions of the proxy in unseen regions.To mitigate this issue, forward approaches mostly consider training a robust surrogate model against adversarial optimization of inputs and applying gradient-based maximization. Trabucco et al. [6] train a proxy with the regularization term to prevent overestimation on OOD designs. Fu and Levine [7] leverage normalized maximum likelihood estimator to handle uncertainty on unseen regions.38th Conference on Neural Information Processing Systems (NeurIPS 2024).There are also several works that focus on fine-tuning the proxy for robustness on unexplored regions [8, 9, 10] . However, the generalization of the proxy outside of the dataset still remains challenging.On the other hand, inverse approaches learn a mapping from function values to the input domain. Then, they generate high-scoring designs by querying the learned mapping with a high score. Prior approaches utilize expressive generative models to learn a mapping, such as variational autoencoders [11, 12] , generative adversarial nets [13] , autoregressive models [14] or diffusion models [15] . While these methods show promising results, they still suffer from the difficulty of learning highly unsmooth distributions and utilizing valuable information about the landscape of the black-box function.Recently, a new perspective has emerged on tackling the MBO by learning to improve solutions with synthetic trajectories constructed from the dataset [16, 17] . These methods aim to generate a sequence of designs toward high-scoring regions. It seems more promising than learning an inverse mapping that generates only a single design, as we can utilize information from sequences of designs that can help better understand the landscape of the target function. However, there is still room for improvement in this perspective. First, prior approaches construct trajectories with simple heuristics, which may lead to generating trajectories with inconsistent directions of improvement. Furthermore, the sequential nature of autoregressive models may lead to error accumulation during sampling [18] .To this end, we propose a novel conditional generative modeling approach to solve the MBO problem. Unlike prior inverse approaches, which generate a single design, we generate a sequence of designs toward high-scoring regions with guided sampling. Our method consists of four stages. First, we construct trajectories from the dataset while incorporating locality bias to distill the knowledge of the landscape of the target function into the generator. Then, we train a conditional diffusion model that generates the whole trajectory at once to bypass error accumulation and an auxiliary proxy. After training, we sample multiple trajectories conditioned on context data points and high score values. Finally, we select high-fidelity designs among generated trajectories by filtering with the proxy.We empirically demonstrate that our method achieves superior performance on Design-Bench, a well-known benchmark for MBO with a variety of real-world tasks. Furthermore, we explore more practical settings, such as sparse or noisy datasets, verifying the generalizability of our method. 
problem setup: 
 In offline model-based optimization (MBO), we aim to find a design x that maximizes the target black-box function f . Unlike the typical black-box optimization setting, we can only access an offline dataset D, and online evaluations are unavailable. The problem setup can be described as follows:find x * = arg max x∈R d f (x) s.t only an offline dataset D = {(x i , y i )} N i=1 is given (1)where x is a decision variable and y = f (x) is a target property we want to maximize.Diffusion probabilistic models [19, 20] are a class of generative models that approximate the true distribution q 0 with a parametrized model of the form: p θ (x 0 ) = p θ (x 0:T )dx 1:T , where x 0 ∼ q 0 and x 1 , • • • , x T are latents with the same dimensionality. The joint distribution p θ (x 0:T ) is called the reverse process, defined as a Markov chain starting from standard Gaussian p T (x T ) = N (0, I):EQUATIONwhere p θ (x t-1 |x t ) is parametrized Gaussian transition from timestep t to t -1.We define a forward process, which is also fixed as a Markov chain that adds Gaussian noise to the data with the variance schedule Training diffusion models can be performed by maximizing the variational lower bound on the log-likelihood E q0 [log p θ (x 0 )], which is equivalent to minimizing the following loss:EQUATIONEQUATIONwhere ϵ θ (x t , t) is the parameterization suggested by [20] ,µ θ (x t , t) = 1 √ αt x t -βt √ 1-ᾱt ϵ θ (x t , t) .For modeling conditional distribution q(x|y), we can use classifier-free guidance [21] . In classifierfree guidance, we train both a conditional ϵ θ (x t , y, t) and unconditional model ϵ θ (x t , t) with the following loss:EQUATIONFor sampling, we start from Gaussian noise x T and refine x t into x t-1 with the perturbed noise from the learned model ϵ θ at each diffusion timestep t:EQUATIONwhere ω is a scalar value that controls the guidance scale. 
methodology: 
 In this section, we introduce GTG, Guided Trajectory Generation, a conditional generative modeling approach for solving MBO problem by learning to improve solutions using the offline dataset. We first construct trajectories towards high-scoring regions while incorporating locality bias for consistent improvement directions. Then, we train the conditional diffusion model to generate trajectories and a proxy model. Finally, we sample multiple trajectories using the diffusion model with guided sampling and filter high-fidelity designs with the proxy. Figure 1 shows the overview of the proposed method.We construct a set of trajectories D traj from the offline dataset D to gather information on learning to improve designs. In this paper, each trajectory τ ∈ D traj is a set of H input-output pairs and can be represented as a two-dimensional array:EQUATIONWhile prior works construct trajectories via sorting heuristics or sampling from high-scoring regions, we focus on constructing trajectories that give us more valuable information for learning to improve designs towards higher scores. To achieve this, we develop a novel method to construct trajectories based on two desiderata.First, the trajectory should be towards high-scoring regions while containing information on the landscape of the target black-box function. Second, the trajectories should be diverse and not converge to a single data point with the highest score of the dataset, as our objective is to discover high-scoring designs beyond the offline dataset by generalizing the knowledge of learning to improve solutions. for h = 1, • • • , H -1 do 5:Find K nearest neighbors of x h whose score is higher thanmax{y 1 , • • • , y h } -ϵ 6:Sample (x h+1 , y h+1 ) from the K neighbors and update τ ←-τ ∪ {(x h+1 , y h+1 )}end for 8:Update D traj ←-D traj ∪ {τ } 9: end for To this end, we introduce a novel strategy to construct trajectories from the dataset. We illustrate the procedure in Algorithm 1. For each trajectory, we first sample an initial data point (x 1 , y 1 ) from a relatively low score distribution, pth percentile of D. After initialization, we employ a local search strategy to select the next data point to generate a smooth trajectory toward high-scoring regions that contain the information on the landscape of the target function. Specifically, for each round h, we find K nearest neighbors of x h whose score is higher than max{y 1 , • • • , y h } -ϵ, where ϵ is a small, non-negative real number. By allowing small perturbations using ϵ, we can prevent generated trajectories from converging a single maximum of the offline dataset. Then, we sample (x h , y h ) from the K neighbors randomly to generate diverse trajectories. We repeat the procedure until constructing a trajectory of length H. By moving towards high-scoring regions while staying in a local region, we can effectively guide the generator to learn diverse and consistent paths for improving solutions.Note that identifying K nearest neighbors of a data point whose values are above a certain threshold does not require substantial computational time compared to training and evaluation. We explain in more detail our trajectory construction procedure in Appendix B.1Given our trajectory dataset D traj , our objective is to learn the conditional distribution of trajectories towards high-scoring regions. We choose diffusion models, which have a powerful capability to learn the distribution of complex and high-dimensional data [22, 23] , to generate trajectories. Our objective is then transformed from searching high-scoring designs to maximizing the conditional likelihood of trajectories, which can be achieved by minimizing the loss in Equation ( 5):EQUATIONwhere y(τ ) = H h=1 y h is the sum of scores in the trajectory τ . By training a diffusion model to generate a sequence of designs instead of a single design, we can efficiently distill the knowledge of the complex landscape of the target function into the diffusion model.In addition, we also train a forward proxy f ϕ using the dataset D. We can use the proxy to filter high-scoring designs from the trajectories generated by the trained diffusion model.After training, we sample trajectories with guided sampling. We use classifier-free guidance to generate trajectories. To be specific, we sample τ from the diffusion model using Equation (6) , where y * (τ ) is the target conditioning value. Following prior works [13, 16] , we assume that we know the maximum score y * and set y * (τ ) = α • (Hy * ), where α controls the exploration level of the generated trajectories. We discuss the role of α in more detail in the subsequent section.To fully utilize the expressive power of diffusion models, we introduce an additional strategy, context conditioning, during the sampling. We generate trajectory with diffusion model while inpainting the C context data points of the trajectory with τ ctx , which is a subtrajectory sampled from D traj . By conditioning trajectories in different contexts, we can effectively explore diverse high-scoring regions. Compute ε(t) using p θ (τ |y * (τ )) by Equation ( 6) ▷ Classifier-free Guidance Compute τ (t-1) using τ ctx , ε(t) and τ (t) by Equation ( 9) ▷ Context-Conditioning7:end for 8:Update Formally, for each denoising timestep t, we refine τ (t) into τ (t-1) with the following procedure:D cand ←-D cand ∪ {x C+1 , • • • , x H } from τ (= τ (0)EQUATION)where m is the mask for the first C context data points and ε(t) is computed from the Equation (6).After generating trajectories, we introduce filtering to select candidates for evaluation. In other words, we select top-Q samples in terms of the predicted score from the proxy. By filtering with the proxy, we can exploit the knowledge from the dataset to search high-scoring designs [13, 14, 24] . 
experimental evaluation: 
 In this section, we present the results of our experiments on various tasks. First, we analyze our method in a toy 2D experiment. Then, we present the results on the Design-Bench and its practical variants to verify the effectiveness of the method. We also conduct extensive analyses on various aspects to deepen our understanding of the proposed method.We first evaluate our method using a toy setting to analyze each component of our method thoroughly. We choose Branin, a synthetic 2D function with three distinct global maxima. Figure 2 shows the contour plot of the Branin function. The analytical form of the Branin function is as follows: where a = 1, b = 5.1 4π 2 , c = 5 π , s = 10, t = 1 8π and the range of 15] . For the MBO setting, we uniformly sample 5000 data points and remove the top 10% percentile to make the task more challenging. We construct trajectories with a length of 64 using our trajectory construction strategy and other strategies suggested by prior methods, BONET [16] and PGS [17] .EQUATION(x 1 , x 2 ) is [-5, 10] × [0,Figure 2a shows the trajectories generated from prior methods. As shown in the figure, we find that constructed trajectories show uncorrelated movements, which makes the model hard to capture knowledge on the landscape of the target black-box function. Unlike prior methods, our method constructs trajectories that improve the solution with the local movements, as illustrated in Figure 2b . Such trajectories help the diffusion model to learn how to improve solutions efficiently. We also find that trajectories do not converge into a single data point and toward diverse high-scoring regions via random sampling from K neighbors and perturbations from ϵ.Figure 2c shows the trajectories generated by the trained diffusion model with context conditioning and classifier-free guidance. As shown in the figure, GTG can generalize the knowledge on improving solutions to find diverse high-scoring solutions. GTG achieves a maximum score of -0.490 ± 0.070, which is near-optimal compared to the global optimum (-0.398) and far beyond the maximum value of the dataset (-6.031). Please refer to Appendix A.1 for more details of the toy experiment.In this section, we present the experiment results of our method on Design-Bench tasks [5] . We conduct experiments on two discrete tasks and three continuous tasks. For each task, we have an offline dataset from an unknown oracle function. We present the detailed task description below.TFBind8 and TFBind10 [1] . We aim to find a DNA sequence of lengths 8 and 10 with maximum binding affinity with a particular transcription factor.Superconductor [2] . We aim to design a chemical formula, represented by an 86-dimensional vector, for a superconducting material with a high critical temperature.Ant and D'Kitty Morphology [4, 25] . We aim to optimize the morphological structure of two simulated robots. The morphology parameters include size, orientation, and the location of the limbs. Ant has 60 continuous parameters, and D'Kitty has 56 continuous parameters.For baselines, we prepare four main categories to solve MBO problems. First, we compare our method with traditional methods widely used in online black-box optimization settings, such as BO-qEI [26] , CMA-ES [27] , REINFORCE [28] , and Gradient Ascent. The second category comprises recently proposed forward approaches, including COMs [6] , NEMO [7] , RoMA [8] , BDI [24] , and ICT [9] . The third category encompasses inverse approaches, and we select CbAS [11] , MINs [13] , and DDOM [15] as our baselines. Finally, we also compare with baselines which construct synthetic trajectories and generalize the knowledge of learning to improve solutions, BONET [16] and PGS [17] .For evaluation, we follow the protocol of prior works. We identify Q = 128 designs selected by the algorithm and report a normalized score of 100 th percentile design. For all algorithms, we run experiments over 8 different seeds and report mean and standard errors.To evaluate our method, we construct trajectories of length H = 64 and train a conditional diffusion model for each task. After training, we sample N = 128 trajectories conditioning on C = 32 context data points and setting α = 0.8 across all tasks. Finally, we filter top-128 candidates among generated designs with the predicted score from the proxy for evaluation.As shown in the Table 1 , GTG achieves an average rank of 1.6, the best among all competitive baselines. It performs best on two tasks and is runner-up on three tasks, demonstrating superior performance across different tasks. We observe that GTG generally surpasses forward approaches, which struggle to fall into OOD designs, especially in high-dimensional settings. We also observe that our method outperforms inverse approaches, including DDOM, which also utilizes a diffusion model. It demonstrates that generating trajectories towards high-scoring regions can be more effective than generating a single design, as we can distill the knowledge of the landscape of the target function into the generator. Our method achieves higher performance compared to BONET, which also generates trajectories. It indicates that our novel trajectory construction strategy effectively guides the diffusion model to explore diverse paths toward high-scoring regions.In this section, we present experiment results in a more practical setting of Design-Bench tasks. While Design-Bench assumes a large, unbiased offline dataset containing thousands of data points for the training model, such a setting is impractical in most cases. Therefore, we prepare two additional practical settings, sparse and noisy datasets, to verify the robustness of our method in such extreme cases. In a sparse setting, we only provide x% of the original dataset for training. For the noisy setting, we add x% of standard Gaussian noise to the normalized score values. We choose recent papers published after 2022, BDI, ICT, DDOM, and BONET for primary baselines. Please refer to Appendix A.2 for detailed experiment settings and Appendix D.5 for results with more baselines. Table 2 shows the results of our method and recent baselines in sparse datasets. The table shows that our method mostly outperforms other baselines even in sparse datasets, demonstrating the superiority of exploiting knowledge of the target function by constructing diverse trajectories from the dataset. Table 3 reports the experiment results on the noisy settings. We find that even with 50% of noise, our method can find relatively high-scoring designs, demonstrating its robustness in practical settings.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide detailed experiment settings in the Appendix. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
additional analysis: 
 In this section, we carefully analyze the effectiveness of each component in our method.Ablation on trajectory construction. We propose a novel trajectory construction strategy by incorporating locality bias. To verify the effectiveness of the strategy, we compare our strategy with prior approaches, SORT-SAMPLE and Top-p Percentile, suggested by BONET and PGS, respectively. Table 4 shows that our strategy outperforms prior strategies across various tasks. We conduct additional analysis on trajectory construction strategies in Appendix D.1.Ablation on sampling procedure. We analyze the effectiveness of strategies we introduced during the sampling procedure, namely context conditioning (CC), classified-free guidance (CF), and filtering (F). Across various tasks, it is evident that all components are crucial for improving performance as demonstrated in Table 5 . We conduct further analysis on sampling strategies in Appendix D.2.Hyperparameter sensitivity. We also conduct experiments on the effect of various hyperparameters we introduced in this paper. We first train a conditional diffusion model with various lengths (H). As shown in Figure 3a , increasing H leads to achieving higher performance. We also conduct experiments by varying the number of contexts (C) and the exploration level (α). Figure 3b shows that C = 32 achieves superior performance while conditioning with too many contexts degrades performance. Finally, Figure 3b shows a strong correlation between α and the score, demonstrating the effectiveness of guided sampling. We conduct further analysis on hyperparameters in Appendix D.2.Varying evaluation budget. We provide experiment results with a small number of evaluation budgets (Q). As shown in Figure 4 , we generally outperform most baselines even with a relatively low evaluation budget.Assumption on optimal score. We assume that the optimal value y * is known, following prior works [13, 16] . We conduct experiments by relaxing the aforementioned assumption in Appendix D.2 and find that GTG can achieve comparable performance even without knowing y * .Effect of unsupervised pretraining. It might be beneficial to pretrain the diffusion model when we have a large-scale unlabeled dataset and a few designs of labeled points [29] . To this end, we discuss the effectiveness of pretraining diffusion models with unlabeled datasets in Appendix D.3.Time complexity of sampling procedure. We also conduct analysis on the time complexity of the sampling procedure of our method in Appendix D.4. Experiment results demonstrate that we can decrease the number of denoising timesteps even one-tenth with minimal loss in performance. Recently, a new perspective on solving the MBO problem has emerged by learning to improve solutions from synthetic trajectories and generalizing the knowledge to find designs beyond the dataset [16, 17] . BONET [16] trains an autoregressive model to generate optimal trajectories conditioned on a low regret budget. PGS [17] trains RL policy with trajectories consisting of high-scoring designs to roll out optimal trajectories. MATCH-OPT [30] also constructs monotonic trajectories and matches the gradient field with the proxy. GTG falls under this category but adopts a unique approach to constructing trajectories with local search and utilizing diffusion models to enhance performance. 
generative models for decision making: 
 Generative models have emerged as a powerful tool for decision-making problems, including bandit problems [31] , reinforcement learning [18, 32, 33, 34, 35] , and optimization [15, 36] . In offline MBO, there are inverse approaches to learning a mapping from function values to input domains with generative models and sample designs from high-scoring regions [11, 12, 14, 15] . DDOM [15] utilizes a conditional diffusion model and generates high-scoring samples with reweighted training and classifier-free guidance. DiffOPT [36] considers a constrained optimization setting and introduces a two-stage framework that begins with a guided diffusion process for warm-up, followed by a Langevin dynamics stage for further correction.As concurrent works, DEMO [37] trains a diffusion model to match a pseudo-target distribution constructed by gradient ascent and uses the model to edit designs in the offline dataset. Diff-BBO [38] measures the uncertainty of generated designs to select the optimal target value for conditioning the diffusion model. Our method distinguishes itself from prior works by utilizing diffusion models to generate trajectories toward high-scoring regions by learning to improve solutions from the dataset.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All of these can be found in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion and conclusion: 
 In this paper, we introduce GTG, a novel conditional generative modeling approach for learning to improve solutions from synthetic trajectories constructed with the dataset. First, we construct diverse trajectories toward high-scoring regions while incorporating locality bias. Then, we train the conditional diffusion model and proxy function. After training, we generate trajectories with classifier-free guidance and context-conditioning to generalize the knowledge on how to improve solutions. Lastly, our filtering strategy for selecting candidates further improves the performance.Our extensive experiments demonstrate the generalizability of GTG.Limitation and future work. While our method shows powerful generalizability on Design-Bench tasks, some evaluation methods may not fully capture real-world complexities. For example, in the superconductor task, we find that the offline dataset has multiple copies of the same inputs but with different outputs. As a result, the random forest oracle which is fit on this offline data is not reliable. Moreover, we resort to filtering designs with the proxy function, which may result in inaccurate predictions on OOD regions. Although our filtering strategy works well in sparse and noisy settings, one may consider constructing a robust proxy model to handle the uncertainty of its predictions.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We conduct all experiments with multiple random seeds and report error bars.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impact of the paper in Appendix E.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we provide more details on the baselines used for our experiments.Baselines from Design-Bench [5] . We take the implementations of most baselines from open-source code 1 . It contains baselines of BO-qEI [26] , CMA-ES [27] , REINFORCE [28] , Gradient Ascent, CbAS [11] , MINs [13] , and COMs [44] . We reproduce the results with 8 independent random seeds.NEMO [7] . NEMO leverages a normalized maximum likelihood estimator to handle uncertainty in unseen regions and prevent adversarial optimization while performing gradient ascent. As there is no open-source code, we refer to the results of NEMO from [9] .BDI [24] . BDI learns forward mapping from low-scoring regions to high-scoring regions, and its backward mapping distills the knowledge of the offline dataset to search for optimal designs. We follow the hyperparameter setting of the paper and reproduce the results with the open-source code 2 .ICT [9] . ICT maintains three symmetric proxies and enhances the performance of the ensemble by co-teaching and importance-aware sample reweighting. We follow the hyperparameter setting of the paper and reproduce the results with the open-source code 3 .DDOM [15] . DDOM leverages diffusion models to model distribution over high-scoring regions and sample designs with classifier-free guidance. We follow the hyperparameter setting of the paper except for the evaluation budget Q for a fair comparison. We find that there is a performance drop in several tasks when we use Q = 128 instead of 256. We reproduce the results with the open-source code 4 .BONET [16] . BONET trains an autoregressive model with trajectories constructed from the offline dataset and generalizes the knowledge to explore high-scoring regions. We follow the hyperparameter setting of the paper except for the evaluation budget Q for a fair comparison. We find that there is a performance drop in several tasks when we use Q = 128 instead of 256. We reproduce the results with the open-source code 5 .PGS [17] . PGS trains a policy to guide gradient-based optimization by reformulating the MBO problem as an offline RL problem. We follow the hyperparameter setting of the paper and reproduce the results with the open-source code 6 .In this section, we present additional analysis on GTG which is not included in the main section due to the page limit.We conduct additional analysis on our trajectory construction method. We try to generate diverse trajectories toward high-scoring regions by randomly selecting subsequent designs from K neighbors and allowing local perturbations. To this end, we visualize the shift in the distribution of function values via various trajectory construction strategies in the Superconductor task. As shown in Figure 7 , the SORT-SAMPLE strategy suggested by BONET constructs trajectories solely on high-scoring designs, which can be easily trapped into local optima. Unlike SORT-SAMPLE, our method shifts distribution towards high-scoring regions while using the information of low-scoring regions to distill the knowledge of the landscape of the target function to the generator. We also conduct additional analysis on hyperparameters in trajectory construction, K and ϵ. Figure 8 shows the performance of GTG in TFBind8 task by varying K and ϵ. While using too large K or too small ϵ may lead to a relatively low performance, we do not see much variation with different values. In this section, we explore various strategies for guiding diffusion models to generate high-scoring designs. As we also generate score values, it could be possible to guide diffusion models to generate high-scoring designs by inpainting score values with the desired values. To this end, we conduct additional experiments on Design-Bench tasks by generating trajectories with inpainting instead of classifier-free guidance. Specifically, we inpaint the y values of the generated trajectories as y * , the normalized score of the optimal design.Table 12 shows the performance of different guiding strategies. It confirms that conditioning by classifier-free guidance performs better than the inpainting strategy, justifying our decision choice. In this section, we explore the trade-off between performance and diversity via filtering strategy. While the filtering strategy boosts the performance of our method by eliminating potentially suboptimal designs, it may reduce the diversity of candidates, which may be crucial in tasks such as drug discovery due to proxy misspecification [45] .To this end, we measure the diversity of the candidates, following the procedure of [14] . For measurement, we use the average of the pairwise distance between candidates as below.EQUATIONwhere d(x, x ′ ) is a pairwise distance between samples. For discrete tasks, we use the hamming-ball distance metric. For continuous tasks, we compute L2 distance.Table 13 illustrates the effect of filtering on performance and diversity. As expected, we achieve higher performance through filtering while sacrificing the diversity of the candidate set. It might be beneficial to automatically balance performance and diversity trade-off by measuring the uncertainty of the proxy function. We leave it as a future work. In this section, we present extended experiment results in sparse and noisy datasets. As shown in Tables 16 and 17 , our method outperforms most baselines in various practical settings. Note that we cannot conduct experiments with NEMO and RoMA, as there is no code publicly available. BO-qEI 0.744 ± 0.089 0.716 ± 0.091 0.579 ± 0.114 0.891 ± 0.003 0.891 ± 0.012 0.884 ± 0.000 CMA-ES 0.968 ± 0.011 0.961 ± 0.013 0.876 ± 0.061 0.863 ± 0.022 0.852 ± 0.014 0.839 ± 0.012 REINFORCE 0.825 ± 0.054 0.879 ± 0.041 0.819 ± 0.051 0.409 ± 0.171 0.560 ± 0.194 0.619 ± 0.193 Grad Ascent 0.955 ± 0.022 0.938 ± 0.025 0.917 ± 0.034 0.911 ± 0.009 0.856 ± 0.024 0.830 ± 0.047 COMs 0.928 ± 0.028 0.938 ± 0.040 0.915 ± 0.057 0.936 ± 0.009 0.926 ± 0.012 0.925 ± 0.013 BDI 0.980 ± 0.005 0.886 ± 0.051 0.873 ± 0.048 0.929 ± 0.008 0.908 ± 0.010 0.918 ± 0.016 ICT 0.941 ± 0.013 0.950 ± 0.023 0.921 ± 0.054 0.940 ± 0.029 0.914 ± 0.024 0.896 ± 0.000 CbAS 0.916 ± 0.041 0.916 ± 0.034 0.906 ± 0.033 0.901 ± 0.009 0.898 ± 0.017 0.888 ± 0.013 MINs 0.885 ± 0.057 0.947 ± 0.032 0.883 ± 0.068 0.941 ± 0.006 0.938 ± 0.008 0.932 ± 0.008 DDOM 0.896 ± 0.048 0.887 ± 0.065 0.887 ± 0.065 0.944 ± 0.009 0.945 ± 0.011 0.926 ± 0.020 BONET 0.904 ± 0.044 0.822 ± 0.113 0.773 ± 0.143 0.942 ± 0.008 0.927 ± 0.024 0.924 ± 0.010 PGS 0.906 ± 0.030 0.911 ± 0.033 0.869 ± 0.039 0.942 ± 0.005 0.923 ± 0.009 0.891 ± 0.016 GTG (Ours) 0.976 ± 0.015 0.967 ± 0.026 0.948 ± 0.029 0.955 ± 0.008 0.947 ± 0.015 0.937 ± 0.013We present additional visualization results from the Toy 2D experiment. As shown in Figure 11 , GTG is able to generate diverse trajectories toward high-scoring designs by conditioning on different context points and classifier-free guidance. Optimization for real-world designs presents both opportunities and risks. For instance, while the design of new pharmaceuticals holds the promise of curing previously untreatable diseases, there is the potential for misuse, such as creating harmful biochemical agents. Researchers should be diligent to ensure that their innovations are employed in ways that contribute positively to societal welfare.Justification: We do not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.https://github.com/brandontrabucco/design-baselineshttps://github.com/GGchen1997/BDIhttps://github.com/StevenYuan666/Importance-aware-Co-teachinghttps://github.com/siddarthk97/ddomhttps://github.com/siddarthk97/bonethttps://github.com/yassineCh/PGS 


Decision: This paper is concerned with model-based optimization (MBO). The authors propose to leverage the diffusion models with guided sampling methods to synthesize trajectories to improve the solutions. Overall, I think the authors make a good contribution to MBO as they successfully demonstrate that the diffusion models can be used as a trajectory synthesizer for MBO tasks, which will motivate more studies on along this direction. The authors also provide good responses to ease the concerns from the reviewers. As a result, I recommend acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces Guided Trajectory Generation (GTG), a novel conditional generative modeling approach to solve the MBO problem. GTG consists of three parts, including trajectories construction, model training and trajectories sampling and filtering. Experimental results on various tasks, including a toy 2D experiment and Design-Bench variants, demonstrate the method's effectiveness.
strength_weakness: strength_weakness: 1. The method proposed is similar to Decision Diffuser [1]. The method proposed focuses on the offline model-based optimization problem while Decision Diffuser is used to sovle the offline RL problem.
2. It is not clear that in loss function, only x is used to calculate loss or both x and y are used.
3. In this paper, the concept of filtering in Section 3.4 and Section 5 are not clarified clearly.

[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
questions: questions: 1. Please clarify the difference between the method proposed in this paper and Decision Diffuser [1].
2. In Section 3.4 and Appendix D.3, the distribution of data used to train models must be close to the real distribution. Otherwise, out of distribution could occur, leading to an error evaluation of trajectories.
3. Please clarify the concept of filtering in Section 3.4 and Section 5. Whether they are the same operation or not?

[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
limitations: limitations: Yes, the authors have discussed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The paper consider the problem of offline optimization where the goal is to find the optima of a black-box function in a zero-shot manner without online evaluations. The key idea is to generate trajectories with a locality biased heuristic and employ a conditional diffusion model to learn the distribution of these trajectories. Experiments are performed on multiple tasks from design bench benchmark.
strength_weakness: strength_weakness: - This is not specific to the paper as such but the tasks in design-bench benchmark have multiple issues and as a result, some of the results are not reliable. For example, the offline dataset in superconductor task has multiple copies of the same inputs but with different outputs. As a result, the random forest oracle which is fit on this offline data is not reliable.

- The paper calls existing trajectory generation approaches (like sorted monotonic trajectories) as heuristics but the proposed approach is also a heuristic. Is there more principled/theoretical explanation for the proposed approach?
questions: questions: Please see weaknesses section.
limitations: limitations: Please see weaknesses section.
paper_summary: paper_summary: The paper introduces Guided Trajectory Generation (GTG) for offline model-based optimization (MBO). GTG creates synthetic trajectories from offline data, using locality bias to ensure consistent improvement. A conditional diffusion model generates these trajectories based on their scores. The method then uses guided sampling and a proxy function to identify the best designs. GTG shows superior performance on the Design-Bench benchmark compared to other methods.
strength_weakness: strength_weakness: - TFBind8 and TFBind10 are very similar. It would be better to see experiments on more varied discrete benchmarks like NAS. Also, why do the results for TFBind10 have such high variance?
- It’s not clear how this approach deals with out-of-distribution (OOD) problems. Diffusion models usually generate in-distribution data, so where does the robustness against OOD desings come from?
- The proxy function uses the standard regression approach, which can be quite fragile when estimating OOD designs. Have you investigated using COMS or ROMA models as a proxy?
- There isn't a principled mathematical explanation for why this approach works.
- The paper doesn’t include 50th percentile results, which are a standard part of evaluation in this field.
questions: questions: - Please check the weaknesses
- For the discrete tasks, Is a latent space used or is everything kept discrete during diffusion and proxy training?
- How many gradient steps are used for gradient ascent methods like COMs and RoMA? Is it 64? And is the trajectory length the same for BONET and PGS?
- Were the hyperparameters picked using the proxy or the oracle? 
- Are designs picked at the end of the generation process, or can they be selected from the middle of the trajectory? Is the filtering applied to results generated by BONET or PGS? How does this affect their outcomes? 
- How many designs are initially considered, and how many are selected at the end (128 out of how many)?
- Why is the perturbation epsilon different for DKitty?
limitations: limitations: The authors address the limitations and potential negative impacts in their paper.
paper_summary: paper_summary: The paper proposes a novel conditional generative modeling approach using diffusion models for offline model-based optimization (MBO). The method constructs synthetic trajectories toward high-scoring regions, trains a conditional diffusion model, and samples multiple trajectories to explore and select high-fidelity designs.
strength_weakness: strength_weakness: 1.	The authors have adopted the Diffusion model for offline model-based optimization, which is a current trending approach. It is recommended that the authors emphasize their unique contributions and clarify the distinctions between their method and previous approaches.
2.	The model filters candidates through a trained proxy model. The authors should elaborate on how data handling is managed during experiments to ensure that there is no label leakage and discuss whether the comparisons with other models are conducted fairly.
3.	The authors should provide a more detailed description and discussion of the experimental results.
questions: questions: Please see weaknesses
limitations: limitations: n/a

Comments: title: Response to Rebuttal
comment: Thank you for your kind response. If there are any remaining issues of discussion, please feel free to share them with us. We are always ready to engage in further discussion. Once again, we appreciate your thoughtful feedback.
Thank you for your answers. I have raised my rating.
Dear Reviewers,

We wanted to remind you that the author-reviewer discussion period ends in two days. We value your feedback and kindly request your response to our rebuttal.

Your participation will significantly help us enhance our work. If you have any questions or need more information, please don't hesitate to ask.

We appreciate your time and look forward to your input.

Best regards,

The Authors
Thank you for your positive feedback! We will incorporate an explanation of the unreliability of the Superconductor task and a discussion about the recent paper by Hoang et al. (2024) into our manuscript.

Please let me know if you have any further suggestions or adjustments you would like us to consider.
Thanks for your response to my questions. I am fairly convinced that this will be a good addition to the offline model based optimization literature. I would request to add a discussion about following things: 

- I appreciate the special focus on fixing the seed and reproducing the baselines but the Superconductor task has one specific issue that it has multiple copies of the same inputs but with different outputs. As a result, the oracle is not reliable even though we can reproduce the results. Please add this while presenting the results about this task.

- Hoang, M., Fadhel, A., Deshwal, A., Doppa, J., & Hoang, T. N. (2024). Learning Surrogates for Offline Black-Box Optimization via Gradient Matching. In Forty-first International Conference on Machine Learning. is another recent paper that constructs trajectories. It will be good to add discussion about this paper in the related work section.