Title: Robust Guided Diffusion for Offline Black-box Optimization
Venue: NeurIPS 2024
Accepted: no
Keywords: offline model-based optimization

black-box optimization

diffusion models

score-based SDE

guided diffusion

classifier diffusion guidance

classifier-free diffusion guidance
Abstract: Offline black-box optimization aims to maximize a black-box function using an offline dataset of designs and their measured properties. Two main approaches have emerged: the forward approach, which learns a mapping from input to its value, thereby acting as a proxy to guide optimization, and the inverse approach, which learns a mapping from value to input for conditional generation. (a) Although proxy-free (classifier-free) diffusion shows promise in robustly modeling the inverse mapping, it lacks explicit guidance from proxies, essential for generating high-performance samples beyond the training distribution. Therefore, we propose proxy-enhanced sampling which utilizes the explicit guidance from a trained proxy to bolster proxy-free diffusion with enhanced sampling control. (b) Yet, the trained proxy is susceptible to out-of-distribution issues. To address this, we devise the module diffusion-based proxy refinement, which seamlessly integrates insights from proxy-free diffusion back into the proxy for refinement. To sum up, we propose Robust Guided Diffusion for Offline Black-box Optimization (RGD), combining proxy and proxy-free diffusion for effective conditional generation. Empirical evaluations on design-bench underscore the efficacy of RGD. Our code is here.

Abstract

Offline black-box optimization aims to maximize a black-box function using an offline dataset of designs and their measured properties. Two main approaches have emerged: the forward approach, which learns a mapping from input to its value, thereby acting as a proxy to guide optimization, and the inverse approach, which learns a mapping from value to input for conditional generation. (a) Although proxy-free (classifier-free) diffusion shows promise in robustly modeling the inverse mapping, it lacks explicit guidance from proxies, essential for generating highperformance samples beyond the training distribution. Therefore, we propose proxy-enhanced sampling which utilizes the explicit guidance from a trained proxy to bolster proxy-free diffusion with enhanced sampling control. (b) Yet, the trained proxy is susceptible to out-of-distribution issues. To address this, we devise the module diffusion-based proxy refinement , which seamlessly integrates insights from proxy-free diffusion back into the proxy for refinement. To sum up, we propose R obust G uided D iffusion for Offline Black-box Optimization ( RGD ), combining the advantages of proxy (explicit guidance) and proxy-free diffusion (robustness) for effective conditional generation. RGD achieves state-of-the-art results on various design-bench tasks, underscoring its efficacy. Our code is here.

1 Introduction

Creating new objects to optimize specific properties is a ubiquitous challenge that spans a multitude of fields, including material science, robotic design, and genetic engineering. Traditional methods generally require interaction with a black-box function to generate new designs, a process that could be financially burdensome and potentially perilous [1, 2]. Addressing this, recent research endeavors have pivoted toward a more relevant and practical context, termed offline black-box optimization (BBO) [3, 4]. In this context, the goal is to maximize a black-box function exclusively utilizing an offline dataset of designs and their measured properties.

There are two main approaches for this task: the forward approach and the reverse approach. The forward approach entails training a deep neural network (DNN), parameterized as J ϕ ( · ) , using the offline dataset. Once trained, the DNN acts as a proxy and provides explicit gradient guidance to enhance existing designs. However, this technique is susceptible to the out-of-distribution (OOD) issue, leading to potential overestimation of unseen designs and resulting in adversarial solutions [5].

The reverse approach aims to learn a mapping from property value to input. Inputting a high value 31 into this mapping directly yields a high-performance design. For example, MINs [6] adopts GAN [7] 32 to model this inverse mapping, and demonstrate some success. Recent works [4] have applied 33 proxy-free diffusion 1 [8], parameterized by θ , to model this mapping, which proves its efficacy over 34

other generative models. Proxy-free diffusion employs a score predictor ˜ s θ ( · , · , ω ) . This represents a 35 linear combination of conditional and unconditional scores, modulated by a strength parameter ω to 36 balance condition and diversity in the sampling process. This guidance significantly diverges from 37 proxy (classifier) diffusion that interprets scores as classifier gradients and thus generates adversarial 38 solutions. Such a distinction grants proxy-free diffusion its inherent robustness in generating samples. 39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

xd

1

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

xd

2

Negative Rosenbrock Function

Initial points

Proxy-free diffusion

Proxy-enhanced sampling

3600

3000

2400

1800

1200

600

0

Nevertheless, proxy-free diffusion, initially designed for in-distribution generation, such as synthesizing specific image categories, faces limitations in offline BBO. Particularly, it struggles to generate high-performance samples that exceed the training distribution due to the lack of explicit guidance 2 . Consider, for example, the optimization of a two-dimensional variable ( x d 1 , x d 2 ) to maximize the negative Rosenbrock function [9]: y ( x d 1 , x d 2 ) = -(1 -x d 1 ) 2 -100( x d 2 -x 2 d 1 ) 2 , as depicted in Figure 1. The objective is to steer the initial points (indicated in pink) towards the high-performance region (highlighted in yellow). While proxy- free diffusion can nudge the initial points closer to this high-performance region, the generated points (depicted in blue) fail to reach the high-performance region due to its lack of explicit proxy guidance.

To address this challenge, we introduce a proxy-enhanced sampling module as illustrated in Figure 2(a). It incorporates the explicit guidance from the proxy J ϕ ( x ) into proxy-free diffusion to enable enhanced control over the sampling process. This module hinges on the strategic optimization of the strength parameter ω to achieve a better balance between condition and diversity, per reverse diffusion step. This incorporation not only preserves the inherent robustness of proxy-free diffusion but also leverages the explicit proxy guidance, thereby enhancing the overall conditional generation efficacy. As illustrated in Figure 1, samples (depicted in red) generated via proxy-enhanced sampling are more effectively guided towards, and often reach, the high-performance area (in yellow).

Forward diffusion

Reverse diffusion

Final design

Diffusion-based

proxy refinement

Proxy-enhanced sampling (optimizing    )

(a)

(b)

Diffusion distribution

Proxy

Probability flow ODE

...

Yet, the trained proxy is susceptible to out-ofdistribution (OOD) issues. To address this, we devise a module diffusion-based proxy refinement as detailed in Figure 2(b). This module seamlessly integrates insights from proxy-free diffusion into the proxy J ϕ ( x ) for refinement. Specifically, we generate a diffusion distribution p θ ( y | ˆ x ) on adversarial samples ˆ x , using the associated probability flow ODE 3 . This distribution is derived independently of a proxy, thereby exhibiting greater robustness than the proxy distribution on adversarial samples. Subsequently, we calculate the KullbackLeibler divergence between the two distributions on adversarial samples, and use this divergence minimization as a regularization strategy to fortify the proxy's robustness and reliability.

To sum up, we propose R obust G uided D iffusion for Offline Black-box Optimization ( RGD ), a novel framework that combines the advantages of proxy (explicit guidance) and proxy-free diffusion (robustness) for effective conditional generation. Our contributions are three-fold:

We propose a proxy-enhanced sampling module which incorporates proxy guidance into proxy-free diffusion to enable enhanced sampling control.

We further develop diffusion-based proxy refinement which integrates insights from proxy-free diffusion back into the proxy for refinement.

RGD delivers state-of-the-art performance on various design-bench tasks, emphasizing its efficacy. 87

2 Preliminaries 88

2.1 Offline Black-box Optimization 89

90

91

Offline black-box optimization (BBO) aims to maximize a black-box function with an offline dataset. Imagine a design space as X = R d , where d is the design dimension. The offline BBO [3] is:

In this equation, J ( · ) is the unknown objective function, and x ∈ X is a possible design. In this 92 context, there is an offline dataset, D , that consists of pairs of designs and their measured properties. 93 Specifically, each x denotes a particular design, like the size of a robot, while y indicates its related 94 metric, such as its speed. 95

A common approach gradient ascent fits a proxy distribution p ϕ ( y | x ) = N ( J ϕ ( x ) , σ ϕ ( x )) to the 96 offline dataset where ϕ denote the proxy parameters: 97

For the sake of consistency with terminology used in the forthcoming subsection on guided diffusion, we will refer to p ϕ ( ·|· ) as the proxy distribution and J ϕ ( · ) as the proxy. Subsequently, this approach performs gradient ascent with J ϕ ( x ) , leading to high-performance designs x ∗ :

converging to x M after M steps. However, this method suffers from the out-of-distribution issue where the proxy predicts values that are notably higher than the actual values.

2.2 Diffusion Models

Diffusion models, a type of latent variable models, progressively introduce Gaussian noise to data in the forward process, while the reverse process aims to iteratively remove this noise through a learned score estimator. In this work, we utilize continuous time diffusion models governed by a stochastic differential equation (SDE), as presented in [10]. The forward SDE is formulated as:

where f ( · , t ) : R d → R d represents the drift coefficient, g ( · ) : R → R denotes the diffusion coefficient and w is the standard Wiener process. This SDE transforms data distribution into noise distribution. The reverse SDE is:

with ∇ x log p ( x ) representing the score of the marginal distribution at time t , and ¯ w symbolizing the reverse Wiener process. The score function ∇ x log p ( x ) is estimated using a time-dependent neural network s θ ( x t , t ) , enabling us to transform noise into samples. For simplicity, we will use s θ ( x t ) , implicitly including the time dependency t .

2.3 Guided Diffusion

Guided diffusion seeks to produce samples with specific desirable attributes, falling into two categories: proxy diffusion [11] and proxy-free diffusion [8]. While these were initially termed classifier diffusion and classifier-free diffusion in classification tasks, we have renamed them to proxy diffusion and proxy-free diffusion , respectively, to generalize to our regression context. Proxy diffusion combines the model's score estimate with the gradient from the proxy distribution, providing explicit guidance. However, it can be interpreted as a gradient-based adversarial attack.

Proxy-free guidance, not dependent on proxy gradients, enjoys an inherent robustness of the sampling process. Particularly, it models the score as a linear combination of an unconditional and a conditional score. A unified neural network s θ ( x t , y ) parameterizes both score types. The score s θ ( x t , y )

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

where

η

denotes the learning rate.

approximates the gradient of the log probability ∇ x t log p ( x t | y ) , i.e., the conditional score, while s θ ( x t ) estimates the gradient of the log probability ∇ x t log p ( x t ) , i.e., the unconditional score. The score function follows:

Within this context, the strength parameter ω specifies the generation's adherence to the condition y , which is set to the maximum value y max in the offline dataset following [4]. Optimization of ω balances the condition and diversity. Lower ω values increase sample diversity at the expense of conformity to y , and higher values do the opposite.

3 Method

In this section, we present our method RGD, melding the strengths of proxy and proxy-free diffusion for effective conditional generation. Firstly, we describe a newly developed module termed proxy-enhanced sampling . It integrates explicit proxy guidance into proxy-free diffusion to enable enhanced sampling control, as detailed in Section 3.1. Subsequently, we explore diffusion-based proxy refinement which incorporates insights gleaned from proxy-free diffusion back into the proxy, further elaborated in Section 3.2. The overall algorithm is shown in Algorithm 1.

3.1 Proxy-enhanced Sampling

As discussed in Section 2.3, proxyfree diffusion trains an unconditional model and conditional models. Although proxy-free diffusion can generate samples aligned with most conditions, it traditionally lacks control due to the absence of an explicit proxy. This is particularly significant in offline BBO where we aim to obtain samples beyond the training distribution. Therefore, we require explicit proxy guidance to achieve enhanced sampling control. This module is outlined in Algorithm 1, Line 8 - Line 16 .

Optimization of ω . Directly updating the design x t with proxy gradient suffers from the OOD issue and determining a proper condition y necessitates the manual adjustment of multiple hyperparameters [6]. Thus, we propose to introduce proxy guidance by only optimizing the strength parameter ω within ˜ s θ ( x t , y, ω ) in Eq. (6). As discussed in Section 2.3, the parameter ω balances the condition and diversity, and an optimized ω could achieve a better balance in the sampling process, leading to more effective generation.

Enhanced Sampling. With the score function, the update of a noisy sample x t +1 is computed as:

where the solver is the second-order Heun solver [12], chosen for its enhanced accuracy through a predictor-corrector method. A proxy is then trained to predict the property of noise x t at time step t , denoted as J ϕ ( x t , t ) . By maximizing J ϕ ( x t ( ω ) , t ) with respect to ω , we can incorporate the explicit proxy guidance into proxy-free diffusion to enable enhanced sampling control in the balance between condition and diversity. This maximization process is:

We leverage the automatic differentiation capabilities of Py-

Torch [13] to efficiently compute the above derivatives within the context of the solver's operation. 169

170

The optimized

ˆ

ω

then updates the noisy sample

x

t

+1

through:

This process iteratively denoises x t , utilizing it in successive steps to progressively approach x 0 , 171 which represents the final high-scoring design x ∗ . 172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

Proxy Training. Notably, J ϕ ( x t , t ) can be directly derived from the proxy J ϕ ( x ) , the mean of the proxy distribution p ϕ ( ·| x ) in Eq. (2). This distribution is trained exclusively at the initial time step t = 0 , eliminating the need for training across time steps. To achieve this derivation, we reverse the diffusion from x t back to x 0 using the formula:

where s θ ( x t ) is the estimated unconditional score at time step t , and σ ( t ) 2 and µ ( t ) are the variance and mean functions of the perturbation kernel at time t , as detailed in equations (32-33) in [10]. Consequently, we express

This formulation allows for the optimization of the strength parameter ω via Eq. (8). For simplicity, we will refer to J ϕ ( · ) in subsequent discussions.

3.2 Diffusion-based Proxy Refinement

In the proxy-enhanced sampling module, the proxy J ϕ ( · ) is employed to update the parameter ω to enable enhanced control. However, J ϕ ( · ) may still be prone to the OOD issue, especially on adversarial samples [5]. To address this, we refine the proxy by using insights from proxy-free diffusion. The procedure of this module is specified in Algorithm 1, Lines 3 -7 .

Diffusion Distribution . Adversarial samples are identified by gradient ascent on the proxy as per Eq. (3) to form the distribution q ( x ) . Consequently, these samples are vulnerable to the proxy distribution. Conversely, the proxy-free diffusion, which functions without depending on a proxy, inherently offers greater resilience against these samples, thus producing a more robust distribution. For an adversarial sample ˆ x ∼ q ( x ) , we compute p θ (ˆ x ) , p θ (ˆ x | y ) via the probability flow ODE, and p ( y ) through Gaussian kernel-density estimation. The diffusion distribution regarding y is derived as:

which demonstrates inherent robustness over the proxy distribution p ϕ ( y | ˆ x ) . Yet, directly applying diffusion distribution to design optimization by gradient ascent is computationally intensive and potentially unstable due to the demands of reversing ODEs and scoring steps.

Proxy Refinement. We opt for a more feasible approach: refine the proxy distribution p ϕ ( y | ˆ x ) = N ( J ϕ (ˆ x ) , σ ϕ (ˆ x )) by minimizing its distance to the diffusion distribution p θ ( y | ˆ x ) . The distance is quantified by the Kullback-Leibler (KL) divergence:

We avoid the parameterization trick for minimizing this divergence as it necessitates backpropagation through p θ ( y | ˆ x ) , which is prohibitively expensive. Instead, for the sample ˆ x , the gradient of the KL divergence D ( p ϕ || p θ ) with respect to the proxy parameters ϕ is computed as:

Complete derivations are in Appendix A. The KL divergence then acts as regularization in our loss L :

where D is the training dataset and α is a hyperparameter. We propose to optimize α based on the validation loss via bi-level optimization as detailed in Appendix B.

4 Experiments

In this section, we conduct comprehensive experiments to evaluate our method's performance.

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

4.1 Benchmarks

Tasks. Our experiments encompass a variety of tasks, split into continuous and discrete categories.

The continuous category includes four tasks: (1) Superconductor (SuperC) 4 : The objective here is to engineer a superconductor composed of 86 continuous elements. The goal is to enhance the critical temperature using 17 , 010 design samples. This task is based on the dataset from [1]. (2) Ant Morphology (Ant): In this task, the focus is on developing a quadrupedal ant robot, comprising 60 continuous parts, to augment its crawling velocity. It uses 10 , 004 design instances from the dataset in [3, 14]. (3) D'Kitty Morphology (D'Kitty): Similar to Ant Morphology, this task involves the design of a quadrupedal D'Kitty robot with 56 components, aiming to improve its crawling speed with 10 , 004 designs, as described in [3, 15]. (4) Rosenbrock (Rosen): The aim of this task is to optimize a 60-dimension continuous vector to maximize the Rosenbrock black-box function. It uses 50000 designs from the low-scoring part [9].

For the discrete category, we explore three tasks: (1) TF Bind 8 (TF8): The goal is to identify an 8 -unit DNA sequence that maximizes binding activity. This task uses 32 , 898 designs and is detailed in [16]. (2) TF Bind 10 (TF10): Similar to TF8, but with a 10 -unit DNA sequence and a larger pool of 50 , 000 samples, as described in [16]. (3) Neural Architecture Search (NAS): This task focuses on discovering the optimal neural network architecture to improve test accuracy on the CIFAR10 dataset, using 1 , 771 designs [17].

Evaluation. In this study, we utilize the oracle evaluation from design-bench [3]. Adhering to this established protocol, we analyze the top 128 promising designs from each method. The evaluation metric employed is the 100 th percentile normalized ground-truth score, calculated using the formula y n = y -y min y max -y min , where y min and y max signify the lowest and highest scores respectively in the comprehensive, yet unobserved, dataset. In addition to these scores, we provide an overview of each method's effectiveness through the mean and median rankings across all evaluated tasks. Notably, the best design discovered in the offline dataset, designated as D ( best ) , is also included for reference. For further details on the 50 th percentile (median) scores, please refer to Appendix C.

4.2 Comparison Methods

Our approach is evaluated against two primary groups of baseline methods: forward and inverse approaches. Forward approaches enhance existing designs through gradient ascent. This includes: (i) Grad: utilizes simple gradient ascent on current designs for new creations; (ii) ROMA [18]: implements smoothness regularization on proxies; (iii) COMs [5]: applies regularization to assign lower scores to adversarial designs; (iv) NEMO [19]: bridges the gap between proxy and actual functions using normalized maximum likelihood; (v) BDI [20]: utilizes both forward and inverse mappings to transfer knowledge from offline datasets to the designs; (vi) IOM [21]: ensures consistency between representations of training datasets and optimized designs.

Inverse approaches focus on learning a mapping from a design's property value back to its input. High property values are input into this inverse mapping to yield enhanced designs. This includes: (i) CbAS [22]: CbAS employs a VAE model to implicitly implement the inverse mapping. It gradually tunes its distribution toward higher scores by raising the scoring threshold. This process can be interpreted as incrementally increasing the conditional score within the inverse mapping framework. (ii) Autofocused CbAS (Auto.CbAS) [23]: adopts importance sampling for retraining a regression model based on CbAS. (iii) MIN [6]: maps scores to designs via a GAN model and explore this mapping for optimal designs. (iv) BONET [24]: introduces an autoregressive model for sampling high-scoring designs. (v) DDOM[4]: utilizes proxy-free diffusion to model the inverse mapping.

Traditional methods as detailed in [3] are also considered: (i) CMA-ES [25]: modifies the covariance matrix to progressively shift the distribution towards optimal designs; (ii) BO-qEI [26]: implements Bayesian optimization to maximize the proxy and utilizes the quasi-Expected-Improvement acquisition function for design suggestion, labeling designs using the proxy; (iii) REINFORCE [27]: enhances the input space distribution using the learned proxy model.

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

4.3 Experimental Configuration

In alignment with the experimental protocols established in [3, 20], we have tailored our training methodologies for all approaches, except where specified otherwise. For methods such as BO-qEI, CMA-ES, REINFORCE, CbAS, and Auto.CbAS that do not utilize gradient ascent, we base our approach on the findings reported in [3]. We adopted T = 1000 diffusion sampling steps, set the condition y to y max , and initial strength ω as 2 in line with [4]. To ensure reliability and consistency in our comparative analysis, each experimental setting was replicated across 8 independent runs, unless stated otherwise, with the presentation of both mean values and standard errors. These experiments were conducted using a NVIDIA GeForce V 100 GPU. We've detailed the computational overhead of our approach in Appendix D to provide a comprehensive view of its practicality.

4.4 Results and Analysis

In Tables 1 and 2, we showcase our experimental results for both continuous and discrete tasks. To clearly differentiate among the various approaches, distinct lines separate traditional, forward, and inverse approaches within the tables For every task, algorithms performing within a standard deviation of the highest score are emphasized by bolding following [5].

We make the following observations. (1) As highlighted in Table 2, RGD not only achieves the top rank but also demonstrates the best performance in six out of seven tasks, emphasizing the robustness and superiority of our method. (2) RGD outperforms the VAE-based CbAS, the GAN-based MIN

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

and the Transformer-based BONET. This result highlights the superiority of diffusion models in modeling inverse mappings compared to other generative approaches. (3) Upon examining TF Bind 8, we observe that the average rankings for forward and inverse methods stand at 10 . 3 and 6 . 0 , respectively. In contrast, for TF Bind 10, both methods have the same average ranking of 8 . 7 , indicating no advantage. This notable advantage of inverse methods in TF Bind 8 implies that the relatively smaller design space of TF Bind 8 ( 4 8 ) facilitates easier inverse mapping, as opposed to the more complex space in TF Bind 10 ( 4 10 ). (4) RGD's performance is less impressive on NAS, where designs are encoded as 64 -length sequences of 5 -category one-hot vectors. This may stem from the design-bench's encoding not fully capturing the sequential and hierarchical aspects of network architectures, affecting the efficacy of inverse mapping modeling.

4.5 Ablation Studies

In this section, we present a series of ablation studies to scrutinize the individual contributions of distinct components in our methodology. We employ our proposed approach as a benchmark and methodically exclude key modules, such as the proxy-enhanced sampling and diffusion-based proxy refinement , to assess their influence on performance. These variants are denoted as w/o proxy-e and w/o diffusion-b r . Additionally, we explore the strategy of directly performing gradient ascent on the diffusion intermediate state, referred to as direct grad update . The results from these ablation experiments are detailed in Table 3.

Our analysis reveals that omitting either module results in a decrease in performance, thereby affirming the importance of each component. The w/o diffusion-b r variant generally surpasses w/o proxy-e , highlighting the utility of the proxy-enhanced sampling even with a basic proxy setup. Conversely, direct grad update tends to produce subpar results across tasks, likely attributable to the proxy's limitations in handling out-of-distribution samples, leading to suboptimal design optimizations.

To further dive into the proxy-enhanced sampling module, we visualize the strength ratio ω/ω 0 -where ω 0 represents the initial strength-across diffusion steps t . This analysis is depicted in Figure 3 for two specific tasks: Ant and TF10. We observe a pattern of initial decrease followed by an increase in ω across both tasks. This pattern can be interpreted as follows: The decrease in ω facilitates the generation of a more diverse set of samples, enhancing exploratory capabilities. Subsequently, the increase in ω signifies a shift towards integrating high-performance features into the sample generation. Within this context, conditioning on the maximum y is not aimed at achieving the

0

100

200

300

400

500

600

700

800

900

1000

Diffusion step t

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Strength ratio

/

0

Ant

TF10

dataset's maximum but at enriching samples with high-scoring attributes. Overall, this adjustment of ω effectively balances between generating novel solutions and honing in on high-quality ones.

In addition, we visualize the proxy distribution alongside the diffusion distribution for a sample ˆ x from the Ant task in Figure 4, to substantiate the efficacy of diffusion-based proxy refinement. The proxy distribution significantly overestimates the ground truth, whereas the diffusion distribution closely aligns with it, demonstrating the robustness of diffusion distribution. For a more quantitative

318

319

analysis, we compute the expectation of both distributions and compare them with the ground truth. The mean of the diffusion distribution is calculated as E p θ ( y | ˆ x ) [ y ] = E p ϕ ( y | ˆ x ) [ p θ ( y | ˆ x ) p ϕ ( y | ˆ x ) y ] .

0.6

0.8

1.0

1.2

0

1

2

3

4

Prob Density

Proxy Distribution

p

(

y

|

x

)

Peak of Proxy Distribution

Diffusion Distribution

p

(

y

|

x

)

Peak of Diffusion Distribution

Ground-truth

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

The MSE loss for the proxy distribution is 2 . 88 , while for the diffusion distribution, it is 0 . 13 on the Ant task. Additionally, we evaluate this on the TFB10 task, where the MSE loss for the proxy distribution is 323 . 63 compared to 0 . 82 for the diffusion distribution. These results further corroborate the effectiveness of our proposed module.

Furthermore, we (1) investigate the impact of replacing our trained proxy model with alternative approaches, specifically ROMA and COMs, (2) analyze the performance with an optimized condition y and (3) explore a simple annealing approach of ω . For a comprehensive discussion on these, readers are referred to Appendix E.

Y

4.6 Hyperparameter Sensitivity Analysis

This section investigates the sensitivity of RGD to various hyperparameters. Specifically, we analyze the effects of (1) the number of diffusion sampling steps T , (2) the condition y , and (3) the learning rate η of the proxy-enhanced sampling. These parameters are evaluated on two tasks: the continuous Ant task and the discrete TFB10 task. For a detailed discussion, see Appendix F.

5 Related Work

Offline black-box optimization. A recent surge in research has presented two predominant approaches for offline BBO. The forward approach deploys a DNN to fit the offline dataset, subsequently utilizing gradient ascent to enhance existing designs. Typically, these techniques, including COMs [5], ROMA [18], NEMO [19], BDI [20, 28], IOM [29] and Parallel-mentoring [30], are designed to embed prior knowledge within the surrogate model to alleviate the OOD issue. The reverse approach [6, 31] is dedicated to learning a mapping from property values back to inputs. Feeding a high value into this inverse mapping directly produces a design of elevated performance. Additionally, methods in [22, 23] progressively tailor a generative model towards the optimized design via a proxy function and BONET [24] introduces an autoregressive model trained on fixed-length trajectories to sample high-scoring designs. Recent investigations [4] have underscored the superiority of diffusion models in delineating the inverse mapping. However, research on specialized guided diffusion for offline BBO remains limited. This paper addresses this research gap.

Guided diffusion. Guided diffusion seeks to produce samples with specific desirable attributes. Contemporary research in guided diffusion primarily concentrates on enhancing the efficiency of its sampling process. [32] propose a method for distilling a classifier-free guided diffusion model into a more efficient single model that necessitates fewer steps in sampling. [33] introduce an operator splitting method to expedite classifier guidance by separating the update process into two key functions: the diffusion function and the conditioning function. Additionally, [34] presents an efficient and universal guidance mechanism that utilizes a readily available proxy to enable diffusion guidance across time steps. In this work, we explore the application of guided diffusion in offline BBO, with the goal of creating tailored algorithms to efficiently generate high-performance designs.

6 Conclusion

In conclusion, we propose R obust G uided D iffusion for Offline Black-box Optimization ( RGD ). The proxy-enhanced sampling module adeptly integrates proxy guidance to enable enhanced sampling control, while the diffusion-based proxy refinement module leverages proxy-free diffusion insights for proxy improvement. Empirical evaluations on design-bench have showcased RGD's outstanding performance, further validated by ablation studies on the contributions of these novel components. We discuss the broader impact and limitation in Appendix G.

368

369

370

371

References

Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational materials science , 2018.

Karen S Sarkisyan et al. Local fitness landscape of the green fluorescent protein. Nature , 2016.

Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-Bench: bench372 marks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450 , 373 2022. 374

375

376

Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Diffusion models for black-box optimization. Proc. Int. Conf. Machine Learning (ICML) , 2023.

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proc. Int. Conf. Machine Learning (ICML) , 2021.

Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2020.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2014.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

HoHo Rosenbrock. An automatic method for finding the greatest or least value of a function. The computer journal , 1960.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Proc. Int. Conf. Learning Rep. (ICLR) , 2021.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Proc. 392 Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2021. 393

Endre Süli and David F Mayers. An introduction to numerical analysis . Cambridge university 394 press, 2003. 395

396

397

398

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: an imperative style, high-performance deep learning library. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2019.

399

400

401

402

403

404

405

406

407

408

409

410

411

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: robotics benchmarks for learning with low-cost robots. In Conf. on Robot Lea. (CoRL) , 2020.

Luis A Barrera et al. Survey of variation in human transcription factors reveals prevalent DNA binding changes. Science , 2016.

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2017.

Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: robust model adaptation for offline model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2021.

Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. Proc. Int. Conf. Learning Rep. (ICLR) , 2021.

Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline 412 infinite-width model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2022. 413

Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven model-based optimization via 414 invariant representation learning. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2022. 415

David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling 416 for robust design. In Proc. Int. Conf. Machine Learning (ICML) , 2019. 417

Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. Proc. 418 Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2020. 419

Satvik Mehul Mashkaria, Siddarth Krishnamoorthy, and Aditya Grover. Generative pretraining 420 for black-box optimization. In Proc. Int. Conf. Machine Learning (ICML) , 2023. 421

Nikolaus Hansen. The CMA evolution strategy: a comparing review. Towards A New Evolu422 tionary Computation , 2006. 423

James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparame424 terization trick for acquisition functions. arXiv preprint arXiv:1712.00424 , 2017. 425

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce426 ment learning. Machine learning , 1992. 427

Can Chen, Yingxue Zhang, Xue Liu, and Mark Coates. Bidirectional learning for offline 428 model-based biological sequence design. In Proc. Int. Conf. Machine Lea. (ICML) , 2023. 429

Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven model-based optimization via 430 invariant representation learning. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2022. 431

432

433

Can Chen, Christopher Beckham, Zixuan Liu, Xue Liu, and Christopher Pal. Parallel-mentoring for offline model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2023.

434

435

436

437

438

439

440

441

442

443

Alvin Chan, Ali Madani, Ben Krause, and Nikhil Naik. Deep extrapolation for attributeenhanced generation. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2021.

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proc. Comp. Vision. Pattern. Rec.(CVPR) , 2023.

Suttisak Wizadwongsa and Supasorn Suwajanakorn. Accelerating guided diffusion sampling with splitting numerical methods. In Proc. Int. Conf. Learning Rep. (ICLR) , 2023.

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proc. Comp. Vision. Pattern. Rec.(CVPR) , 2023.

Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline 444 infinite-width model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2022. 445

A Derivation 446

This section provides a derivation of the gradient of the KL divergence. Let's consider the KL 447 divergence term, defined as: 448

The gradient with respect to the parameters ϕ is computed as follows: 449

B Hyperparameter Optimization 450

We propose adjusting α based on the validation loss, establishing a bi-level optimization framework: 451

Within this context, D v represents the validation dataset sampled from the offline dataset. The inner optimization task, which seeks the optimal ϕ ∗ ( α ) , is efficiently approximated via gradient descent.

C Evaluation of Median Scores

While the main text of our paper focuses on the 100 th percentile scores, this section provides an in-depth analysis of the 50 th percentile scores. These median scores, previously explored in [3], serve as an additional metric to assess the performance of our RGD method. The outcomes for continuous tasks are detailed in Table 5, and those pertaining to discrete tasks, along with their respective ranking statistics, are outlined in Table 6. An examination of Table 6 highlights the notable success of the RGD approach, as it achieves the top rank in this evaluation. This finding underscores the method's robustness and effectiveness.

D Computational Overhead

In this section, we analyze the computational overhead of our method. RGD consists of two core components: proxy-enhanced sampling ( proxy-e sampling ) and diffusion-based proxy refinement ( diffusion-b proxy r ). Additionally, RGD employs a trained proxy and a proxy-free diffusion model, whose computational demands are denoted as proxy training and diffusion training , respectively.

Table 4 indicates that experiments can be completed within approximately one hour, demonstrating efficiency. The diffusion-based proxy refinement module is the primary contributor to the computational overhead, primarily due to the usage of a probability flow ODE for sample likelihood computation.

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

However, as this is a one-time process for refining the proxy, its high computational cost is offset by its 470 non-recurring nature. In contexts such as robotics or bio-chemical research, the most time-intensive 471 part of the production cycle is usually the evaluation of the unknown objective function. Therefore, 472 the time differences between methods for deriving high-performance designs are less critical in 473 actual production environments, highlighting RGD's practicality where optimization performance 474 are prioritized over computational speed. This aligns with recent literature (A.3 Computational 475 Complexity in [35] and A.7.5. Computational Cost in [28]) indicating that in black-box optimization 476 scenarios, computational time is relatively minor compared to the time and resources dedicated to 477 experimental validation phases. 478

E Further Ablation Studies

479

In this section, we extend our exploration to include alternative proxy refinement schemes, namely 480 ROMA and COMs, to compare against our diffusion-based proxy refinement module. The objective 481 is to assess the relative effectiveness of these schemes in the context of the Ant and TFB10 tasks. 482 The comparative results are presented in Table 7. Our investigation reveals that proxies refined 483 through ROMA and COMs exhibit performance akin to the vanilla proxy and they fall short of 484 achieving the enhancements seen with our diffusion-based proxy refinement. We hypothesize that 485 the diffusion-based proxy refinement, by aligning closely with the characteristics of the diffusion 486

model, provides a more relevant and impactful signal. This alignment improves the proxy's ability to 487 enhance the sampling process more effectively. 488

Additionally, we contrast our approach, which adjusts the strength parameter ω , with the MIN method that focuses on identifying an optimal condition y . The MIN strategy entails optimizing a Lagrangian objective with respect to y , a process that requires manual tuning of four hyperparameters. We adopt their methodology to determine optimal conditions y and incorporate these into the proxy-free diffusion for tasks Ant and TF10. The normalized scores for Ant and TF10 are 0 . 950 ± 0 . 017 and 0 . 660 ± 0 . 027 , respectively. The outcomes fall short of those achieved by our method as detailed in Table 7. This discrepancy likely stems from the complexity involved in optimizing y , whereas dynamically adjusting ω proves to be a more efficient strategy for enhancing sampling control.

Last but not least, we explore simple annealing approaches for ω . Specifically, we test two annealing scenarios considering the default ω as 2 . 0 : (1) a decrease from 4 . 0 to 0 . 0 , and (2) an increase from 0 . 0 to 4 . 0 , both modulated by a cosine function over the time step ( t ). We apply these strategies to the Ant Morphology and TF Bind 10 tasks, and the results are as follows:

The empirical results across both strategies illustrate their inferior performance compared to our approach, thereby demonstrating the efficacy of our proposed method.

F Hyperparameter Sensitivity Analysis

RGD's performance is assessed under different settings of T , y , and η . We experiment with T values of 500 , 750 , 1000 , 1250 , and 1500 , with the default being T = 1000 . For the condition ratio y/y max , we test values of 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 , and 2 . 5 , considering 1 . 0 as the default. Similarly, for the learning rate η , we explore values of 2 . 5 e -3 , 5 . 0 e -3 , 0 . 01 , 0 . 02 , and 0 . 04 , with the default set to η = 0 . 01 . Results are normalized by comparing them with the performance obtained at default values.

As depicted in Figures 5, 6, and 7, RGD demonstrates considerable resilience to hyperparameter variations. The Ant task, in particular, exhibits a more marked sensitivity, with a gradual enhancement in performance as these hyperparameters are varied. The underlying reasons for this trend include: (1) An increase in the number of diffusion steps ( T ) enhances the overall quality of the generated samples. This improvement, in conjunction with more effective guidance from the trained proxy, leads to better results. (2) Elevating the condition ( y ) enables the diffusion model to extend its reach beyond the existing dataset, paving the way for superior design solutions. However, selecting an optimal y can be challenging and may, as observed in the TFB10 task, sometimes lead to suboptimal results. (3) A higher learning rate ( η ) integrates an enhanced guidance signal from the trained proxy, contributing to improved performances.

In contrast, the discrete nature of the TFB10 task seems to endow it with a certain robustness to variations in these hyperparameters, highlighting a distinct behavioral pattern in response to hyperparameter adjustments.

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

500

750

1000

1250

1500

Diffusion step

t

0.990

0.995

1.000

1.005

1.011

Score ratio

Ant

TF

10

0.5

1.0

1.5

2.0

2.5

Condition

y

/

ymax

0.960

1.000

1.004

1.008

1.012

Score ratio

Ant

TF

10

G Broader Impact and Limitation

Broader impact. Our research has the potential to significantly accelerate advancements in fields such as new material development, biomedical innovation, and robotics technology. These advancements could lead to breakthroughs with substantial positive societal impacts. However, we recognize that, like any powerful tool, there are inherent risks associated with the misuse of this technology. One concerning possibility is the exploitation of our optimization techniques to design objects or entities for malicious purposes, including the creation of more efficient weaponry or harmful biological agents. Given these potential risks, it is imperative to enforce strict safeguards and regulatory measures, especially in areas where the misuse of technology could lead to significant ethical and societal harm. The responsible application and governance of such technologies are crucial to ensuring that they serve to benefit society as a whole.

Limitation. We recognize that the benchmarks utilized in our study may not fully capture the 533 complexities of more advanced applications, such as protein drug design, primarily due to our current 534 limitations in accessing wet-lab experimental setups. Moving forward, we aim to mitigate this 535 limitation by fostering partnerships with domain experts, which will enable us to apply our method 536 to more challenging and diverse problems. This direction not only promises to validate the efficacy 537 of our approach in more complex scenarios but also aligns with our commitment to pushing the 538 boundaries of what our technology can achieve. 539

0.0025

0.005

0.01

0.02

0.04

Learning rate

0.985

0.995

1.005

1.015

1.020

Score ratio

Ant

TF

10

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Appendix G.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide our code link in the abstract and detail our settings in Section 4.3. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to our source code in the abstract and thoroughly describe our experimental settings in Section 4.3.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We detail our setting in Section 4.3 and also discuss hyperparameter sensitivity in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: To ensure reliability and consistency in our comparative analysis, each experimental setting was replicated across 8 independent runs, unless stated otherwise, with the presentation of both mean values and standard errors.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have discussed these in Section 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We preserve anonymity and conform with the NeurIPS Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss both potential positive and negative impacts in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any datasets nor pre-trained models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have duly credited all utilized assets and adhered to their respective licenses and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We plan to open-source our code and have ensured thorough documentation of the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not engage in crowdsourcing or involve studies with human participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not engage in crowdsourcing or research involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposed RGD, a novel method for integrating classifier guidance into classifier-free guidance diffusion models for solving offline MBO problems. Experiment results and ablation studies validate that the method outperforms state-of-the-art baselines and each proposed component is resonable.

STRENGTHS:
- Idea is intuitive and easy to follow

- Motivating example in the introduction makes the reader easy to understand the limitations of the prior method and the advantages of the proposed method

- Strong experiment results and detailed ablation studies make the proposed method more convincing

WEAKNESSES:
- For the diffusion-based proxy refinement part, it seems that there are several estimations to compute the distance between $p_{\phi}(y\vert \hat{x})$ and $p_{\theta}(y\vert \hat{x})$. Furthermore, it incurs additional hyperparameter $\alpha$, which should be carefully tuned.

QUESTIONS:
- For the diffusion-based proxy refinement part, have the authors tried other approaches for estimating $p(y)$? I wonder if the choice of estimation affects the performance significantly. 

- To compute the regularization loss term in Eq (15), we need to collect samples from the adversarial distribution. I cannot find the detailed procedure for collecting adversarial samples ($M$, $\eta$, ...). Could authors elaborate more on that part?

LIMITATIONS:
There are a few minor comments on the manuscript.

- For figure 2, it seems that $\tilde{s}(x_T, y, \omega)$ should be written as $\tilde{s}(x_T, y, \hat{\omega})$. Furthermore, at first, it makes me confusion that RGD conducts classifer-guidance. However, that misleading part has been resolved after reading the manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper,  the authors proposed to combine both classifier guidance and classifier-free guidance for offline black-box optimization.  In addition, the authors propose a Proxy Refinement procedure by minimizing KL divergence between the Proxy distribution and diffusion distribution regarding $y$.\n\nSTRENGTHS:\n1.  The paper is well-written and well-organized. \n\n\n\n2.  The paper introduces several refinement procedures to boost the offline optimization performance.  The proposed Diffusion-based Proxy Refinement procedure is interesting.\n\nWEAKNESSES:\n1.  **Technical contribution seems to be incremental**\n\nEmploying diffusion models for offline black-box optimization is not new.  The technical contribution of this paper seems to be incremental. The draft extends the paper \"Diffusion Models for Black-Box Optimization\" [1]. However,   detailed discussions about the relationship between the proposed method and the paper [1] are missing.\n\n[1] Siddarth Krishnamoorthy, Satvik Mashkaria, and Aditya Grover. \"Diffusion Models for Black-Box Optimization.\" ICML 2023. \n\n\n2.  **Part of the technical details are not clear.**\n\n(a) In Equation (12),  the concrete computation procedure of $p_\\theta (\\hat{\\boldsymbol{x}} | y)$ and $p_\\theta (\\hat{\\boldsymbol{x}})$  via diffusion model is not clear. \n\n(b) The derivation of Equation (10) is not given.   It seems that Equation (10) is from the forward pass of the diffusion model.  However,  the forward pass  (Eq.32-32 in [10]) is regarding the distribution.  And the concrete  $\\boldsymbol{x} _ t $ is constructed via the backward pass with  $s_\\theta(\\boldsymbol{x}_k,k)$ for $k \\in T,\\cdots, t+1$. \n In addition,  how to choose $\\mu(t)$ and $\\sigma(t)$ in Equation (10)  is not clear.  \n\n3.  **The additional proxy training, sample refinement procedure and proxy refinement procedure  increase the computation cost**\n\nThe additional proxy training, sample refinement procedure and proxy refinement procedure increase the computation cost. However, the time comparison with baselines is missing. \n\n4.  **The additional proxy training, sample refinement procedure and proxy refinement procedure bring many additional hyperparameters, which may overfit the offline BBO task**\n\nIn the offline BBO tasks,  the offline dataset is provided.  The evaluation is the black-box function value at the generated query at one time. \nThe long-term convergence properties and exploration/exploitation balance are not considered.  As a result, there are risks that overfit the evaluation metric for the offline tasks.  The paper Introduces lots of additional hyperparameters, which increases the overfitting risks.\n\nQUESTIONS:\n1. Please provide more discussion about the technical details in Weakness 2.\n\n\n2.  What is the computation time of the proposed method? Please provide time comparisons with baselines.\n\n\n3. Please provide explanations about the overfiting issue.\n\nLIMITATIONS:\nAdditional computation cost and overfitting risk may be additional limitations besides the limitations discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework called Robust Guided Diffusion for the problem of Offline Black-box Optimization. The key idea is to formulate the solution as conditional generation of high-performance designs using a diffusion model which has explicit guidance from a proxy (surrogate) model. This proxy model is also refined/updated via a proxy-free diffusion procedure. Experimental analysis is shown on multiple tasks from design-bench benchmark.\n\nSTRENGTHS:\n- Overall, I like the paper because it includes two simple changes to an existing approach (DDOM) that shows improved performance and the changes are validated by ablation choices.\n\nWEAKNESSES:\n- One major premise (repeated multiple times in the paper) in the paper is that proxy guidance conditional generation is more robust than updating the design with standard gradient ascent on the proxy. However, it is not immediately clear why this should be true and the justification for this key point is somewhat limited. If true, this will be much bigger insight going beyond black-box optimization. If it is only about the exploration/exploitation balance driven by w, we could also make standard gradient have this property by optimizing a upper/lower confidence bound on the objective. Please describe why this is the case either via some empirical experiment or theoretical insight. Also, in equation 11, we might evaluate the proxy far away from the training data depending on the values of s_\\theta(x_t), \\sigma(t), \\mu(t).\n\n- The related work coverage and corresponding experimental analysis of the paper can be improved. This problem has seen an extensive body of work recently. Please see the references below and discuss/compare them appropriately. Some of them are included in references but not compared in the experiments ([1], [2], [3]):\n\n- [1] Yuan, Ye, et al. \"Importance-aware co-teaching for offline model-based optimization.\" Advances in Neural Information Processing Systems 36 (2023).\n- [2] Kim, Minsu, et al. \"Bootstrapped training of score-conditioned generator for offline design of biological sequences.\" Advances in Neural Information Processing Systems 36 (2023).\n- [3] Nguyen, Tung, Sudhanshu Agrawal, and Aditya Grover. \"ExPT: Synthetic pretraining for few-shot experimental design.\" Advances in Neural Information Processing Systems 36 (2023).\n- [4] Chemingui, Yassine, et al. \"Offline model-based optimization via policy-guided gradient search.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 38. No. 10. 2024.\n- [5] Yao, Michael S., et al. \"Generative Adversarial Bayesian Optimization for Surrogate Objectives.\" arXiv preprint arXiv:2402.06532 (2024).\n\nQUESTIONS:\nSome of the tasks in design-bench benchmark has errors which makes them not so informative for evaluation. For example, the offline dataset in superconductor task has multiple copies of the same inputs but with different outputs. As a result, the random forest oracle which is fit on this offline data is not reliable. It is mentioned that \"This issue has now been rectified by the development team.\" How is it fixed?\n\nLIMITATIONS:\nPlease see weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a robust guided diffusion framework for offline black-box optimization, combining proxy and proxy-free diffusion for conditional generation. Key improvements include proxy-enhanced sampling and diffusion-based proxy refinement to address out-of-distribution issues. Experiments on the Design-Bench benchmark show the method outperforms existing techniques, validated by ablation studies.\n\nSTRENGTHS:\n- The regularization of the proxy using the diffusion model is interesting. Additionally, optimizing the alpha parameter in an offline manner aligns well with the offline setup, enhancing the method's consistency and applicability.\n- Experiments and ablations on four continuous and three discrete tasks validate the effectiveness of the proposed RGD method, showing improved performance and robustness.\n\nWEAKNESSES:\n- The paper lacks comparison with relevant approaches like ICT [1] and TRI-mentoring [2]. Despite referencing the latter in the related work section, it’s overlooked in the results.\n- It is unclear why the results without proxy-enhanced sampling still achieve competitive outcomes, surpassing the dataset y_max. This contradicts the claims in lines 40-46. Where does the out-of-distribution (OOD) problem arise then? What is the distribution of the generated 128 candidates with and without the sampling? \n- The BDI reported results are significantly lower than in the original paper, especially for the ANT and TFBIND8 tasks. This also seems to be the case for BONET results. Did the authors change the evaluation setup?\n\n\n[1]: Importance-aware Co-teaching for Offline Model-based Optimization, https://arxiv.org/abs/2309.11600\n\n[2]: Parallel-mentoring for Offline Model-based Optimization, https://arxiv.org/abs/2309.11592\n\nQUESTIONS:\n- How are the initial 128 designs selected? Do you generate N designs and then use the proxy to select 128?\n- Are the diffusion parameters like T the same for DDOM?\n- Can you show, for a task, how the values (proxy/oracle) of the generated designs progress throughout the diffusion steps?\n- How were the discrete tasks handled? Were logits used, or were they kept discrete?\n- Previous methods typically evaluate on the Hopper task; why was it removed and Rosenbrock is added instead?\n\nLIMITATIONS:\nThe authors address the limitations and potential negative impacts in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method, named RGD, for Offline Black-box Optimization (BBO). RGD incorporates an improved proxy to guide the previous proxy-free method (i.e. DDOM[4]). Key technical innovations includes (1) improving the robustness of the proxy function against adversarial samples by consistency regularization with the diffusion process; (2) dynamic per-sample reweighting between proxy-guided and proxy-free sampling. Compared to previous approaches, RGD demonstrates superior performance on Design-Bench [3].\n\nSTRENGTHS:\nMethodology: RGD integrates forward and reverse approaches for BBO, in a way that they can help with each other (e.g. using forward proxy to guide the reverse sampling and using the diffusion process to improve the forward proxy), which is technically sound and interesting.\n\nExperiment: RGD demonstrates superior performance on Design-Bench, compared to the baselines.\n\nAblation: Ablations on different components of RGD are provided.\n\nWEAKNESSES:\nThe reviewer would prefer some clarifications on the method and the experiments\n\ni) Algorithm 1, Line 4, how to identify the adversarial examples? From Line 187-188, it looks like gradient ascent is utilized to find the x that maximize y, it is unclear to the reviewer that how to determine if the obtained x is an adversarial example\n\nii) Algorithm 1, Line 7, refine the proxy function via eq 15. It would be best if the author could provide further details on how to optimize eq (15), e.g. number of validation and adversarial samples, number of iterations for the bi-level optimization discussed in Appendix B.\n\niii) Algorithm 1 Line 13, optimizing \\omega. Again, it would be best if the author could provide extra info on how to optimize \\omega. From Algorithm 1, it looks like \\omega is time dependent and optimized for each time step. How many training iterations are required for each time step. The reviewer also wonder if the obtained \\omega are dramatically different between different time steps. \n\niv) From Line 257-258, it looks like the baselines shown in Table 1 & 2 were re-implemented. If this is the case, the authors are encouraged to include more implementation details, e.g. the model architecture for the score function, etc. This could help follow-up works to reproduce the reported results. The reviewer also wonders if the source code will be made public.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nLimitations have been discussed in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}