Exploring validation metrics for offline model-based optimisation with diffusion models
Authors: Christopher Beckham
Alexandre Piché
David Vazquez
Christopher Pal
Venue: Accepted by TMLR ()

Abstract: In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle, which is expensive to compute since it involves executing a real world process. In offline MBO we wish to do so without assuming access to such an oracle during training or validation, with makes evaluation non-straightforward. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. Measuring the mean reward of generated candidates over this approximation is one such `validation metric', whereas we are interested in a more fundamental question which is finding which validation metrics correlate the most with the ground truth. This involves proposing validation metrics and quantifying them over many datasets for which the ground truth is known, for instance simulated environments. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation, which is the ultimate goal behind leveraging generative models for MBO. While our evaluation framework is model agnostic we specifically evaluate denoising diffusion models due to their state-of-the-art performance, as well as derive interesting insights such as ranking the most effective validation metrics as well as discussing important hyperparameters.

Published in Transations on Machine Learning Research (05/2024)
Exploring validation metrics for offline model-based optimi- sation with diffusion models
Christopher Beckham
first.last@mill.. quebec
Mila, Polytechnique Montréal, ServiceNow Research!
Alexandre Piché
first. @wwroseenow..oooomooo
Mila, Université de Montréal, NorviceNow Research
David Vazquez
first. wissllass@@eerriceeou..o
ServiceNow Research
Christopher Pal
first. fast.last ca
Mila, Polytechnique Montréal, ServiceNow Research, CIFAR Al Chair
Reviewed OpenReview: https:// openreveee..........m.eu net/ ttrfoomm? id= #CC2II0000
Abstract
In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to black box function called the (ground truth) oracle, which is expensive to compute since it involves executing a real world process. In offiine MBO we wish to do so without assuming access to such an oracle during training or validation, with makes evaluation non-straightforrwad While an approximation to the ground oracle can be trained and used in place model validation to measure the mean reward over generated candidates, the evaluation approximate and vulnerable adversarial examples. Measuring the mean reward of generated candidates over this approximation is one such 'validation metric', whereas are interested in more fundamental question which is finding which validation metrics correlate the most with the ground truth. This involves proposing validation metrics and quantifying them over many datasets for which the ground truth is known, for instance simulated onvironments. This is encapsulated under our proposed evaluation framework which also designed to measure extrapolation, which is the ultimate goal behind leveraging generative models for MBO. While our evaluation framework is model agnostic we specifically evaluate denoising diffusion models due their state-of-the-art performance, as well as derive interesting insights such as ranking the most effective validation metrics as well as discussing important hyperparameters.
during
1 Introduction
In model-based optimisation (MBO), we wish to learn a model of some unknown objective function f X' y where f is the ground truth 'oracle', æ E X' is some characterisation an input and y E & the reward. The larger the reward is, the more desirable is. practice, such a function (a real world process) is often prohibitively expensive compute because involves executing a real-worh! process. For instance if x X is a specification of protein and the reward function is its potency regarding a particular target in a cell, then synthesising and testing the protein amounts to laborious work in wet lab. In other cases, synthesising and testing candidate may also be dangerous, for instance components for vehicles or aircraft. In MBO, we want to learn models that can extrapolate that is, generate inputs whose rewards are beyond that of what we have seen in our dataset. However, we also need ot be rigorous in how we evaluate our models since generating the wrong designs can come at a time, safety, or financial cost.
Work done while author was interning at ServiceNow Research.

---

Published in Transations on Machine Learning Research (05/2024)
Model evaluation in MBO is not straightforward. Firstly, it does not adhere to a typical train/valid/ pipeline that one would expect in other problems. Secondly, it involves evaluating samples that are not from the samne distribution as the training set (after all, we want to extrapolate beyond the training set). To understand these difficulties more preciscly, we give a quick refresher on a typical training and evaluation pipeline for generative models. we assume the setting of empirical risk minimisation [Vapnik 1991 then the goal is to find parameters 0* which minimise some training loss on the training set Derain:
0- = arg min ((Drraini 0) Em.gy-aanie l(x, Vi 0),
(1)
for any model of interest that is parameterised by 0, e.g. generative model po(x). Since we do not wish to overfit the training set, model selection is performed on a validation set Dvalid and we can write a variant of Equation where the actual model we wish to retain is the following:
0+ arg min MM vallid: E Dr..y-Daani m( y; 0), 8ee
where M is a validation metric and θ = [0] }- comprises a collection of models 0j. i.e. each of them is the result of optimising Equation ! under a different hyperparameter configuration or seed. Whatever the
best model is according to Equation| 2 is finally evaluated on the test set Drest as an unbiased mcasure of performance. Since Drest already comes with labels from the ground truth, it simply sulfices to just evaluate Equation 2 and report the result. However, in MBO we wish to generate new examples (in particular ones with high reward), which is akin to generating true values of (the reward) of its examples unless we evaluate the ground truth oracle f on each generated example, which is very expensive. Secondly, the synthetic test set that we have generated is not intended to be from the same distribution as the training set, since the goal of MBO is to extrapolate and generate examples conditioned on larger rewards than what was observed in the original dataset. This means that the validation set in Equation 2 should not be assumed to be the same distribution as the training set, and evaluation should reflect this muance.
own 'synthetic' test set. However, we don't know the
To address the first issue, the most reliable thing to do is to simply evaluate the ground truth our synthetic test set, but this is extremely expensive since each 'evaluation' of the ground truth f means executing a real world process (i.c. synthesising protcin). Furthermore, since we focus on offline MBO we cannot assume an active learning setting during training like Bayesian optimisation where we can construct a feedback loop between the oracle and the training algorithm. Alternatively, we could simply substitute the ground truth with an approximate oracle f(x), but it isn't clear how reliable this due to the issue of adversarial examples (see Paragraph 2. Section 1]. For instance, ff could overscore examples in our generated test set and lead
to believe our generated exmples are much better than what they actually are.
Although this is an unavoidable issue in offline MBO, we can still try to alleviate it by finding validation metrics (i.c. M in Equation 2 which correlate well with the ground truth over a range of datasets where it is known and cheap to evaluate, for instance simulated environments. If we can do this over such datasets then we can those empirical findings to help determine what validation metric we should use for a real world oflline problem, where the ground truth is not easily accessible (see Figure []. To address the second issue, intentionally construct our train/valid, /test pipeline such that the validation set is designed to contain examples with larger reward than the training set, and this sets our work apart from existing literature examining generative models in MBO.
We note that this very similar to empirical research in reinforcement learning, where agents are evaluated under simulated environments for which the reward function known and can be computed in silico. However, these environments are ultimately there to inform much grander goal, which to have those agents operate safely and reliably in the real world under real world reward functions.
2

---

Published in Transations on Machine Learning Research (05/2024)
MBO tasks where the ground truth oracle is known (this paper)
MBO tasks where the ground truth oracle is not known
Drrain
Dallid
f(x)
Drrain
D valid
trann
validate
knowlodge transfer
pa ( 9)
metric
po I, y)
correlation
real world generation
Figure 1: We want to produce designs æ that have high reward according to the ground truth oracle y = f(x), but this is usually prohibitively expensive to compute since it involves executing real-world process. If we instead considered datasets where the ground truth oracle is cheap to compute (for instance simulations), we can search for coooptt validation metrics that correlate well with the ground truth. In principle, this can facilitate faster and more economical generation of novel designs for real-world tasks where the ground truth oracle expensive to compute.
1.1 Contributions
Based on these issues propose training and evaluation framework which is amenable to finding good validation metrics that correlate well with the ground truth, In addition, we also assume that the training and validation sets are not from the same ground truth distribution. We lay out our contributions as follows
We propose a conceptual evaluation framework for generative models in offline MBO, where we would like to find validation metries that correlate well with the ground truth oracle. We assume these validation metrics are cheap to compute. While computing said correlations requires the use of datasets where the ground truth oracle can be evaluated cheaply (i.c. simulations), they can still be useful inform more real world MBO tasks where the ground truth is expensive to evaluate. In that case, finding good validation metrics potentially provide large economic savings.
While our proposed evaluation framework is agnostic to the class of generative model, specifically demonstrate it using the recently-proposed class of denoising diffusson probabilistic models (DDPMs) Ho et al. 2020 For this class model, we examine two conditional variants: classifier- based (Dhaiiwll & Nichol 2021 | and classifier-free (Ho & Salimans 2022) guidance. Since DDPMs appear to be relatively unexplored MBO, we consider our empirical results on these class of models be an orthogonal contribution of our work.
We explore five validation metrics in our work against four datasets in the Design Bench Trabucco et al, 2022) framework, motivating their use as well as describing their advantages and disadvantages. We run a large scale study over different heprpaarameters and rank these validation metrics by their correlation with the ground truth.
Lastly, we derive some additional insights such as which hyperparamcters are most important tune. For instance, we found that the classifier guidance term is extremely important, which appears underscore the trade-off between sample quality and sample diversity, which is commonly discussed dilemma in generative modelling. We also contribute some thoughts on how diffusion models in offline MBO can be bridged with online MBO.
CCorresponding code he found here: http://///.teeoooeeteteeeeeeeeee

---

Published in Transations on Machine Learning Research (05/2024)
2 Motivation and proposed framework
We specifically consider the subfield of MBO that is data-driven (leverages machine Icarning) and is offline. Unlike online, the oflline case doesn't assume an active learning loop where the ground truth can periodically be queried for more labels. Since we only deal with this particular instantiation of MBO, for the remainder of this paper we will simply say MBO instead of offtnne data-driven MBO. In MBO, very simple approach to generation is to approximate the ground truth oracle f by training an approximate oracle fo(x) a regression model - from some dataset D, and exploiting it through gradient ascent to generate a high reward candidates:
arg max f(x maggax//a fo
(3)
x
which can be approximated by iteratively running gradient ascent on the learned regression model for E TT)...,T):
1c+1 f 24 vo
(4)
and To is sampled from some prior distribution. The issue here however is that for most problems, this will produce an input that is either invalid (e.g. not possible to synthesise) or is poor yet receives a large reward from the approximate oracle (overestimation). This is the case when the space of valid inputs lies on a low-dimensional manifold in a much higher dimension space (Kumar & Levine 2020). How these problems are mitigated depends on whether one approaches MBO from a discriminative modelling point of view (Fu &
Levine 2021 Trabucco et al. 2021 Chen et al. 2022a), or a generative modelling one (Brookes et al. 2019
Fannjiang Listgarten 2020 Kumar& & Levine 2020). For instance, Equation
3 implies a discriminative
approach where fo( is a regression model. However, in Equation 4 it. is reinterpreted as an energy model Welling & Teh 20111. While this complicates the distinction between discriminative and generative, we refer to the generative approach as one where it is clear that joint distribution po(x, y) is being Icarned. For instance, if we assume that (0(2, v) (y(y)z))p((( (x), then the former is a probabilistic form of the regression model and the latter models the likelihood, and modelling some notion of it will almost certainly mitigate the adversarial example issue since it is modelling the prior probability of observing such an input. Because of this, we argue that a generative view of MBO is more appropriate and we will use its associated statistical language for the remainder of this paper.
Let us assume have trained conditional generative model of the form po(x))) on our training set Dirain. If Pirain (3) denotes the empirical distribution over the y's in the training set, then this can be used to write the joint distribution as po(a. y) = po (a P)Prrain(U) We do not wish to sample from this joint distribution however, because we ultimately want to generate x's 's whose y's are as large as possible. To do this we could switch out the prior (t) for one that reflects the range of values we wish to sample from. However, it wouldn't be clear if the model has generalised in this regime of y's; for instance the sampled I's may be invalid. What that are able to extrapolate; in other words, models which assign small loss to exaiples that have larger y 's than those observed in the lraining sel. As an example, one such appropriate loss could be the negative log likelihood.
want to do is be able to measure and select for models (i.c. po(xly) for some good 0)
We can mcasure this through careful construction of our training and validation sets without having to leave the offline setting. Assume the full dataset D {(,; y: ) and (æ, ~ p(x,,))) the ground truth joint distribution. Given some threshold 7, we can imagine dealing with two truncated forms of the ground truth po,, (x, y) and (z v), where:
Po. (x. v) {(x,v) ~ p (w, E [0, 71)
(x, = {(xx v) ~ p( pl.p.()lll))))
v)ly E (7. oo]),
(5)
where Py is the distribution of samples which are not seen during training but nonetheless we would like our generative model to explain well. Therefore, if we split D based on y then we can think of the left split Dirain as finite collection of samples from p., (ax, y) and the right split Dvalid from (æ, We would like to train models on D Derain and maximise some measure of desirability on Deatia (via hyperparameter tuning) to find the best model.
4

---

Published in Transations on Machine Learning Research (05/2024)
D Dvalid
fxx)
D train
fo(x))
metric
(())
train
validation
scoring
po @ y)
po,y P0,y((c, (C, y
extrapolate
generate
S
Algorithm 1
Algorithm 2
Figure 2: A visualisation of our evaluation framework. Here, assume joint generative models of the form po(ar y). Models are trained on Dirain as per Section 2.1 and in this paper assume the use of conditional denoising diffusion probabilistic models (DDPMs). For this class of model the joint distribution po Pa( y) decomposes into pa (z)))))p(y) and the way we condition the model on is described in Section 2.1.1 In order to generate samples conditioned on rewards larger than what was observed in the training set, we must switch the prior distribution of the model, which corresponds to 'extrapolating' it and described Section 2.2 Validation done periodically during training and the best weights are saved for each validation metric considered. The precise details of this are described in Algorithm [ When the best models have been found perform final evaluation the real ground truth oracle, and this process is described in Algorithm 2
2.1 Training and generation
While our proposed evaluation framework is agnostic the class of generative model used, in this paper we specifically focus on denoising diffuusin probabilistic models (DDPMs) (Sohl-Dickstein al. 2015 Ho et al. 2020). DDPMs are currently state-of-the-art in many generative tasks and do not suffer from issues exhibited from here classes of model. They can be seen as multi-latent generalisat ceneralisations of variational autoencoders, and just like VAËs they optimise a variational bound on the negative log likelihood of the data First, let us consider an unconditional generative model pa (x), whose variational bound is:
log pla (2))
B(mo log (xo) s SE (p(
(6)
pe po ... DT)
p(xT)
Pe (pt (xt) log
E (T)
log p(xT (D)
log (a 2
(7)
p x I,, 20
Lo
Here, No ~ p((o)) is the real data (we Do instead of to be consistent with DDPM notation, but they are the samc), and p(x,/_x,1) for t E (1...., T} is a predefined Gaussian distribution which samples noisy which contains more noise than æ;-1, i.e. as gets larger the original data zo progressively becomes more noised. These conditional distributions collectively define the forward process. The full joint distribution of this forward process is
T p( (x: t=1
.0.....,.77... FT) -((x)
II
8
define the real distribution, but here use be consistent with earlier notation, though it not be confused with pe, the learned distribution.
2Typical DDPM literature
Note we have still used orange denote that p(#o) the ground truth data distribution, but we have omitted for p(ay(ate11 make it clear that these are predefined noise distribuutons.

---

Published in Transations on Machine Learning Research (05/2024)
and each conditional noising distribution has the form:
p (2;-1) = Na v 3,0 3,1),
(9)
where Br follows predefined noise schedule. Since the product of Gaussians are also Gaussian, can also define the t-step noising distribution p(a, æu) which is more computationally efficient:
p(a, mo) Nixx; v Mapa + (1 d
(10)
Va + v(1 - (1)e,,
((O
(11)
where at 1 Br and at II ag. Generally speaking, we wish to learn a neural network (x to undo noise in the forward process, and these learned conditionals comprise a joint distribution for the reverse process:
po po (z:) - (ax- po (); 32-11).
(12)
where pa is expressed via a neural network μ4 which learns to predict the mean of the conditional distribution for æt-1 given Hi, and we wish to find parameters to minimise the expected value of Equation @ over the entire dataset. practice however, if the conditionals for po and p are assumed to be Gaussian, one can dramatically simplify Equation 0 and re-write Le (for any t) as noise prediction task. where instead we parameterise neural network e ca( () to predict the noise from at ~ p( as shown in Equation [1 via the reparamterisationn trick. Note that cø(x;, t) and po( t) differ by closed form expression (and likewise with po (z) but for brevity's sake we defer those details to Ho et al. (2020) and simply state the final loss function to be:
-- [lle co (Varso VI dge, ()13),
(13)
where v 00 VI- Orer is simply just writing out æ; p( via the reparamterisation trick (Equation [1,, and {ar)E= defines a noising schedule. Since we have defined our training set Durain to be samples from po,y (x,, y): our training loss is simply:
min LDSM (0) min Bao~D [llex co(vōnzo + - 0)
(14)
In order to generate samples, stochastic Langevin dynamics (SGLD) is used in conjunction with the noise predictor Eg to construct a Markov chain that by initialising øT ~ p(xT) = N I) and running the following equation for E - 1)....0)
It-1
2t
Ote
ar (o + 9iz, ~001).
(15)
VI
Note that while the actual neural network that is trained is a noise predictor co(2z, t) which in turn has a mathematical relationship with the reverse conditional po(2;-1laz) we will generally refer to the diffusion model as simply (=) throughout the paper, since the use of Equation 15 implies a sample from the distribution (xo), and we have already defined = Tn.
2.1.1 Conditioning
Classifier-based guidance Note that Equation 14 defines an unconditional model (=)).. In order to be able to condition on the reward y, we can consider two options The first is classifier-based guidance Dhhriwwl & Nichol 2021). Here, we train an unconditional diffusion model po(x), but during generation we define a conditional noise predictor which leverages pre-trained classifier po po(y)æ,) which predicts the reward
from Zy;
cø (a t. v) = cØ (Z) - vI a Va, log Pe (x)))
(16)
classifer-based guidance
WWile possible derive conditional ELBO from which DDPM can be derived from, the most popular method for conditioning DDPMs appears be via guiding an unconditional model instead.
6

---

Published in Transations on Machine Learning Research (05/2024)
where (y(()x) is also trained on Dirain and R+ is a hyperparameter which balances sample diversity and sample quality. Equation 16 is then plugged into the SGLD algorithm (Equation 15 to produce sample from the conditional distribution pe( Note that since pa(y| (x; is meant to condition on D for any t, it idcally requires a similar training setup to that of the diffusion model, i.c. sample different x 's via Equation Irain the network to predict y.
Classificr-free guidance In classifier-free guidance (Ho & Salimans, 2022); the noise prodictor is re- parameterised to support conditioning on label y, but it is stochastically 'doppedd' during training with some probability in which case y is replaced with mull token:
(O:DsM(0:T) -e
|||e- -ee(Veea vI are, (xer (v),
(17)
where 1xer(M) (v) is the indicator function and returns null token if X < otherwise is returned. The additional significance of this is that at generation time, one can choose various tradeoffs of (conditional) sample quality and diversity by using the following noise predictor, for some hyperparameter w:
6 a1 v, = (w + 1)co (, w t)
(18)
classifier-free guidance
2.2 Extrapolation
Given one of the two conditioning variants in Section 2.1.1 we can denote our generative model as (ally) which has been trained on (2, pairs from Derain- If we denote the distribution over y in the training set as (v) (v) then through Bayes' rule we can write the joint likelihood as: po( y) ps (a) (t), This equation essentially says: to generate a sample (æ. y) from the joint distribution defined by the model, we first sample y ~ (v)) then we sample æ ~ po pa y) via SGLD (Equation 15 Samples from (v) can be approximated by sampling from the empirical distribution of y's over the training set. Decomposing the joint distribution into likelihood and a prior over y means that we can change the latter at any time. For instance, if we wanted to construct an 'extrapolated' version of this model, we can simply replace the prior in this equation with (v), which is the prior distribution over for the validation set. We define this as the eztrapolated model (Figure 2 eztrapolate caption)
pa,> (x. y) = Pa (o(a)))),
(19)
and samples py (v) can be approximated by sampling from the empirical distribution over y from the validation set. Of course, is not clear to what extent the extrapolated model would be able to generate high quality inputs from y's much larger than what it has observed during training. This is why we need to perform model selection via the use of some validation metric that characterises the model's ability to extrapolate.
2.3 Model selection
Suppose we trained multiple diffusion models, each model differing by their set of hyperparameters. If we denote the j'th model's weights as fj. then model selection amounts to selecting 0" θ = {0, which minimises some 'goodness of fit' measure computed the held-out validation set Dvalia- One such metric that is fit for a generative model would be the expected log likelihood over samples in the validation sel, assuming it is tractable:
the validation set. We call this a validation metric, which is
8" = arg max DEE
log Pa, (æ.v)
(20)
D
(
where (x, y) = (o(x)))p, (v) is the extrapolated model as originally described in Equation 19 have already established that the validation set comprises candidates that are higher scoring than the training set, this can be thought of as selecting for models which are able to ezplain (i.e. assign high conditional
Since we
There are other ways to infer an extrapolated model from 'base' model, and describe some of these approaches in Section g
7

---

Published in Transations on Machine Learning Research (05/2024)
likelihood) to those samples. However, cannot compute the conditional likelihood since it isn't tractable, and nor can we use its evidence lower bound (ELBO). This is because both conditional diffusion variants (Section 2.1.1 assume a model derived from the unconditional ELBO which is a bound on pe(x), rather than the conditional ELBO is correspondingly a bound on po(= Despite this, if we are training the classificr- diffusion variant, we can simply use Equation 17 as a proxy for it. but computed over the validation set:
Mc.DsM(Pvalite 0) Dvalid
Σ
||er - co ys 1)|
(21)
(Ma-3))~Doule)
where æt ~ p( Equation 21 can be thought, of selecting for models which are able to perform a good job of conditionally denoising examples from the validation set. While Equation 21 seems reasonable, arguably it is not directly mcasuring what we truly want, which is a model that can generate candidates that are high rewarding as possible. Therefore, we should also devise a validation metric which favours models that, when 'extrapolated" during generation time (i.e. we sample æ y where y's are drawn from py))), are likely under an approximation of the ground truth oracle. This approximate oracle is called the validation oracle fø, and is trained both Dirain U Dvalid (see Figure 2):
0* arg max (e.yy-o.., ( = arg max (al=
(22)
lEe
dge
We consider biased variant of Equation 22 where the expectation is computed over the best 128 samples generated by the model, to be consistent with Trabucco el. al. (2022). This can be written as the following equation:
Mreward (5;0,0) (S000c((S;0.0) = K Σ fo (o)))
1
S, ~ (a, y)
(23)
where sorted(S, fo) sorts S } in descending order via the magnitude of prediction, and [S] >> K.
Lastly, there are two additional validation metrics we propose, and these have been commonly used for adversarial-basee generative models (which do not permit likelihood evaluation). For the sake of space we defer the reader lo Section A.1 and simply summarise them below:
Fréchet distance (Heusel et al. 20170 (Equation S28 given samples from the validation set and samples from the extrapolated model, fit multivariate Gaussians to both their embeddings (a hidden layer in fa is used for this) and measure the distance between them. This can be thought of as a likelihood way to measure the distance between two distributions. We denote this as MPD
Naeem et al. (2020) proposed 'density' and 'coverage' as improved versions of precision and recall, respectively (O'Donoghue et al. 2020; Kynkäänniemi et al., 2019). In the generative modelling literature, precision and recall measure both sample quality and mode coverage, i.e. the extent to which the generative model can explain samples from the data distribution. We denote this as MnC, which is a simple sum over the density and coverage metric.
Density and coverage (Equation 532
We summarise all validation metrics well as detail their pros and cons - in Table 1 We also summarise all preceding subsections in Algorithm [ which constitutes our evaluation framework.
2.3.1 Final evaluation
We may now finally address the core question presented in this paper: what validation metrics are best correlated with the ground truth? Since we have already defined varions validation metrics in Section 2.3 the only thing left is to define an additional metric - a test metric which a function of the ground truth oracle. We simply define this to be an unbiased estimate of Equation 23 which uses the ground truth oracle f to rank the top 128 candidates:
K
1
Mrest-roward (1s00.)
/ (sorted(S:Ia)))
S ~ (xr,(
(24)
K
8

---

Published in Transations on Machine Learning Research (05/2024)
Algorithm Training algorithm, with early implicit early stopping, For m predefined validation metrics, the best weights for each along with their values and stored and returned. (For consistency of notation, each validation metric of the form M, 0) (); though the true arguments that are taken may be a subset. of these. See Table
Require:
Threshold 7, s.t. Dusin ~ (o,,((; y), D Dvalid ~ (x,
Equs.
Number of training epochs Repochs; validation rate neval
Validation metrics (functions): {MaJy=1 Mj has arguments M,(D,D,0,0)
h E H hyperparameters used to initialise Validation oracle fø(x)
pre-trained on train + val Dirain U Dvaald set
1: Xtrain Yrain t- Dirainy Xvalid, Yvalid +- Dvalid
2: 0 + initialise(1))
3: best _eights + [0]
> store best weights so far for each validation metric & store best (smallest) values seen per metric
best _etrics < f00 j=1
for epoch in {1..... Nepochs) do sample (ar train
0 - 0 - (Z) y)
C: eqn. 14
with either equ. 16 or equ.
18
8:
if epoch % neval = O then
9:
& Model selection (Sec. 2.3
10:
D - ((o;, ) where T; pa (=)l)) and yı (Yvalid);
sample using eqn.
15
11:
Evaluate validation metrics (Table
12:
for j in {1....,,m m) do mj + -M,(Dvvvial D,
13:
14:
if mj < best metrics, then best coe__metrics f- my
15:
found new best metric value for metric j save new weights for metric j
16:
best beig_we +
17:
end if 18: end for
19: end if
20:
end for
21: return best weights, best. metrics
Algorithm Final evaluation algorithm. As per Algorithm I each validation metric Mj is associated with the best weights 02 found through hyperparameter tuning and early stopping. For each fg we generate a candidate set S; of examples, retain the K best candidates as per the predicted reward from validation oracle, and then compute the real reward on the ground truth oracle.
Require:
best weights = {0,} test res_rdewrrds + fm for in (1,..., m) do 0 < best weights,
assumed to be the best weights found for each validation metrie Mj
Load in best weights for metric j
S (a whore E. po and yi ~ Yvalid valid rewards (S) L.W.
b predict reward for each generated candidate get top K examples wrt to fg predictions b get unbiased estimate of test rewards
argsort validd scores 1...,K K
7: test_ rewardes + (f (())
8: end for
9:
return test. rewards
This the test metric which used determine how well correlated our validation metrics are with the ground truth. For instance, suppose we have trained many different generative models via Algorithm D each model corresponding to different set of hyperparameters if Algorithm 2 each of these

---

Published in Transations on Machine Learning Research (05/2024)
models then we can compute quantitative metrics such correlation between best metrics (Alg. 1 and
host_rewaards (Alg- 2 and indeed this is what we will be demonstrating in Section 4
While it seems reasonable to assume that Equation 24 would be most correlated with its validation set equivalent (Equation 28 this would only be the case in the limit a sufficient number of examples to be used to train the validation oracle fa- Otherwise, the less data that used the more it will be prone overscoring 'adversarial' examples produced by the generative model. Therefore, it may be wise to consider validation metrics which put less emphasis on the magnitude of predictions produced by the oracle, e.g. measuring the distance between distributions.
Table 1:
The column 'distance?" asks whether the validation metric is measuring some form of distance between the extrapolated distribution py (x, y) and the extrapolated model pe,, (x) y). t: RKL = reverse KL divergence, see Sec. A.2.2 for proof it an approximation of this divergence; +: FKL = forward KL divergence, since diffusion models optimise an evidence lower bound (Eqn. 6 * distances here are measured not in data space, but in the semantic space defined by one of the hidden layers inside fe-
eqn.
require fo model agnostic?
distance?
MC.DSMI (D (Dvatid;)) 0)
21
x
(approx. FKL);
-rrward (S; 0
23
MARe(Dun (aallaa 0;
26
(approx. RKL) Fréchet
MFD(X Xvalid; X;
;))
$28
-Mpo (valid. x))
S32
/
3
Related work
Design Bench Design Bench is evaluation framework by Trabucco et al. (2022) that facilitates the training and evaluation of MBO algorithms. Design Bench, of time of writing, provides four discrele and four continuous datasets. These datasets can be further categorised based on two attributes: whether a ground truth oracle exists or not, and whether the input distribution is fully observed (i.c. the combined train and test splits contain all possible input combinations). In terms of evaluation, Design Bench does not prescribe a validation set. (only a Training set and test oracle), which we argue important in order to address the core question of our work, which is finding validation metrics that correlate well with the ground truth oracle. While Trabucco al. (2022) docs allude to validation sets in the appendix, these do not convey the same semantic meaning our validation set since theirs is assumed to be subsample of the training set, and therefore come from the training distibution. Lastly, while the same appendix provides examples for different validation metrics per baseline, the overall paper itself is concerned with establishing reliable benchmarks for comparison, rather than comparing validation metrics directly.
Validation metrics To be best of our knowledge, rigorous exploration of validation metrics has not yet been explored in MBO. The choice of validation metric is indeed partly influenced by the generative model, since one can simply assign the validation metric to be the same as the training loss but evaluated on the validation set. For example, if we choose likelihood-based generative models (essentially almost all generative models apart from GANs), then we can simply evaluate the likelihood on the validation set and use that as the validation metric (Equation 20 However, it has been well established that likelihood is a relatively poor measure of sample quality and more biased towards sample coverage (Huszár 2015 Theis et al. 2015 Dosovitskiy & Brox 2016). While GANs have made difficult to evaluate likelihoods they are non hikelihooo--aaeeee generative models it has fortunately given rise to extensive literature proposing 'ikelihood-ffee' evaluation metrics [Borji 2022 and these are extremely useful to explore for this study for two reasons. Firstly, likelihood-free metrics are agnostic to the class of generative model used, and secondly they are able to probe various aspects of generalive models that are hard to capt.rr with just likelihood. As an example, the Fréchet Distance (FID) (Heusel et al. 2017) is commonly used to evaluate the realism of generated samples with respoct to reference distribution, and corrclates well with human judgement of sample quality. Furthermore, metrics based on precision and recall can be used to quantify sample quality and sample coverage, respectively (Sajjadi al. 2018 Kynkäänniemi al. 2019).
10

---

Published in Transations on Machine Learning Research (05/2024)
Pinia
Paak
Doate
triin
train
train
validation
tasd (30%) and walld (50%)
Test dataset)
(a) Design Bench
(b) Ours (case) 1)
(c) Ours (case 2)
Design Bench only prescribes a training split which is determined by threshold y to only filter examples whose y's are less than or equal to this threshold. The full dataset, while technically accessible, is not meant to be accessed for model select ion as per the intended use of the framework. While the training set. could be subsampled to give an 'inner' training set and validation set, the validation set would still come from the same distribution as training, which mcans we cannot effectively measure how well a generative model extrapolates. To address this, we retain the training set but denote everything else (examples whose rewards are > ?) to be the validation set (3b), and the validation oracle fa is trained on Drrain U Dvalid No test set. needs to be created since the ground truth oracle f is the 'test set'. However, if the ground truth oracle does not exist because the MBO dataset is not exact, we need to also prescribe test set
Figure 3: 3a
(3c). Since there is no ground truth oracle f, we must train a 'test oracle' ʃ on Drrain D U (i.e. the full dataset). Note that this remains compatible with the test oracles prescribed by Design Bench, since they are also trained on the full data. Furthermore, our training sets remain identical to theirs.
Use of validation set Compared to other works, the use of a validation set varies and sometimes details surrounding how the data split is opaque. For example, in Kumar & Levine (2020) there is no mention of a training or validation set; rather, we assume that only Dirsin and Diest exists, with the generative model being trained on the former and test oracle on the latter (note that if the test oracle is approximate there is no need for Drest). This also appears to be the case for Fannjiang & Listgarten (2020) While Design Bench was proposed to standardise evaluat ion, its API does not prescribe a validation sel While the training set could in principle be subsetted into smaller training set and validation set (such as in Qi et al. (2022) ), the latter would no longer carry the same semantic meaning as our notion of a validation sct, which is intentionally designed to not be from the same distribution as the training set. Instead, our evaluation framework code accesses the full dataset via an internal method call to Design Bench, and we construct our own validation set from it. We illustrate these differences in Figure 3
Bayesian optimisation Bayesian optimisation (BayesOptt) algorithms are typically used in online MBO, where the online setting permits access to the ground truth during training, as a way generate and label new candidates. These algorithms can be seen as sequential decision making processes in which Bayesian probabilistic model fe(æ) is used to approximate the ground truth oracle (e.g. Gaussian process). Bayesian optimisation alternates between an acquisition function choosing which candidate to sample next and updating fo to reflect the knowledge gained by having labelled æ with ground truth f. However, these algorithms require the specification of a prior over the data which can be cumbersome, and therefore recent works have explored the pre-training of such priors on offline data Wang et al. (2021): Hakhamaneshi et al. (2021) Wistuba & Grabockal (2021).
Learned black box optimisers Black box optimisation comprises a wide range of algorithms, for instance simulated annealing, genetic algorithms, and Bayesian optimisation (Paragraph 3 Typically these algorithms assume a fixed dimensionality for the input space, making it difficult for them to adapt to different input dimensionalitics (for instance among different tasks). Because of this there is interest in learning black box optimisers, with one such instance being Former''r'' Chen et al. (2022b) (a transformer- model). Former is trained on a large corpus of precollected hyperparameter optimisation trajectories for various
6 However, in Trabucco
2022 (their Appendix F) some examples are given to what validation metrics could he used.
11

---

Published in Transations on Machine Learning Research (05/2024)
black box algorithms and tasks (i... different policies). In principle this can also be utilised for MBO but it would require precollected trajectories from existing policies.
While our focus is on offline MBO, in a real world setting MBO is never fully offline; this is because one needs to eventually verify that the candidates generated are actually useful and this can only be done with the ground truth. In other words, offline MBO a way to 'bootstrap' an online MBO pipeline, by leveraging past oracle evaluations to learn a model which can inform future evaluations. One way we can combine both styles is to consider factorised generative models. For instance, in the case of diffusson models (Section 2.1.1 one can factorise joint density ps(x.3) 3) into a prior over the data pa( and a classifier (o(y)z). If we assume that classifiers are casier to update in online fashion, then we can train po(y as Bayesian probabilistic model. At generation time, we can hold the diffusion model fixed and finetune the classifier by performing Bayesian optimisation.
In the more general case however, for generative models that admit latent space encodings of the data, a very Frraightforwart way to leverage knowledge of the former is to train BayesOpt algorithms on those encodings (Maus et al., 2022). This can be especially useful if the latent space is of much smaller dimension than the data, if the data space is discrete.
3.1 Modelling approaches
Model inversion networks The use of generative models for MBO was proposed by Kumar & Levine (2020), under the name model inversion networks (MINs). The name is in reference to the fact that one can learn the inverse of the oracle fe- : + X, which is a generative model. In their work, GANs are chosen for the generative model, whose model we will denote as Go(z. y) that is to say: x ~ po( implics we sample from the prior 2 ~ p(z), then produce a sample a = Go(z.y). At generation time the authors propose the learning of the following prior distribution as a way to extrapolate the generative mode
Pç(2, == arg max Ex ( + ((EN))) (1 log Poglo()))) + c2 l2)gg(())) Pc(z.w)
(25)
where é1 and 62 are hyperparameters that weight the agreement and the prior probabilily of the candidate z. The agreement is measuring the log likelihood of ã = Go(z,)) being classified as under the training oracle fe (expressed probabilistically as (=(v)))), and can be thought of measuring to what extent the classifier and generative model 'agree' that o has a score of y. The log density p(z) can be thought of as a regulariser to ensure that the generated candidate z is likely under the latent distribution of the GAN.
We note that agreement can be easily turned into validation metric by simply substituting pe (a)) for the validation oracle (o(lz)) Note that if we assume that Po(y(z)) is a Gaussian, then the log density of some input y turns into the mean squared error up to some constant terms, so we we can simply write agreement out as (Gog v)) - v This leads us to our second validation metric, which we formally define as:
MAge (Dvalidi 0) |Deattal
lf = where H, po(xly) (D.y))/Donidd
Σ
(26)
We remark that Equation 26 has a mathematical connection to the reverse KL divergence, one of many divergences used to measure the discrepency between generative and ground truth distribution. For inclined readers, we provide derivation expressing this relationship in the appendix, Section A.2.2
Discriminative approaches In the introduction we noted that MBO methods can be seen as approaching the problem from either a discriminative or a generative point of view (though some overlap can also certainly exist between the two). In the former case, regularising the approximate oracle so (x) key, and it is also the model that is sampled from (e.g. as per gradient ascent. in Equation 3 The key idea that the approximate
In Kumar& & Levine 2020 this optimisation is expressed for single (z, v) pair, but here we formalise it as learning a joint distribution over these two variables. If this optimisation is expressed for a minibatch of 12,y),s ss then it can be seen learning an empirical distribution over those variables.
12

---

Published in Transations on Machine Learning Research (05/2024)
oracle should act conservatively pessimistically in out-of-distribution regions. Some examples include
mining for and penalising adversarial examples (Trabueco et al., 2021), encouraging model smoothness Yu
et al. (2021), conservative statistical estimators such as normalised maximum likelihood (Fu & Levine 2021).
learning bidirectional mappings (Chen et al. 2022a) and mitigating domain shift (Qi ct al. 2022
(019)) propose the use of variational inference to learn a sampling distribution pa po(2)5) given a probabilistic form of the oracle po(S( as well as a pre-trained prior distribution over the data pa)))) Here, S' is some desirable range of y's, and therefore pe pa(x) (z] can be thought of as the 'extrapolatod" gencrative model. In Scction 1.2.I we discuss how such a model can be viewed within our evaluation framework. Lastly, |aanjiang & Listgarten (2020) proposes MBO training within the context of a min-max game between the generative model and approximate oracle. Given some target range y S an iterative min-max game is performed where the generative model (x) updates its parameters to maximise the expected conditional probability over samples generated from that range, and the approximate oracle fo( updates its parameters to minimise the error between the generated predictions and that of the ground truth. Since the latter isn't accessible, an approximation of the error is used instead. In relation to our evaluation framework, the extrapolated model would essentially be the final set of weights ((1) for (((x)))(a=m) when the min-max game has reached equilibrium.
Gencrative approaches Brookes ct al.
For both approaches, there is notion of leveraging initial generative model (o and fine-tuning it with the oracle so that it generates higher-scoring samples in regions that it was not initially trained on. Both the min-max and variational inference techniques can be thought of as creating the 'extrapolated" model within the context of our evaluation framework (Figure 2 Therefore, while our evaluation framework does not preclude these more sophisticated techniques, we have chosen to use the simplest extrapolation technique possible which is simply switching out the prior distribution as explained in Section 2.1
Diffusion models Recently, diffusion models Ho et al. 2020 have attracted significant interest due to their competitive performance and ease of training. They are also very closely related to score-based gencrative models (Song & Ermon 2019 2020 In diffusion, the task is to learn a neural network that can denoise any 3t to 1 where ((z0..... 2T) defines a joint distribution over increasingly perturbed versions of the real data q(x)). Assuming that q = p(ær) for some prior distribution over TT, generate a sample Langevin MCMC is used to progressively denoise a prior sample æт into To, and the result is a sample from the distribution po (= [ To the best of our knowledge, we are not aware of any existing works that evaluate diffusion or score-based generative models on MBO datasets provided by Design Bench, and therefore we consider our exploration into diffusion models here as an orthogonal contribution.
4 Experiments and Discussion
Dataset Our codebase is built on top of the Design Bench (Trabucco et al., 2022) framework. We consider all contimuous datasets in Design Bench datasets: Ant Morphology, D'Kitty Morphology, Superconductor, and Hopper. Contimuous datascts are chosen since we are using Gaussian denoising diffusion models, though discrete variants also exist and we leave this to future work.
Both morphology datasets are ones in which the morphology of a robot must be optimised in order to maximise a reward function. For these datasets, the ground truth oracle is a morphology-conde condiioonn controller. For Superconductor, the ground truth oracle not accessible and therefore it is approximate. For Hopper, the goal is to sample large (>v 5000 dimensional) set of weights which are used to parameterise controller.
Data splits While our framework is built on top of Design Bench, as mentioned in Section 3 the evaluation differs slightly. In Design Bench, the user is only officially prescribed Derain; and any users intending to perform validation model selection should not use anything external to Durain However, this is fundamentally incompatible with our evaluation framework since we want our validation set to be out-of-distributiin,, and subsampling a part of Dtrain to create Dealid does not satisfy this. As illustrated in Figure 3 we break this
8The Langevin MCMC procedure theoretically guaranteed to produce sample from po(x) Welling & Teh 2011 As
opposed Equation g where no noise is injected into the procedure and therefore samples are mode seeking.
13

---

Published in Transations on Machine Learning Research (05/2024)
Table 2: Summary of datasets used in this work. t: thresholds shown are all defaults from Design Bench, with the exception of Hopper 50% which is subset of the original Hopper dataset (see Paragraph 4 for justification); f: because no exact oracle exists, the way the dataset split corresponds that shown in Figure 3c
|| # features
[Pirain| / [D]
71
exists?
Ant Morphology
60
10004 / 25009
165.33
Kitty Morphology
56
10004 / 25009
199.36
Superconductor Hopper 50%
86
17014 / 21263
74.0
5126
1600 / 3200
434.5
convention and define the validation split to be Derain DD i.c. their set difference. Note that if ground truth oracle exists, there is no need to define a Drost, and this is the case for all datasets except Superconductor (Figure 3b). Otherwise, for Superconductor a random 50% subsample of (Duain \ D) is assigned to Dvalid (Figure Beb and we use the pre-trained test oracle RandomForest-vo provided with the framework, which was trained on the full datasot D.
One maance with the Hopper dataset is that the full dataset D and the training set Dirain are equivalent, presumably because of the scarcity of examples. This means that validation set cannot be extracted unless the training set itself is redefined, and this means that the training set in our framework is no longer identical to that originally proposed by Design Bench. To address this, we compule the median with respect to Duain: and take the lower half as Durain and the upper half as Dvalid. We call the final dataset 'Hopper 50%;, to distinguish it from the original dataset.
Oracle pro-training The validation oracle fo is MLP comprising of four hidden layers, trained on Dusan U Dvalid with the mean squared error loss function. We do not apply any special regularisation tricks to the model. Note that in the case of the Superconductor - the only dataset that doesn't t admit a ground truth oracle f - it is not to be confused with the approrimate test oracle, which is trained on D D= Derai U valid U Dests The approximate test oracle we use for Superconductor is RandomForest-vo, which is provided with the framework.
Architecture The architecture that we use is a U-Net derived from HuggingFace's 'annotated diffusion model' D whose convolutional operators have been replaced with fully connected layers for all datasets except Hopper. For Hopper, use 1D convolutions because MLPs performed poorly and significantly blew up the number of learnable parameters. Furthermore, since we know that the Hopper dataset is a feature vector of neural network weights is useful to exploit locality.
For all experiments we train with the ADAM optimiser (Kingma Ba 2014), with a learning rate of 2 x 10-5, 3 = (0.0, 0.9), and diffusion timesteps T 200. Experiments are trained for 5000 epochs with single P-100 GPUs. Input data is normalised with the min and max values per feature, with the min and max values computed over the training set Derain The same is computed for the score variable y, i.e. all examples in the training set have their scores normalised to be within [0, 1].
Experiments For each validation metric and dataset, we run many experiments where each experiment is a particular instantiation of hyperparameters (see Section A.3.1 for more details), and the experiment is run as per Algorithm Ι By running this algorithm for all experiments we can derive an m N matrix U of validation metric values, where Uú denotes the best validation metric value found for metric i and experiment j. Then, if invoke Algorithm 2 on the same experiments this will give us matrix V of m x N of test rewards. By plotting U, against V. we obtain the scatterplots shown in Figure 5 and the Pearson correlation can be defined simply as pearson V:) for the i'th validation metric.
14

---

Published in Transations on Machine Learning Research (05/2024)
Mc-DSM
Mro
-Mcc
-Mreward
MAg
0.2
0.0
-0.4
-0.6
-0.8
Ant (c.fg.))
Kitty (c.f...)
SD (c.fg..)
Hopper50 (c.f.g.)
Ant (c.g.)
Kitty (c.g.)
SD (c.g.)
Figure 4: The Pearson correlation computed for each dataset diffusion variant. sPearson correlations are computed as per the description in Paragraph 4 Since each validation metric is desgned to be minimised, the ideal metric should highly negatively correlated with the test reward (Equation 24), which to be maximised. By counting the best metric per experiment, we obtain the following counts (the more ticks the better): -Mreward: V MED: Vv, MAgr: VVV, Mc-DSME
4.1 Results
Classifter-free guidance In Figure f we plot the Pearson correlations achieved for each dataset as well as each diffusion variant, classifier-free guidance ('cfg') and classifier- ('eg'). Since all validation metrics are intended to be minimised,
interested in metrics that correlate the most negatively with the final test reward, iee the smaller some validation metric is, the larger the average test reward per Equation 24 We first consider the first four columns of Figure 4 which correspond to just the classifier-free guidance experiments over all the datasets. The two most promising metrics appear to be MAg and Mreward- Interestingly, Mc-DSM performs the best for Hopper50, but we also note that this was our worst performing dataset and no we did not obtain favourable results for any validation metric considered. Since we had modify the dataset permit a training split, contains very few examples as well as an unusually large feature to exmaple ratio (i.e. 5126 features for 1600 examples). Since Mc-DSM the only validation metric which is not function of the validation oracle, it is possible the validation oracle is too poor for the other metrics to perform well.
Interestingly, MDC does not perform well for any of the datasets. It is unclear why however, since Mnc is a sum of two terms measuring precision and recall which are both useful things to measure for generative models (see Section A.1). may require further exploration in the form of weighted sums instead, since it currently defined give equal weighting to precision and recall. Since testing out various weighted combinations of the two would have required significantly more compute, we did not explore it.
Classifier-based guidance The last three groups barplots in Figure constitute the classifier guidance experiments. For this of experiments the most promising metrics appear to be MFD and M Note that while have also plotted Mc-DSM it is not appropriate as validation metric for classifier-based guidance this is because during those experiments 7 is deterministically set to which means the matching loss in Equation 14 is never conditioned on the variable, and therefore never sees y. (The barplots also corroborate this, as the correlation is virtually zero for all experiments.)
Summarising both guidance variants If we count the best validation metric per subplot, then the top three are MAg (3 wins), MFD (2 wins), and Mreward (1 win), respectively. This suggests that if we were to perform a real world MBO then these metrics should be most considered with respect to their rankings. However, since this paper only concerns itsolf with Gaussian DDPMs, we can not confidently extrapolate this ranking to other classes of generative model or discrete datasets. We leave the testing of discrete diffusion variants future work.
Sample quality vs diversity We find that the guidance hyperparameter w makes a huge difference to sample quality, and this is shown Figure / for all datasets and across cach validation metric. (For brevity's sake, we only show results for classifier-fre guidance, and defer the reader to Soction A.4 for the equivalent set of plots for classifier guidance.) In each subplot individual points represent experiments and these also
15

---

Published in Transations on Machine Learning Research (05/2024)
= -0.1262
0.0564
P =0000512
= -0.3081
p -0.2772
-oot reward
Mest rereard
Meet rereard
500
renard
800
0.06
0:07
0.08
1.0
1.5
2.0
15
1.0
0.5
1.4
1.3
1.2
20000
40000
60009
.MC DSM
MMD
-Moc
-rewaadd
MAgr
0.00
0.50
0.75
1,00
1.25
1.50
1.75
2,00
W
(a) Ant Morphology. Mzeward and MAge are the most negatively correlated with the test reward.
Mtest- roward
-0.0046
Mrrect roword
-0,2192
Mrast roward
0.1479
Mrast roword
-0.1611
Mrest reword
-0,0255
250
250
250
250
250
200
200
200
200
200
-20
JMC DSM
MPD
-MCC
Mrwwwrrd
.MAgr
(b) Kitty Morphology (c.f.g.). Mzeward and MED are most negatively correlated with the test reward.
Misst reward
-0.1529
Maust roward
0.1828
Masst rewwer
==
0.4337
Mexx reward
p = -0.1714
Maast reward
=
-0.2003
100
200
100
100
50
50
50-
50
50
0.004
0.005
0,006
25
50
75
100
-15
-10
-14
-12
-10
500
1000
MC DSM
-Mraaard
MAg
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
W
(c) Superconductor (c.f.g.). Miwward and MAge most negatively correlated with the test reward.
Most Mosttrowap reward
-0,5357
Most- reward
-0.0964
Mest reward
0.3893
M Hest reword
0.3143
Mest reward
p 0.5286
250
250
250
250
250
-250
-250
-250
250
-250
JMC- DSM
Mrp
-MMC
-Myeward
Magg
20
60
100
W
(d) Hopper 50% (c.f.g.). MED and Mc-DSM are the most negatively correlated with the test reward.
Figure 5: Correlation plots for each dataset using the classifier-free guidance (c.f.g.) diffusion variant. Each point is colour-coded by w, which specifies the strength of the 'implicit' classifier that derived (Equation 18). We can see that makes a discernible difference with respect to most of the plots shown. For additional plots for other datasets, please see Section A.4
colour-coded with guidance hyperparameter w. We can see that the choice of w makes huge difference with respect to the test. reward, and appears to highlight a well-established 'dilemma" in generalive modelling, which is the trade-off between sample quality and sample diversity (Ho & Salimans 2022 Brock et al. 2018 Burgess et al. 2018 Dhariwal & Nichol 2021). For instance, sample quality is too hcavily weighted, then sample diversity suffers and as a result the candidates we generate which supposedly comprise high reward may actually truly be bad candidates when scored by the ground truth. Conversely, if sample diversity
16

---

Published in Transations on Machine Learning Research (05/2024)
is too heavily weighted then we miss out modes of the distribution which correspond to high-rewarded candidates.
In Table 3 we present the 100th percentile test rewards for each combination of dataset and conditioning variant. For each of these we simply choose the most promising validation metric as per Figure 4 find the best experiment, and then compute its 100th percentile score as consistent with evaluation Design Bench:
M1 max (s(ortad((S..))))) S po., (x.
(27)
Note that Equation 27 differs from our test reward equation (Equation 24 in that a max is used as the aggregator function instead of the mean. Lastly, we present 50th percentile results in Table S5 which corresponds to a median instcad of max.
Aut Morphology
D'Kitty Morphology
Superconductor
Hopper50
Drain
0.565
0.884
0.400
0.272
Auto. CbAS
0.882 + 0.045
0.906 0.006
0.421 + 0.045
CbAS
0.876 = 0.081
0.892 0.008
0.503 = 0.069
BO-qEI
0.819 + 0.000
0.896 0.000
0.402 + 0.034
CMA-ES
1.214 + 0.732
0.724 1 0.001
0.465 1 0.024
Grad ascent
0.293 i 0.023
0.874 1 0.022
0.518 1 0.024
Grad ascent (min)
0.479 = 0.064
0.889 + 0.011
0.506 + 0.009
Grad ascent (ensemble)
0.445 + 0.080
0.892 + 0.011
0.499 = 0.017
REINFORCE
0.266 + 0.032
0.562 + 0.196
0.481 = 0.013
MINs
0.445 + 0.080
0.892 t 0.011
0.499 1 0.017
COMs
0.944 = 0.016
0.949 + 0.015
0.439 = 0.033
Cond. Diffusion (c.f.g.)
0.954 1 0.025
0.972 1 0.006
0.645 1 0.115
0.143 1 0.037
Cond. Diffusion (c.g.)
0.929 + 0.013
0.952 1 0.010
0.664 1 0.007
Table 3: 100th percentile test rewards for methods from Design Bench (Trabucco al. 2022) as well as our diffusion results shown in the last two rows, with c.f.g standing for classijier-free guidance (Equation 18] and c.g, standing for classifer--uuidance (Equation 16]. Each result is average computed over six different runs (seeds). test rewards min-max normalised with respect to the smallest and largest oracle scores in the full dataset, i.c. any scores greater than 1 are greater than any score observed in the full dataset. Design Bench results are shown for illustrative purposes only while our training sels are equivalent to theirs, we use a held-out validation set to guide model selection, which makes a direct comparison to Design Bench difficult.
Which conditional variant should be used? From Table both conditioning variants perform roughly on par with each other, though it appears classifier-fret free guidance performs slightly better. However, they are not necessarily equally convenient to use in real-world MBO setting. A real-world MBO setting would involve some sort of online learning component since the ground truth oracle needs to be eventually queried (see Section 3 Paragraph 3). Because of this, we recommend the use of classifier-based guidance, where the unconditional generative model (=(x) can be independently trained offline while the classifier (5(y)z can be Bayesian probabilistic model which is able to be updated per-example basis (i.c. after each query of the ground truth).
5 Conclusion
In this work, we asked fundamental question pertaining to evaluation in offline MBO for diffusion-based generative models: which validation metries correlate well with the ground truth oracle? The key idea is that if we can run our presented study at scale for a both difficult and diverse range of datascts for which the ground truth is known, insights derived from those findings (such as what are robust validation metries) can be transferred to more real-world offline MBO tasks where the actual ground truth oracle is expensive to evaluate. To approach this, our evaluation framework designed measure how well a generative model extrapolates: the training and validation sets are seen as coming from different y-truncated distributions, where examples in the validation set comprise range of y's that not covered by the training set and are
17

---

Published in Transations on Machine Learning Research (05/2024)
larger than those in the training set. Therefore, from the point of view of the generative model, the validation set is out-of-distribution. Because model selection involves measuring some notion of desirability on the validation set (via a validation metric), we are effectively trying to select for models that can extrapolate.
While our proposed evaluation framework is model-agnostic, we presented it in the context of Gaussian DDPMs on four continuous datasets prescribed by Design Bench, as well as across five different validation metrics and two forms of label conditioning for diffusion models: classifier-free and classifier-based guidance. The five validation metrics we chose were inspired by existing MBO works well the GAN literature. After ranking the validation metrics based on their correlation with the test reward, we found that the best three performing ones in descending order were agreement, Fréchet Distance, and the validation score, respectively. While we ran à considerable number of experiments in this study, further exploration should be done in testing on discrete datasets (which would require discrete or latent diffusion models) as well as a deeper exploration into how these models perform under different sampling algorithms and design choices (for instance see Karras et al. (2022)).
Lastly, we derived some interesting insights from our work. Regardless of which conditioning variant is used, we found that the most important hyperparameter to tune is the classifier guidance value, which controls the trade-off between sample quality and sample diversity. Furthermore, we posit that the classifier- variant of diffuusio is likely more convenient in practice since it makes it easier to bridge the gap between offline and online MBO.
References
Asmussen, Søren and Glynn, Peter W. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007.
Borji, Ali. Pros and cons of GAN evaluation measures: New developments. Computer Vision and Image Underslanding, 215:103329, 2022.
Brock, Andrew, Donahue, Jeff, and Simonyan, Karen. Large scale gan training for high fidelity natural image 11096, 2018.
synthesis. Xiviv preprinl
Brookes, David, Park, Hahnbeom, and Listgarten, Jennifer. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pp- 773-782 PMLR, 2019.
Burgess, Christopher P, Higgins, Irina, Pal, Arka, Matthey, Loic, Watters, Nick, Desjardins, Guillaume, and Terchner, Alexander. Understanding disentangling beta-vae. arXiv preprint arXiv: N0o:1804..000 03599, 2018.
Chen, Can, Zhang, Yingxuelf, Fu, Jie, Liu, Xue (Steve), and Coates, Mark. Bidaa learning for oflline infinite-width model-based optimization. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 29454 29467. Curran Associates, Inc., 2022a. URL https: https::/prooceding..nett neurips....ppp.pr.. cc/paper files/ /alles//pper//22// .df.
Chen, Yutian, Song, Xingyou, Lee, Chansoo, Wang, Zi, Zhang, Richard, Dohan, David, Kawakami, Kazuya, Kochanski, Greg, Doucet, Arnaud, Ranzato, Marclaurelio, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems, 35:32053 32068, 2022b.
Dhariwal, Prafulla and Nichol, Alexander. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780 8794, 2021.
Dosovitskiy, Alexcy and Brox, Thomas. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
Dowson, DC and Landau, BV666017. The Fréchet distance between multivariate normal distributions. Journal of mallivariale analysis, 12(3):450-455 04500,4555115 1982.
Fannjiang, Clara and Listgarten, Jennifer. Autofocused oracles for model-basee design. Advances in Neural Information Processing Systems, 22945112566, 2020.
18

---

Published in Transations on Machine Learning Research (05/2024)
Fu, Justin and Levine, Sergey. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint (xxi:2102.099700 2021.
Hakhamaneshi, Kourosh, Abbeel, Pieter, Stojanovic, Vladimir, and Grover, Adilya. Jumbo: Scalable multi-task bayesian optimization using offline data. arXiv preprint orXiv:2106.0000 2021.
Hamidieh, Kam. A data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 154:346-354 -544, 2018.
Heusel, Martin, Ramsauer, Hubert, Unterthiner, Thomas, Nessler, Bernhard, Klambauer, Günter, and Hochreiter, Sepp. GANs trained by two time-scale update rule converge to a nash equilibrium. CoRR, hls/1706.08500, 2017. URL
08500
Ho, Jonathan and Salimans, Tim. Classifier-free diffusion guidance. ai preprint urii::2077.12566 2022.
Ho, Jonathan, Jain, Ajay, and Abbeel, Pieter. Denoising dilfusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6880 2020.
Huszár, Ferenc. How (not) train your generative model: Scheduled sampling, likelihood, adversary? arXiv
preprint ar arXio:1511.05000 2015.
Karras, Tero, Aittala, Miika, Aila, Timo, and Laine, Samuli. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565 26577, 2022.
Kingma, Diederik P and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:141266980 2014.
Kumar, Aviral and Levine, Sergey. Model inversion networks for model-based optimization. Advonces in Neural Information Processing Systems, 33:5126 5137, 2020.
Kynkääänieemi Tuomas, Karras, Tero, Laine, Samuli, Lehtinen, Jaakko, and Aila, Timo. Improved precision and recall metric for assessing generative models. Adoances in Neural Information Processing Systems, 32, 2019.
Maus, Natalie, Jones, Haydn, Moore, Juston, Kusner, Matt J, Bradshaw, John, and Gardner, Jacob. Local latent space bayesian optimization over structured inputs. Advonces in Neural Information Processing Systems, 35:34505-33451 2022.
Nacem, Muhammad Ferjad, Oh, Scong Joon, Uh Youngiung, Choi, Yunjey, and Yoo, Jaejun. Reliable fidelity and diversity metrics for generative models. In International Conference Machine Learning, pp- 7176 7185. PMLR, 2020.
O'Donoguue, Brendan, Osband, lan, and lonescu, Catalin. Making sense of reinforcement learning and probabilistic inference. arXiv preprint arXiv:2001.00805, 2020.
Piche, Alexandre, Pardinas, Rafael, Vazquez, David, Mordatch, Igor, and Pal, Chris. Implicit offline reinforcement learning via supervised learning. arXiv preprint arXiw:2210.1222 2022.
Qi, Han, Su, Yi, Kumar, Aviral, and Levine, Sergey. Data-driven offline decision- via invariant representation learning. arXie preprint arXi::2211.11340, 2022.
Sajjadi, Mehdi SM, Bachem, Olivier, Lucic, Mario, Bousquet, Olivier, and Gelly, Sylvain. Assessing generative models via precision and recall. Advonces in neural information processing systems, 31, 2018.
Sohl-Dickstein, Jascha, Weiss, Eric, Maheswaranathen, Niru, and Ganguli, Surya. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Song, Yang and Ermon, Stefano. Generative modeling by estimating gradients of the data distribution. Aduances in Newral Information Processing Systems, 32, 2019.
19

---

Published in Transations on Machine Learning Research (05/2024)
Song, Yang and Ermon, Stefano. Improved techniques for training score-based generative models. Advances in neural informalion processing systems, 1112488-12445 2020.
Theis, Lucas, Oord, Aäron van den, and Bethge, Matthias. note on the evaluation of generative models. arXiv preprint vrXxx11111.088 2015.
Trabucco, Brandon, Kumar, Aviral, Geng, Xinyang, and Levine, Sergey. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pp. 10358-10368. PMLR, 2021.
Trabucco, Brandon, Geng, Xinyang, Kumar, Aviral, and Levine, Sergey. Design-bench: Benchmarks for data-driven ofline model-based optimization. In International Conference on Machine Learning, pp- 21658 21676. PMLR, 2022.
Vapnik, Vladimir. Principles of risk minimization for learning theory. Aduances in neural information processing systems, 4, 1991.
Wang, Zi, Dahl, George E, Swersky, Kevin, Lee, Chansoo, Mariet, Zelda, Nado, Zachary, Gilmer, Justin, Snoek, Jasper, and Ghahramani, Zoubin. Pro-trained gaussian processes for bayesian optimization. arXiv preprinl ar Xiv:22109 08215, 2021.
Welling, Max and Teh, Yee W. Bayesian learning via stochastic gradient langevin dynamics. Proceedings of the 28th international conference on machine learning (ICML-11), pp. 281-688, 2011.
Weng, Lilian. What are diffusion models? lilianwengggilhubiio Jul 2021. URL https:///iliaanengg github -atpatttt2200000-1-11a-fuuuoooo-
Wistuba, Martin and Grabocka, Josif. Few-shot bayesian optimization with deep kernel surrogates. In International Conference on Learning Representations, 2021.
Yu, Sihyun, Ahn, Sungsoo, Song, Le, and Shin, Jinwoo. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34:4619-4631 2021.
20

---

Published in Transations on Machine Learning Research (05/2024)
A Appendix
A.1 Validation metrics
Frechet Distance 'Likelihood-free' metrics are used almost exclusively in the GAN literature because there is no straightforward way to compute likelihoods for this class of models, i.e. po (a ( y) cannot be evaluated. Furthermore, the search for good metrics is still an active topic research (Borji 2022). Common likelihood-free metrics involve measuring some distance between distributions in some predefned feature space. For instance, for GANs trained on natural image datasets the Fréchet Inception Distance (FID) (Heusel et al. 2017) is used to fit Gaussians to both distributions with respect to the feature space of an NeceptionNet classifier trained on ImageNet. Since the acronym 'FID' specifically refers to very specific InceptionNet--ant based model, we will simply call it 'FD: If we assume that FD is computed in some latent space characterised by an arbitrary feature extractor fn+ : X - H. then FD can be computed in closed form as follows Dowson & Landau 1982
Mr = - - (()((X)) + (2)) (FXT)(())))(((()X)))((()))))))")
(S28a)
where MPD R* and lower FD is better. FD is also known as the 2-Wasserstein distance. Here, X E Nx denotes N samples coming from reference distribution (i.e. ground truth) and X are samples coming from the generative model. H = (X) denotes these inputs mapped to some feature space. Conveniently, we can simply define the feature space to be with respect to some hidden layer of the validation oracle. One caveat of FD is that it may have a stronger bias towards recall (mode coverage) than precision (sample quality) (Kynkäänniemi et al.] 2019) and that reports a single number, which makes it difficult to tease apart how well the model contributes to precision and recall. Furthermore, while there exists a canonical network architecture and set of weights to use for evaluating generative models natural image datasets (i.c. particular Inception-VS. V3 network that gives rise to the Frechel Inceplion Distance), this is not the case for other types of datasets. This means that, unless a particular feature extractor is agreed upon, comparing results between papers is non-trivial.
Density and coverage We also consider 'density and coverage' (Naeem et al., 2020 which corresponds to an improved version of the 'precision and recall' metric proposed in Kynkäännieni et al. (2019). In essence, these methods estimate the manifold of both the real and fake data distributions in latent space via the aggregation of hyperspheres centered on each point, and these used to define precision and recall: precision is the proportion of fake dula that can be explained by real dala (in latent space), and recall is the proportion of real data that can be exploined by fake data (again, in latent space).
Similar to FD (Paragraph) A.1 1 let us denote H, as the example X; embedded in latent space. Let us also define BH, NNDx(N, as the hypersphere centered on H, whose radius is the k-nearest neighbour, and k is a user-specified parameter. 'Density' (the improved precision metric) defined as:
N f NND,) Fl
M
Maensity HH,
1
ΣΣ
(S29)
RN
how many real nelghbourhoods does fake sample R, belong to?
where 1(:) is the indicator function, and large values corresponds to a better density. While coverage (improved 'recall') can be similarly defined by switching around the real and fake terms like so, the authors choose to still leverage manifold around real samples due to the concern of palentially too many outliers in the
21

---

Published in Transations on Machine Learning Research (05/2024)
generated distribution H. As result, their coverage is defined as:
Meovoragel H, HH Σ UIHH, B(H,,
(S30)
1=
3j s.t. H, E
(H.))
(S31)
is there any fake sample belonging to a, 's neighbourhood?
where, again, larger value corresponds to better coverage. This leads to the addition of both metries, MDC, which is simply:
M R))
(X)) + Meoverage (fa (x)): (X):
(S32a)
Similar to FD, we use the validation oracle fo to project samples into the latent space. We do not tune k and simply leave it to k 3, 3, which is a recommended default.
A.2
Related work
A.2.1
Conditioning by adaptive sampling
CbAS (Brookes et al., 2019), like our proposed method, approaches MBO from a generative modelling perspective. Given some pre-trained 'prior' generative model on the input data po(2r), the authors propose the derivation of the conditional generative model pe(xly) via Bayes' rule:
po( y) p(o())
p(ul=))o(a) Ja ((g)
(S33)
where p(ylaz) denotes the oracle in probabilistic form, and is not required to be differentiable. More generally, the authors use to denote some target range of y's that would be desirable to condition on, for instance if p(S(x) p(ull)) then:
Po(= ((S)x))o(e
p(S x x)pp (x) (S) Jo P(S x po ( (x)
(S34)
Due to the intractability of the denominator term, the authors propose the use of variational inference to learn sampling network ge (x) that is close as possible to (=()])) as measured by the forward KL divergence. Here, let us use (S] in place of p(S]x)) and assume the oracle (S(() was trained on D TO
5 = arg min KL pa (x) || (x(2))
forward KL
arg min
Σ po( log po (x]()) - qe (a))
arg min
Σ po(< (og()( - po
(x: (x
(S) log (x)
Po (SI(s))) (o log (())
arg max
H (ar +
<
SS
(s( lo
arg max
C
Po
coust.
arg max Em (xr)
(e log 25 (x))
(S35)
oracle
10 their paper the symbol d is used, but here we use Ç since the former is used denote the validation oracle.
22

---

Published in Transations on Machine Learning Research (05/2024)
The authors mention that in practice importance sampling must be used for Equation S35 This is because the expectation is over samples in po(a;), which in turn was trained only examples with (relatively) small y. i.e. those in Drainn Because of this, p(S[z) is likely to be small in magnitude for most samples. For more details, we defer the reader to the original paper (Brookes et al., 2019
To relate the training of CbAS to our evaluation framework, we can instead consider Equation S35 as part of the volidation part of our evaluation framework. In other words, if we define S == ln, oc] and use the validation oracle P in place of po (S( then we can optimise for the extrapolated model the following:
5* arg min Ex)pe(a)
Po (o(()))) log 96 (z:)
(S36)
oracle
Generally speaking, validation metries should not be optimised over directly since they are functions of the validation set, and the purpose of validation set in turn is to give less biased measure of generalisation than the same metric computed on the training set. However, this may not be too big of deal here since we are not taking gradients with respect to the oracle.
Pyy pr a y)
Po,y at, v)
Dvalid
f(x)
mode
Prrain
fe(
metric
B 8 [()]
train
model selection
extrapolate
generetion
Po (
(c(xz)
S
Figure S6: The training and evaluation of CbAS Brookes al. (2019) in the context of our evaluation
framework. The extrapolation equation is described in Equation S35 and involves variational inference to fine-tune (x) into search model (x)..
A.2.2 Model inversion networks and the reverse KL divergence
It turns out that there is an interesting connection between the agreement and the reverse KL divergence between a specific kind of augmented model distribution and the truncated ground truth (a. y)- To see
this, lct us re-consider the generation time optimisation performed in Kumar & Levine 2020 (which we
called MIN-Opl), which tries to find a good candidate x (2) via the following optimisation:
y x
arg max y + Ej log Po Po 8)) + logp(z) V.z
(S37)
agrecment
prior over
2 and y can be generated by performing gradient ascent with respect to y and z. We can also express MIN-Opt with respect to a batch of (y, 2)'s, and this can be elegantly written we express it as optimising over a distribution Pc(z, y). Then we can find such a distribution that maximises the expected value of Equation S37 over samples drawn from pc(2,y):
Pc(2,0)* y)* arg max E. (x [V + a (oggo((lGo(z. + +2 lolgp(())) pc(x.))
(S38)
where e.g. Ç parameterises the distribution, e.g. a mean and variance if we assume it is Gaussian. Although MIN-Opt was intended to be used generation time to optimise for good candidates, we can also treat it as validation metric, especially if we replace the training oracle po(y)=) with the validation oracle (y]a))) For the sake of convenience, let also replace 6 and 62 with one hyperparameter 7,. This hyperparameter
23

---

Published in Transations on Machine Learning Research (05/2024)
can be seen expressing trade-off between selecting for large scores y versus ones with large agreement and density under the prior distribution. This gives us the following:
arg max E (y (V + (oggolvlGo02, v)) log [og)(
Pr(2,
pe(=.))
arg max E yypcll) + 1/ (2 (ogpo log p(x))
(S39)
pc(x,))
arg max Ey (y) + nB y. w.w... (ul logp(z) p((2,))
(S40)
Note that in the last line we instead use the notation I po(aly, =) (a delta distribution) in place of a = Gg(z,y) which is a deterministic operation. We can show that Equation $39 has a very close resemblence to minimising the reverse KT. divergence between a specific kind of augmented model and the 7-truncated ground truth, with respect to our distribution Pc(z. y). Suppose that instead of the typical augmented model (e.,(x.)) y) po (z) we consider one where z and y are drawn from a learnable joint distribution Pc(2.y) y), and we simply denote pol (aly,, z) to be delta distribution (since Go(z, y) is deterministic). We can write this new augmented model the following:
po.c v) po(arly, Pc y)dz.
(S41)
GAN
Although this distribution is not tractable, we will only be using it to make the derivations more clear.
Let us work backwards here: we take Equation S39 but substitute the inner square bracket terms for the reverse KL divergence between the augmented model of Equation S41 and the ground truth pd, (2., y); we obtain the following:
Pe Pc(2, 3)" arg min - -E - + KL po.c(a, v) p. (2., v)
y-Pe
reverse KL
arg min E (1)) + E Bm.y-po. log (2, - En.yrn.. log (ax, v)]
Pc
arg max E (y)- E E (m.w)) log Po.c (2, y) Emy-po. po v)
Pc
= arg max E (1) + HI [po.c] + Eu..rrro ((m.)) log (z, (z.
Pc
arg max E Byyp + THI (poccc + t. (x.w)
log p ((vlar) log [2y)]
Pt
entropy
agreement
arg max E Ey)D +
H [ wew.
log (vlzz) + log log
(S42)
PC
entropy
agreement
The entropy term is not tractable because we cannot evaluate its likelihood. For the remaining two terms inside the expectation, the agreement can be approximated with the validation oracle Po(y)z) ()). Howeer, it would not be practical to estimate logp, p. (x) since that would require us to train a separate density po (xx to approximate it.
For clarity, let us repeat Equation S39 here:
-² arg max Ew>p + w.w.....r..eeel....eee
log (o + logp(z)).
(S43)
pc(=.))
The difference between the two is that (1) there is no entropy term; and (2) log Po term is replaced with poz)(2) for MIN-Opt, which is tractable since the know the prior distribution for the GAN. From these observations, we can conclude that MIN-Ops (S39) comprises an approximation of the reverse KL divergence where the entropy term is omitted and the log density of the data is replaced with the log density of the prior.
24

---

Published in Transations on Machine Learning Research (05/2024)
A.2.3 Exponentially tilted densities
Once the best model has been found via an appropriate validation metric, one can train the same type of model on the full dataset DD using the same hyperparamelers as before. Ultimately, we would like to be able to generate candidates whose y's exceed that of the entire dataset, and the same time at plausible according to generative model. To control how much tradc-off high likelihood versus high rewarding candidales, we can consider the exponentialyy tilled densily (Asmussen & Glynn 2007 O'Donoghue et al.
2020
Piche et al. 2022]:
(a, y) exp(pi x(n)),
(S44)
where s(n) is a normalisation constant, and smaller y puts larger emphasis on sampling from regions where y is large. Taking the log of Equation S44 we arrive at:
x , = arg max (2, + y 2,9
(S45)
In practice, it would not be clear what the best 7 should be, but a reasonable strategy is to consider a range of 17's, where larger values encode a higher tolerance for 'risk' since these values favour higher rewarding candidates the cost of likelihood. Note that for VAFs and diffusion models, (or,p()),)) y) will need to be
approximated with the ELBO. Interestingly, since diffusion models have an extremely close connection to score-based models, one could 'convert' diffusion model to score-based model (Weng 2021 and derive Va.y log (og,o({z (1 y), and this would make sampling trivial.
One potential issue however relates to our empirical observation that predicted scores for generated candidates exhibit very high variance, i.c the agreement scores are very high (see Figures S7b and S9b In other words, when we sample some y ~ Pa, (xr v) (i.e. from the augmented model) there is significant uncertainty as whether a rcally does have a score y. One potential remody is to take inspiration from the MIN-Opt generation procedure (Section A.2.2 and add the agreement term to Equation $45-
x 1/" arg max log pa,,, v) 1 + a log (vllz) a.y
(S46)
Due to time constraints, we leave additional experimentation here to future work.
A.3
Additional training details
A.3.1 Hyperparameters
The architecture that we use is convolutional U-Net from HuggingFace's 'annotated diffusion model !! whose convolutional operators have been replaced with fully connected layers (since Ant and Kitty morphology inputs are flat vectors).
For all experiments we train with the ADAM optimiser Kingma Ba 2014), with learning rate of 2 10-5 3 = (0.0, 0.9), and diffusion timesteps T 200. Experiments are trained for 5000 epochs with single P-100 GPUs. Input data is normalised with the min and max values per feature, with the min and max values computed over the training set Dtrain- The same is computed for the score variable y, i.e. all examples in the training set have their scores normalised to be within [0, 1].
Here we list hyperparameters that differ between experiments:
diffusion wwwfluuiiiiwwwnrss.ooeo taw:: for classifier-free diffusion models, this is the probability of dropping the label (score) y and replacing it with null token. For classifier guidance models, this is fixed to since this would correspond to training completely unconditional model.
gen_kwargs. dim: channel multiplier for U-Net architecture
wwwpaiiiooovrrrg.veccoc for classifier-based guidance, this is the w that corresponds to the w in Equation
16
25

---

Published in Transations on Machine Learning Research (05/2024)
A.3.2 Hyperparameters explored for classifier-free free guidance
{
ddiffusion_ kwargs. tau': [0,05, 0.1, 0.2, 0.4, 0.5), "engeenwaree dim': {128, 256}
}
A.3.3
Hyperparameters explored for classifier guidance
{
diffusion ddiffusion_kwrrga..v w {1.0, 10.0, 100.0}, epochs': {5000, 10000}, gen_ koankw..rg..d {128, 256}
}
A.3.4 Classifier guidance derivation
Let denote æ, as the random variable from the distribution g(xx), denoting noisy input at timestep Through Bayes' rule we know that ((<,))) ()) a(m(3)) ((vl)))((z.) Taking the score Væ, log (())())))))))))))) (which does not depend on (())), get:
((v)
((1)
V log logg( Va, logg(u/æz;) Va, log (o:)
(S47)
(co(Ze, t, - =,(xx,))) t)):
(S48)
where the last line we make clear the connection between the score function and the noise predictor co Weng (2021). Since we would like to derive the conditional score, we can simply re-arrange the equation oblain
c (2 t. y) = co(2,))) t) - /I - a, 2. log (3)(2;) = ()) - VI a, V.V.. log (3)(x))
(S49)
(S50)
where approximate the classifier ())(z;) with our (approximate) training oracle po(y)æ;). In practice, can also define the weighted version follows, which allows us to balance between conditional sample quality and sample diversity:
co(0,, y, w) w) ea (2 /I U.wVa x, Po
(S51)
Therefore, in order to perform classifer-guided generation, replace co(x),t) in whatever generation algorithm use with eo (x l, yo w) instead.
A.3.5 Classifier-free guidance
In classifier-free guidance conditional score estimator ca(#r, v, is estimated via the algorithm described in
Ho & Salimans 22022), where the y token is dropped during training according to some probability If y is dropped it is replaced with some unconditional token. In other words, the noise predictor (score estimator) is trained both conditionally and unconditionally, which means we have both co t) as well as ca y, t).
From Bayes' rule, we know that: p(ylat) = p(y.z)) p(mll))(()) and that therefore the score Va, log Mog.(yylan, (m))) p(a.) p(m.)
is:
V x. logp(y/<z)) Vx, loge(x)lv)) V.L. log)(
(S52)
26

---

Published in Transations on Machine Learning Research (05/2024)
We simply plug this into Equation 16 to remove the dependence on po(y)z)):
co (2 t, y: w)
c0( v1 a V log Po (b(((x)))
(S53)
= vI deW
m. log (20a (v) vx, log (202)
(S54)
co( 1) v1 ō
1 v1 az
co (a v, -1
-I-ai co(
(S55)
=colxe, 1) + wool v.t) wco(os, t) co 1) + w (o 1. (0(
(S56)
(S57)
27

---

Published in Transations on Machine Learning Research (05/2024)
A.4 Correlation and agreement plots
Agreement plots We plot the conditioning E linspace(Uming Mmax) against. the validation/tesl oracle predictions for candidates conditionally generated with that y. We call these "agreement plots' since the sum of squared residuals for cach point would constitute the agrecment (with a perfect agrecment of zero corresponding to a diagonal dotted line on each graph). Here we demonstrate this amongst the best three models with respect to MAgr: since this is most correlated with the test oracle (see Figure S7ah. The shaded regions denote +1 standard deviation from the mean, and the marker symbols denote the max/min score for each y with respect to cither oracle.
A.4.1 Ant Morphology
-0.1262
0.0564
-0.0512
-0.3080
-0.7772
Most temard
Mostt reward
reward
1000
Mustt
330
300
300
230
200
200
200
200
Mc OSK
MrD
-Moc
Mreward
Mage
0.00
0.75
1.00
1.25
1,50
1.75
2,00
(a) Correlation plot
Macore -1.42
Murore
-1.38
-Mecrre
-1.36
oracle scofe
600
oracle Score
600
SAN
600
M0
400
400
400
oracle
200
200
200
400
200
200
400
600
400
2000
200
400
600
430
200
200
400
600
(b) Agreement plot
Figure S7: Results on Ant Morphology datasct, using the classificr-free free guidance variant (Equation
p -0.0015
p -0.5972
p -0.3130
-0.2952
p 0.4389
Munue reaarr
Must reward
hert- resard
Miese reward
Micstt reward
400
250
205
200
200
200
0.253 0.054 0055
23
-2.40
-3.35
-1.10
-1.05
-1.00
-2.95
100000
122020
-DC--DS
MrD
Moc
Mrowaad
Mag
200
400
600
800
1000
W
(a) Correlation plot
Mrp 1.49
Mrp
1.61
Myo
1.62
oracle SCOP
600
oracle COOE
600
800
400
400
400
200
oracle
400
200
200
400
600
400
200
200
400
600
400
200
200
400
600
(b) Agreement plot
Figure S8: Results on Ant Morphology dataset, using the classifier guidance variant (Equation 16
28

---

Published in Transations on Machine Learning Research (05/2024)
A.4.2 D'Kitty Morphology
See Figures 59 and
$10
0..0446
-0.2192
0.1479
-016611
-0.0255
reword
Muss remord
Mrest rewars
Mrest remard
250
250
250
250
230
200
200
200
220
(CC--Dop DSM
MMD
-Mac
-Merswrrd
Mar
1.5
W
(a) Correlation plot
MED
0.26
MFR = 0.26
MFD
0.27
oracle scare
oracle score
300
300
300
200
oracle
100
100
100
100
150
200
250
300
250
200
350
100
y
(b) Agreement plot
Figure S9: Results on D'Kitty Morphology dataset, using the classifier- free guidance variant (Equation
p -0.0590
Mist rocard
-0.7758
Muss reword
-0.6659
Mosst- ronord
p -0.5141
p -0.7998
200
Misst
0.003
0.062
0.5
-0.5
-1.03
-0.95
20002 40000 sabao 80000 Magr
JMC -059
AMP
-Mrrward
W
(a) Correlation plot
MAg 10842.87
Magr = 12533.25
Magr 12753.73
oracle score
oracle score
oracle Score
200
AM:
200
200
0
-200
1000
-000
200
-00
-000
-200
200
400
8000
200
400
y
(b) Agreement plot
Figure S10: Results on D' D'Kitty Morphology dataset, using the classifier guidance variant (Equation 16
29

---

Published in Transations on Machine Learning Research (05/2024)
A.4.3 Superconductor
See Figure $11
rooard
-0.1529
JM Mree rewars
p 0.1828
Most reward
0.4337
-0.1714
Musst reoard
-0.2003
100
Muss-
50
JMC DDSM
AMFF
Mac
MMeearr
Magr
W
(a) Correlation plot
MAgr 421.91
MAge 439.13
Magr 643.39
oracle score
oracle Score
uiild
oracle score
X wai GX
100
batt f51
100
test to
100
teet
50-
50
50
25
50
75
100
125
25
50
75
100
125
25
50
75
100
125
y
y
y
(b) Agreement plot
Figure S11: Results on Superconductor dataset (Hamidich, 2018 using the classificr-free guidance variant (Equation 18
0.0188
Mest- rowaadd
-0.3922
roward
=
0.1252
Most- roaadd
-0.7096
Mest- rosand
-0.7574
50
MCc. DSM
Mro
-Moc
-Mrwardd
Mags
6.2
1.
W
(a) Correlation plot
-Mscore -10.41
-Mscore -10.39
Mscore -9.68
oracle score
oracle score
salc EAK)
100
100
tai
1.00
oracle
125
100
125
75
100
125
y
y
(b) Agreement plot
Figure $12: Results on Superconductor dataset. Hamidiih (Equation 18).
2018 using the classifier-free guidance variant
30

---

Published in Transations on Machine Learning Research (05/2024)
50th pt.
100th
Denin
||
0.400
Auto. CbAS
0.131 0.010
0.421 = 0.045
CbAS
0.111 0.017
0.503 = 0.069
BO-qEI
0.300 = 0.015
0.402 = 0.034
CMA-ES
0.379 = 0.003
0.463 = 0.024
Grad.
0.476 = 0.022
0.518 = 0.024
Grad. Min
0.471 = 0.016
0.306 = 0.009
Grad. Mean
0.469 = 0.022
0.499 = 0.017
REINFORCE
0.463 1 0.016
0.481 1 0.013
MINs
0.336 + 0.016
0.499 1 0.017
COMs
0.386 + 0.018
0.439 = 0.033
Cond. Diffusion (c.f.g.) ||
0.518 0.045
0.636 1 0.034
Table S4: 100th and 50th percentile rewards for the Superconductor dataset. Results above our conditional diffusion results (bottom-most row) were extracted from Design Bench Trabucco al. (2022). For experiments, each result is an average computed over three different runs (random seeds). test rewards are min-max normalised with respect to the smallest and largest, oracle scores in the full dataset, i.e. any scores greater than are greater than any score observed in the full dataset. Design Bench results are shown for illustrative purposes only, and are not directly comparable to our results due to differences in evaluation setup.
31

---

Published in Transations on Machine Learning Research (05/2024)
A.4.4 Hopper (50%)
See Figure $13
-Mieet rwward
-0.5357
Misst -rooard
0.09964
Must reward
0,3893
Micst renard
0.3143
Miest rowerd
p - 0.5286
-20
-250
-20
-20
-20
JMC - DSK
MFD
-Mac
-Mewward
.Alag
W
(a) Correlation plot
Mc
DSM
0,12
Mc DSM
0.12
DSM
0.12
oracle score
1000
oracle score
1000
vald Goi
oracle Score
1000
valie fi4i
batt 50
testir:
800
tee Aut
400
400
400
200
200
200
250
500
750
1000
1250
2.50
500
750
1000
1250
250
500
750
1000
1250
y
y
y
(b) Agreement plot
Figure S13: Results on Hopper 50% using the classificr-free guidance variant (Equation 18
32

---

Published in Transations on Machine Learning Research (05/2024)
A.5 Sensitivity to hyperparameter
In Figure S14 present the same correlation plots shown in Section A.4 but colour-coded with respect. to T, which is the classifier-free guidance hyperparameter that controls the dropout probability for the label y. For each datasct, for the validation metrics that perform the best (i.c. correlate most negatively with the test reward), smaller values of result in better test rewards. Overall, the results indicate that T is a sensitive hyperparameter and should be carefully tuned.
p -0.1262
0.0564
p = -0,0512
f -0.3081
p= -02772
Mosst score
500
450
500
602
600
Mosst
330
300
300
300
200
230
200
200
200
1.0
1.5
2.0
-1.0
-3.3
-1.2
20000
45000
60003
MCc DSK
MFD
-MMc
-MMooor
Mig
(a) Aut Morphology dataset. For each validation metric, we plot each experiment's 's smallest-achievee metric versus
the test reward (Equation 24 The colourbar represents values of 7.
-0.0046
-02299
0,1479
-0.1611
-0.0255
Misst scoe
Mbost sCoe
Mirett score
Muse score
250
250
253
250
Mise
250
200
200
200
200
200
JMc
DSM
MFD
- -MpC
-M Mgooor
Mag
0.05
030
(b) Kitty dataset.
For each validation metric, we plot each experiment's smallest-achieved metric versus the test.
reward (Equation 24
The colourbar represents values of
-0.1421
0.1828
0.4705
-0.1714
p -0.2003
Moert score
Most- score
Musst score
100
Minor-
50-
59
Muss-
50
50
50
3.008
1000
1.0
MC
Mro
-Mc
-Macore
Mag
0.10
6.22
3.54
0.2t
0.18
0.20
(c) Superconductor dataset. For each validation metric, plot each experiment's smallest-achieved metric versus the test reward (Equation 21 The colourbar represents values т.
Figure S14: Results on cach datasct, using the classifier-free guidance variant (Equation 18 Coloured points represent the byperparameter т, which represents the dropout probability for classifier-free guidance (cfg). Across all datasets, smaller values of correspond to better scores. This suggests that experiments are quite sensitive to the value of this hyperparameter.
33

---

Published in Transations on Machine Learning Research (05/2024)
A.6 Additional results
|| Ant Morphology
D'Kitty Morphology
Superconductor
Auto. CbAS
0.364 + 0.014
0.736 + 0.025
0.131 = 0.010
CbAS
0.384 + 0.016
0.753 + 0.008
0.017 = 0.503
BO-qEI
0.567 + 0.000
0.883 + 0.000
0.300 = 0.015
CMA-ES
-0.045 + 0.004
0.684 + 0.016
0.379 + 0.003
Gradient Ascent
0.134 + 0.018
0.509 + 0.200
0.476 0.022
Grad. Min
0.185 + 0.008
0.746 + 0.034
0.471 = 0.016
Grad. Mean
0.187 + 0.009
0.748 0.024
0.469 1 0.022
MINs
0.618
0.887 + 0.004
1 0.016
REINFORCE
0.138 1 0.032
0.356 0.131
1 0.016
COMs
0.519 + 0.026
0.885 + 0.003
0.386 + 0.018
Cond. Diffusion (c.f.g.)
0.831 + 0.052
0.930 + 0.004
0.492 + 0.112
Cond. Diffusion (c.g.)
0.880 + 0.012
0.935 + 0.006
Table $5: 50th percentile test. rewards for methods from Design Bench (Trabucoo et. al. 2022) well as our diffusion results shown in the last two rows, with c.f.g standing for classifier-free guidance (Equation [8] and c.g. standing for classifier-guidane (Equation 16). Each result is an average computed over six different runs (seeds). lest rewards are min-max normalised with respect to the smallest and largest oracle scores in the full dataset, i.c. any scores greater than are greater than any score observed the full dataset. Design Bench results are shown for illustrative purposes only, and are not directly comparable to our results due differences in evaluation setup.
34


Reviews:
[{"review_id": "tPuGjVjJAS", "reviewer": "Reviewer_5yiS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I had a hard time reading the paper and trying to assess the value of its contributions, and I assume I did not properly understand several parts. At first I thought that would be on me and I should dive deeper, but after looking at the code (which the authors claim to be a contribution), I don't have much motivation to do so anymore: the code doesn't come with any (!) instructions or comments how to run it / set it up, and there are also no comments in any of the files explaining what the files are supposed to do.\n\nIt feels the authors made similar efforts when trying to make their paper accessible to a broad set of readers (that do not necessarily have prior knowledge about model-based optimization or the Design Bench) and to clearly explain / show the value of the paper's contributions:\n*) in the experiments section, why is there no discussion of / comparison to a baseline approach that just evaluates f?\n*) Page 5, before Eq. (11): what is y in D_{train}?\n*) in the literature on classifier-based guidance, y is a class label, but here it's a score\n*) Page 5, Eq. (3): expectation and a minus is missing – cf. Eq. (3) from Ho et al. (2020)\n*) In Section 2.3, the framework is designed \"to be consistent with the evaluation setup of Design Bench\", but the Design Bench is only introduced in Section 3.\n*) While the sketches and tables could be helpful, their captions are not self-contained. For example, what does 'min/max?' mean in Table 1 or what is the purpose of the last step in Figure 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper claims to \n*) propose a conceptual evaluation framework for generative models in offline MBO by finding validation metrics that correlate well with the ground truth oracle\n*) apply the framework to diffusion models to find that the agreement metric is most predictive of the test score and that the classifier guidance strength is a most crucial parameter", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "In my opinion the paper needs to be significantly rewritten so that a broad set of readers can follow and it needs to clearly demonstrate the value of its contributions.\n\nFurthermore, several typos need to be fixed. For example:\n-\tBefore Eq. (2), there are unclosed parentheses in D\n-\tInconsistent notation in Eq. (7), the plus should be a comma\n-\tParagraph after Eq. (9): there is an incomplete sentence\n-\tSec. 2.3, third sentence: which is computed ON the held-out validation set \n-\tAlgorithm 1: “values and stored” -> are stored \n-\tSec 4.1: “and no we did not obtain favourable results” – no?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JiRQt9lsWx", "reviewer": "Reviewer_qSSg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Strengths\n\n- Overall, the paper is very well written. I particularly appreciate the color coding, as it helps to disambiguate different terms and to better follow the paper.\n\n- The paper nicely motivates the proposed validation metrics.\n\n- The empirical evaluation overall seems sound.\n\n\n## Weaknesses\n\n\n**Related work**\n\nI would appreciate a more comprehensive discussion on the relationship between offline model-based optimization and the literature on experimental design or Bayesian optimization. Although I acknowledge that the setting differs because we assume no access to the true objective, recent research on transfer Bayesian optimization or zero-shot optimization also explores scenarios where models are initially trained on offline data. I believe that both fields can mutually benefit from each other, and the paper would become stroner by incorporating a discussion on these related areas.\n\n*Pre-trained Gaussian processes for Bayesian optimization*\nZi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, Zoubin Ghahramani\n\n*Towards Learning Universal Hyperparameter Optimizers with Transformers*\nYutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'aurelio Ranzato, Sagi Perel, Nando de Freitas\n\n*Few-Shot Bayesian Optimization with Deep Kernel Surrogates*\nMartin Wistuba, Josif Grabocka\n\n\n\n\n**Inconclusive results**\n\nLooking at Figure 3, the results seem somewhat inconclusive. In certain cases, the validation scores do not correlate significantly with the underlying ground truth. For instance, the MDC exhibits close to zero correlation on the ANT benchmark when used with a cfg model, while displaying a high correlation with a cg model. This inconsistency makes it challenging to rely on these scores for model selection. \n\n**Limited Experiments**\n\nSo far the paper considers only a single model type. I understand that experiments can always be extended, but the computation budget might not allow that. However, at this point it unclear to me how much these validation metric rely under the underlying model approach.\n\n\n**Nitpicks:**\n\n- Equation 5,6 should be max instead of min?\n\n\n**Questions:**\n\n- How sensitivity are the presented results to hyperparameter  \\tau?\n\n- More high level question: I am wondering if also the uncertainty of the model should be taken into account for the validation metric. In other words how certain is the model in it's prediction of the true objective function.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores offline model-based optimization. The objective of this research is to initially train a model using a collection of observations from the objective function of interest. This objective is assumed to be too costly to optimize directly. Instead, the final solution is obtained by optimizing directly model with, for example, gradient descent.\n\nThe main contribution of the paper is to compare different evaluation strategies for model-based optimization approaches when the real objective is inaccessible, and we can only divide the available datasets. Rather than solely validating the model using hold-out validation data, as is typically done in standard supervised learning, the proposed evaluation process emphasizes the accuracy of the model in regions of the input space that are likely to contain the optimum.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper would be strengthened if the authors address the inconclusive results of the experiments. Additionally, I think the paper would benefit if the authors could expand the related work section, as mentioned above.      \n", "broader_impact_concerns": "no ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XL156DI6GV", "reviewer": "Reviewer_1kLh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The proposed approach seems to be practical and well motivated.\n- A reliable validation function can significantly speed up modeling iterations and exploration, as such the problem is worth studying.\n- Results on benchmark experiments show improved final test scores on the considered datasets.\n\n### Weaknesses\n- The paper is not well organized nor well written. It was hard to understand the setup of the paper until I read the whole paper once and re-read again. Figure 1 on the first page for example doesn't add much to the understanding of the reader. In fact, using the notation before introducing it leads to confusion.\n- Validation metrics are scattered across the paper, making the paper hard to follow.\n- The 'transfer' approach while sounding reasonable, is left to future work - even though that is the main premise of this paper. Furthermore, the benefits of the 'transfer' approach in Section 1.1 are very unclear until one has read the whole paper.\n- The novelty in the exploration approach is not clear. It is unclear if the notation $p_\\gamma$ and $p_{0, \\gamma}$ play any role at all in the proposed method. Is this notation introduced for describing the method, or are they actually modeled in the approach?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores various types of validation metrics for model based optimization. Model based optimization involves generating high scoring designs by learning a model on the existing data. This paper focusses on the offline setting operating on a fixed dataset, and generative models are used to train on the data.\n\nFor MBO to be able to successfully generate high scoring examples, the trained model must be able to extrapolate to high score designs while training on a dataset of low score designs. This is achieved by splitting the data set into train, validation and test sets based on the scores. The model extrapolates by replacing the prior of the generative model.\n\nThe validation oracle is trained on the train and test sets, and is used in the validation metrics proposed in the paper. The purpose of validation is to evaluate the extrapolation performance of the model. The model hyper-parameters are selected using the validation metric. The final generated designs are evaluated on the test set for the final score.\n\nThe methods proposed in this paper are tested on datasets from design bench, and the models used are denoising diffusion probabilistic models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A complete algorithm block showing all the steps of the proposed approach is missing. Is the final algorithm as simple as this?\n  1. Train a validation oracle on D_train + D_val\n  2. Train a DDPM model on D_train and use the validation oracle to select the best hyper-parameters.\n  3. Return highest scoring designs using the trained model.\n\n### Suggestions:\n- The organization and flow of the paper needs significant improvement.\n- Related ideas can be clustered together so that it is easier for the reader to compare them and also find them when referring back.\n- Section 1.1 needs significant improvement to provide a strong first impression of the main idea of the paper.\n- Figure 1 could be moved to Section 1.1 for easier understanding.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The primary concerns around the initial version were writing quality, clarity, and organization. This has been largely addressed in the second revision. However, two of the reviewers still lean towards reject, citing inconclusive results and missing related work.

Regarding inconclusive results, I read the paper and I can't quite tell from the discussion whether the point of Table 3 is to showcase the superiority of the approach compared to the baseline approaches, or to provide some form of insight into the framework. As one reviewer points out, it seems to win in 2 out of 4 cases, although it looks quite competitive in the other 2. How much tuning is required to get to this level of performance? Will the conclusions hold across generative models and other tasks? That is, the framework is interesting, but it's not clear what the general takeaway of the results are beyond the limited scope of the models/datasets considered. Some discussion on this would be very helpful.

The other lingering concern centers around related work, although I do see that you cite and discuss 2/3 of Reviewer qSSg's suggested references. I suggest revisiting their comments and ensuring that you fully discuss related work as per their recommendation.

With these in mind, I am still inclined to accept because I believe that the core idea is interesting and that it sufficiently satisfies the requirements for TMLR of supported claims and audience interest.
Decision: Accept (poster)