Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models
Authors: Masatoshi Uehara
Yulai Zhao
Ehsan Hajiramezanali
Gabriele Scalia
Gökcen Eraslan
Avantika Lal
Sergey Levine
Tommaso Biancalani
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion models
Reinforcement learning
Primary area: diffusion_based_models
TL;DR: Conservative Fine-Tuning of Diffusion Models from Offline Data

Abstract: AI-driven design problems, such as DNA/protein sequence design, are commonly tackled from two angles: generative modeling, which efficiently captures the feasible design space (e.g., natural images or biological sequences), and model-based optimization, which utilizes reward models for extrapolation. To combine the strengths of both approaches, we adopt a hybrid method that fine-tunes cutting-edge diffusion models by optimizing reward models through RL. Although prior work has explored similar avenues, they primarily focus on scenarios where accurate reward models are accessible. In contrast, we concentrate on an offline setting where a reward model is unknown, and we must learn from static offline datasets, a common scenario in scientific domains. In offline scenarios, existing approaches tend to suffer from overoptimization, as they may be misled by the reward model in out-of-distribution regions. To address this, we introduce a conservative fine-tuning approach, BRAID, by optimizing a conservative reward model, which includes additional penalization outside of offline data distributions. Through empirical and theoretical analysis, we demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs through pre-trained diffusion models.

Bridging Model-Based Optimization and Generative Modeling via Conservative Fine- Mone-Tuning of Diffusion Models
Masatoshi Uehara 1- Yulai Zhao 2
Gabriele Scalia Sergey Levine Tommaso Biancalani '
Ehsan Hajiramezanali
Gokcen Eraslan
Avantika Lal
Genentech
SUC Berkeley ?Princeton University
Abstract
Al-driven design problems, such as DNA/protein sequence design, are commonly tackled from two angles: generative modeling, which efficiently captures the feasible design space (e.g., natural images or biological sequences), and model- based optimization, which utilizes reward models for extrapolation. To combine the strengths of both approaches, we adopt a hybrid method that fine-tunes cutting-edge diffusion models by optimizing reward models through RL. Although prior work has explored similar avenues, they primarily focus on scenarios where accurate reward models are accessible. In contrast, we concentrate on an offline setting where a reward model is unknown, and we must learn from static offline datasets, a common scenario in scientific domains. In offline scenarios, existing approaches tend to suffer from overoptimization, as they may misled by the reward model in out-of-distribution regions. To address this, we introduce conservative fine- tuning approach, BRAID, by optimizing a conservative reward model, which includes additional penalization outside of offline data distributions. Through empirical and theoretical analysis, we demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs through pre- trained diffusion models. The main code is available at coom ma saeasssaueeeeeee we/.eeRff... www.e/..eee....nnggoo...m....... fus [tcoooonoossseee__
Introduction
Computational design involves synthesizing designs that optimize particular reward function. This approach finds applications in various scientific domains, including DNA/RNA/protein design (Sample al. 2019 Gosai al.) 2023 Wu et al. 2024). While physical simulations often used in design problems, lacking extensive knowledge of underlying physical processes necessitates solutions that solely rely on experimental data. In these scenarios, we need an algorithm that synthesizes an improved design by utilizing dataset of past experiments (i.e., static offline dataset). Existing research has addressed computational design from two primary angles. The first angle is generative modeling such as diffusion models (Ho et al. 2020), which aim to directly model the distribution of valid designs by emulating the offline data. This approach allows us to model the space of "valid" designs (e.g., natural images, natural DNA sequences, foldable protein sequences (Avdeyev et al.) 2023)). The second angle offline model-based optimization (MBO), which entails learning the reward model from static offline data and optimizing it with respect to design inputs (Brookes et al.)
2019
Trabucco et al. 2021 Angermueller al. 2019 Linder and Seelig 2021 Fannjiang and
"gqual contribution: uehara.. www.maattssiii..oommmmmmomoo (gene com, oot.aeme..ccoo......geeeettooo..
Corresponding authors: svlevine(eecs. berkeberkk...... bew.eley.edu..........e. www..ebiaaaaaaaagen....
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Penalties
Offiine data fx y Oos
Doubly conservative diffusion models
Whole design space
High
Fine-tuning
Valid design space
Valid designs
Pre-trained diffusion models
Low
The left figure illustrates our setup with pre-trained generative model and offline data. On the right, the motivation of the algorithm is depicted. The region surrounded by the green line is the original entire design space, with the colored region indicating the valid design space (e.g.- natural images, human-like DNA sequences). The red region denotes areas with more offline data available, while the blue region indicates areas with less data available. We aim to add penalties to the blue regions using conservative reward modeling to prevent overoptimization while imposing a stricter KL penalty on the non-colored regions to prevent the generation of invalid designs.
Figure 1:
Listgarten, 2020
2022). This class of methods potentially enables us to surpass the best design observed in the offline data by harnessing the extrapolative capabilities of reward models.
Chen et al.
In our work, explore how the generative modeling and MBO perspectives could be reconciled, inspired by recent work on RL-based fine-tuning of diffusion models (e.g., Black et al. (2023); Fan al. 2023)), which aims finetune diffusion models by optimizing down-stream reward functions. Although these studies do not originally address computational design, we can potentially leverage the strengths of both perspectives. However, these existing studies often focus on scenarios where online reward feedback can be queried or accurate reward functions are available. Such approaches are not well-suited for the typical offline setting, where we lack access true reward functions and need to rely solely on static offline data (Levine et al.) 2020 Kidambi et al. 2020 Yu et al. 2020). In scientific fields, this offline scenario is common due to the high cost of acquiring feedback data. In such contexts, existing works for fine-tuning diffusion models may easily lead to overoptimization, where optimized designs are misled by the trained reward model from the offline data, resulting in out-of -i-tr-ddistribttion adversarial designs instead of genuinely high-quality designs.
To mitigate overoptinization, we develop a conservative fine-tuning approach for generate models aimed at computational design. Specifically, we consider critical scenario where we have offline data (with feedback) and pre-trained diffusion model capable of capturing the space of "valid" designs, and propose a two-stage method (Figure T.. In the initial stage, we train a conservative reward model using offline data, incorporating an uncertainty quantification term that assigns higher penalties to of--is-iiituttionnn regions. Subsequently, we finetune pre-trained diffusion models by optimizing the conservative reward model obtain high-quality designs and prevent the generation of out-of -ut-of-distribution designs. In the fine-tuning process, we also introduce KL penalization term to ensure that the generated designs remain within the valid design space.
Our primary contribution lies in the introduction of a novel framework, BRAID (douBly conseRvA- tive fine-tuning diffuslon moDels). The term "doubly conservative" reflects the incorporation of two types of conservative terms, both in reward modeling and KL penalization. By properly penalizing the fine-tuned diffusion model when it deviates significantly from the offline data distribution, we effectively address overoptimization, Additionally, by framing our fine-tuning procedure within the context of soft-entropy regularized Markov Decision Processes, we offer theoretical justification for the inclusion of these conservative terms in terms of regret. This theoretical result shows that fine- tuned generative models outperform the best designs in the offline data, leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs. Furthermore, through empirical evaluations, we showcase the efficacy of our approach across diverse domains, such as DNA/RNA sequences and images.

---

2
Related Works
We summarize related works. For additional works such as fine-tuning on LLMs, refer to Section A
Fine-tuning diffusion models via reward functions. Several previous studies have aimed to improve diffusion models by optimizing reward functions using various methods, including supervised learning (Lee et al. 2023] Wu et al. 2023), RL (Black et al., 2023 Fan et al. 2023) and control- -oased techniques (Clark et al. 2023 Xu et al. 2023 Prabhudesai et al. 2023). In contrast to work, their emphasis is not on an offline setting, i.c., their setting assumes online reward feedback is
available or accurate reward functions are known. Additionally, while Fan al. (023) include the KL term in their algorithms, our innovation lies in integrating conservative reward modeling to mitigate overoptimization and formal statistical guarantees in terms of regret (Theorem
Conditional diffusion models. Conditional diffusion models, which learn conditional distributions of designs given the rewards, have been extensively studied (Ho and Salimans) 2022 Dhariwal and Nichol, 2021] Song et al. 2020) [aassal et al. 2023). However, for the purpose of MBO, these approaches require that the offline data has good coverage values we want to condition (Brandfonbrener et al., 2022). Compared to conditional diffusion models, our approach aims to obtain designs that can surpass the best design in offline data by leveraging the extrapolation capabilities of reward models. We compare these approaches with our work in Section 7
Offline model-based optimization (MBO). Offline MBO is also known as offline black- optimization and is closely related to offline contextual bandits and offline RL (Levine et al. 2020)
While conservative approaches have been studied there (e.g., Kidambi al. (2020); Yu
(2020)
and more in Section A); most of the works are not designed to incorporate a diffusion model, unlike our approach. Hence, remains unclear how these methods can generate designs that remain within intricate valid design spaces (e.g., generating natural images).
It worth noting few exceptions (Yuan al. 2023 Krishnamoorthy et 2023) that attempt to integrate diffusion models into MBO. However, the crucial distinctions lie in the fact that we directly optimize rewards with diffusion models, whereas these prior works focus using conditional diffusion models. Additionally, we delve into the incorporation of conservative terms, an aspect not explored in their works. We compare these methods with ours empirically in Section 7
3 Preliminaries
We outline our framework for offline model- basell-ased optimization with a pre-trained generative model. Subsequently, we highlight the challenges arising from distributional shift. Additionally, we provide an overview of diffusion models, as we will employ them as pre-trained generative models.
3.1 Offline Model-Based Optimization with Pre- rre-Trained Generative Model
Our objective is to find high-quality design within design space, X. Each design E X' is associated with reward, r(x), where r X + [0 is an unknown reward function. Then, our aim is to find high-quality generative model p E A (X)), that yields a high r(z). It is formulated as
argmaxpea(x)) Ex-p]r(x)).
Avoiding invalid designs. In MBO, the design space X' is typically huge. However, in practice, the valid design space denoted by Xpre is effectively contained within this extensive X' as a potentially lower-dimensional manifold. For instance, biology, our focus often centers around discovering highly bioactive protein sequences. While the raw search space might encompass |20] possibilities (where the length), the actual design space corresponding to valid proteins significantly more constrained. Consequently, our problem can be formulated as:
argmax Exmp(((x)] (eqivaletnly, argmax [x=p[[(x)] Expp [[x & Xpre))).
(2)
PE (Xxpe)
pEA(X)
Note supposing that reward r(-) is 0 outside of Xpres this is actually still equivalent
3

---

Offline data with pre-trained generative model. Based on the above motivation, we consider scenarios where we have an offline dataset Doft. used for learning the reward function. More specificallly, the dataset, Dott- = {x(0) y yosf contains pairs of designs æ Poft and their associated noisy reward feedbacks y = r(x) c, where is noise.
Compared to settings in many existing pupers on MBO, we also assume access to a pre-trained generative model (diffusion model) trained on large dataset comprising valid designs, in addition to the offline data Doftt For example, in biology, this is expected to capture the valid design space Xpre such human DNA sequences or physically feasible proteins (Avdeyev et al., 2023 Li al. [024]] Sarkar et al.) 2024 Stark et al.) 2024 Campbell et al. (024). These pre-trained generative models are anticipated be beneficial for narrowing down the raw search space X' to the design space Xpre. In our work, denoting the distribution induced by the pre-trained model by Ppre. we regard the support of Pere Xpre.
3.2 Challenge: Distributional Shift
To understand our challenges, let's first examine a simple approach for MBO with a pre-trained gencrative model. For instance, can adapt methods from Clark et al. (202)); Prabhudesai et al. (2023) to our scenario. This approach involves two steps. In the first step, we perform reward learning: argmin; Di Boff (f(x(()) y0 where FF represents function class that includes mappings from X' to [0, 11, aiming capture the true reward function r(-). Then, in the second step, we fine-tune a pre-trained diffusion model to optimize i.
from|Clark
Despite its simplicity, this approach faces two types of distributional shifts. Firstly, the fine-tuned generative model might produce invalid designs outside of Xpre- As discussed in Section 3.1 we aim to prevent this situation. Secondly, the fine-tuned generative model may over-optimize r. exploiting uncertain regions of the learned model A. Indeed, in regions not covered by offline data distribution Poff. the learned reward f can easily have higher values, while the actual reward values in terms of might be lower due to the higher uncertainty. We aim avoid situations where we are misled by out-to- -odittiiiiiiitoonnnn adversarial designs.
3.3 Diffusion Models
We present overview of denoising diffusion probabilistic models (DDPM) (Song et al. 2020
Ho et al. 2020 Sohl-Dickstein et al. 2015). Note while the original diffusion model was initially introduced in Euclidean spaces, it has since been extended to simplex spaces for biological sequences (Avdeyev et al. 2002)), which we will use in[Section 7] In diffusion models, the goal is to develop a generative model that accurately emulates the data distribution from the dataset. Specifically, denoting the data distribution by Pure E (x)), a DDPM aims to approximate using a parametric model structured as p(zo; 0) = p(xo:T; 1)dzI.7, where p(zo:T; 0) (IT;; 0) II' Pe 0). Here, each pr is considered as a policy, which a mapping from a design space X to a distribution over X. By optimizing the variational bound on the negative log- likelihood, we can obtain a set of policies (ptli=t+1 such that p(r0; 0) = (Zu) For simplicity, in this work, assuming that pre-trained diffusion models are accurate, we denote the pre-trained policy as (p.""(()-)))_rrrr)) and the generated distribution by the pre-trained diffusion model at mi by Ppre- With slight abuse of notation, we often denote pr+1 (-) by pT+1(113
Doubly Conservative Generative Models
We've ve discussed how naive approaches for computational design may yield invalid designs or optimize reward functions, with both challenges stemming from distributional shift. Our goal in this section is develop doubly conservative generative models mitigate this distributional shift.
4.1 Avoiding Invalid Designs
To avoid invalid designs, we begin by considering the following generative model:
cxp(((()/a)ppee( (== argmax Bxmplf(x)) (KL(pplppee)),
(xxp(f())a))ppee (z)da
PEA(X)

---

where (L(pllppre) Ex as an optimizer of loss function composed of two parts: the first component encourages designs with high rewards, while the second component acts as regularizer penalizing the generative model for generating invalid designs. This formulation is inspired by our initial objective (2), where we substitute an indicator function with log(p/Ppn)) P.ree. This regularizer takes co when p is not covered by Ppre. and a governs the strength of the regularizer.
In this formulation, the generative model is designed
4.2 Avoiding Overoptimization
Next, we address the issue of overoptimization. This occurs when we are fooled by the learned reward model in uncertain regions. Therefore, natural approach is to penalize generative models when they produce designs in uncertain regions.
As first step, let's consider having an uncertainty oracle 9 X + [0, 1]. which is random variable of Doft. This oracle expected to quantify the uncertainty of the learned reward function f.
Assumption (Uncertanity oracle). With probability 1 1 8, we have
Ve E Xprei [f(1)) r(x) S ((x)
(4)
These calibrated oracles are well-established when using variety of models such as linear models, Gaussian processes, and neural networks. We will provide detailed examples of such calibrated oracles in Section 4.3,. Essentially, as long as the reward model is well-specified (i.e., there exists i E JF such that Vr Xpre F(x) = r(x)), we can create such calibrated oracle.
Doubly Conservative Generative Models.
Utilizing the uncertainty oracle defined in Assump-
tion I present proposal:
exp ((f - (-)/)a))))))))))) (-)
(+)+)
(:= argmax BagiaxxEepp Be (x-maxx)--p((())()))))))) aKL(pll (5)
J exp ((f - (x)) Pere (z) dx
reward
Penalty
Here, combat overoptimization, expected to prevent n. from venturing into regions with high uncertainty because it would take a higher value in such regions. We refer f.. doubly conservative generative model due the incorporation of two conservative terms.
introduce additional penalty term g(z). This penalty term is
An attentive reader might question the necessity of simultaneously introducing two conservative terms. Specifically, the first natural question is whether KL penalties, intended to prevent invalid designs, can replace
penalties. However, this may not hold true because even if we can entirely avoid venturing outside of Xpre (support of Ppre), we may still output designs on uncertain regions not covered by Pat. The second question is whether uncertainty ceaetty-orrle-bbssse be-daeed penalties can substitute KL penalties. While it is partly true in situations where the support of Pot is contained within that of Ppre. uncertainty oracle-based penalties, lacking leverage on pre-trained generative models, are ineffective in preventing invalid designs. In contrast, KL penaltics are considered a more direct approach stringently avoid invalid designs by leveraging pre-trained generative models.
4.3 Examples of Uncertainty Oracles
Example 1 (Gaussian processes.). When we use an RKHS as F (a.k.a GPs) associated with a kernel k(,-): X x X + R Srinivas et al. 2009), typical construction of f and 9 is
f(.) Y(K +
8(-) c(8) R(.,-),
where c(8) E R R k(x) = [k((x( 1),,), x), ((x ,x)T
Y= , ,0,....,,,((u... (,)n)) (K) hp.g- = k(z(()) ,,1)), k(x, x') = k(xx, k(x)T (K + AT) x(=")
Note that when using deep neural networks, by considering the last layer as a feature map, we can still create a kernel (Zhang et al. 2022 Qiu et al. 2022)
Example 2 (Bootstrap). When we use neural networks as F, it is common to use a statistical bootstrap method. Note many variants have been proposed (Chua et al. 2018), and its theory has been analyzed (Kveton et al. 2019). Generally, our context, we generate multiple models fi, fM by resampling datasets, and then consider argmin, fqz as f g.
5

---

Algorithm BRAID (douBly conseRvAtive flne-tuning Diffusion models)
1: Require: Parameter E RR,, set of policy classes {II}} where II, [X A(X),, pre-trained
diffusion model {pf 1pptee 11 T1=7+1'
2: Train conservative reward model f using offline data Doft-
3: Update a diffusion model as [ft] by solving the planning problem:
(prh =
argmax ---EI---TI1
(pp)) - ](zo)] a - 1 (KL (1) (6)
Penalized reward
KL penalty
where the expectation (p..) [-] is taken with respect to II II 4: Output: A policy {prtt
Conservative Fine-tuning of Diffusion Models
In this section, consider how sample from doubly conservative generative model R.. using diffusion models as pre-trained generative models. Our algorithm is outlined in Algorithm [ Initially, we learn penalized reward F-9 from the offline data and set it as target to prevent overoptimization in (6). Additionally, we integrate a KL regularization term to prevent invalid designs. The parameter governs the intensity of this regularization term.
Formally, this phase can be conceptualized as a planning problem in soft- seef-entrooy--egulaatt MDPs (Neu et al. 2017: Geist et al. 2019). In this MDP formulation:
The state space S and action space A correspond to the design space X'
The reward at time / E [0, (0, _) (E S x A + R) is provided only at T as i g.
The transition dynamics at time 1 (E [S x A + (SS) is an identity 5(84+1 = az).
The policy at time (E S + (AA) corresponds to PT+ X - (Xx).
The reference policy at is policy in the pre-trained model p
In these entropy-rees regularzeed MDPs, the soft optimal policy corresponds to {p:]. Importantly, we can analytically derive the fine-tuned distribution in in/lggrithm aand show that it simplifies to a doubly conservative generative model a.. from which we aim to sample.
Theorem 1. Let p.(-) be an induced distribution from optimal policies {Pe)1-441 in (6) i.e. p(xx) = Then,
when {II,} a global policy class (II, = {X' A(X))
(x) = aa (z).
We have deferred to the proof in Section
While similar results are known in the context of standard entropy regularized RL. (Levine 2018 theorem is novel because previous studies did not consider pre-trained diffusion models.
5.1
Training algorithms. Based on Theorem Equation (6). We can employ any off-the-shelf RL algorithms to solve this planning problem. Given that the transition dynamics are known, and differentiable reward models are constructed in our scenario, a straightforward approach to optimize ( is to directly optimize differentiable loss functions with respect to parameters of neural networks in policies, as detailed in Appendix B Indeed, this approach has recently been used in fine-tuning diffusion models (Clark et al. 2023 Prabhudesai jet al. 2023), demonstrating its stability and computational efficiency.
sample from a.. what we need to to solve
Remark 1 (Novelty of Theorem Ip. A theorem similar to Theorem I Thas been proven for continuous- time diffusion models in Euclidean space (Uehara et al. 2024 Theorem 1). However, the primary distinction lies in the fact that while their findings are restricted to Euclidean space, where diffusion policies take Gaussian polices, our results are not construined to any specific domain. Hence, for example, our Theorem can handle scenarios where the domain discrete or lies on the simplex space (Avdeyev et al. 2023) in order model biological sequences do in Section 7

---

5.1 Sketch of the Proof of Theorem 1
We explain the sketch of the proof of Theorem The detail deferred to Theorem C.I
By induction from to T + 1, we can first show
(c(x(-1(zt) cxp(00-((tt-1//appf11 (
(7)
exp(vr(za)/aa
Here, (x(2 is a soft optimal value function:
Ep(((zo) - ((vu) - a (
Arunt
which satisfies an equation analogous to the soft Bellman equation: vo(x:) = -f(x)) ((()) and for t to +
(T(
(e-1((a-)) a
exp(
exp
pe (wx-1 xt dr)dd
(8)
Now, we aim to calculate a marginal distribution at defined by (x(
Hilkk- (2) Then, by induction, we can show that
(r(),) exp(v, (z,) ()))
(9)
where C is normalizing constant. Indeed, supposing that the above hold at t, the equation also holds for - as follows:
(
f p
(eppoo-1a///////00011001011 (x-1)))
Finally, by setting t = 0, the statement in Theorem is concluded.
Regret Guarantee
In this section, objective is demonstrate that policy P.. from Algorithm
provably
outperform designs in offline data by establishing the regret guarantee.
To assess the performance of our fine-tuned generative model, we introduce the soft-value metrie: [r(z)) - (Plpppllppee)
(p) [x=))r(z))
This metric comprises two components: the expected reward and penalty term applied when produces invalid outputs, as we see in (3) Now, in terms of soft-value (p). our proposal pa offers the following guarantee.
Theorem (Per-step regret). Suppose Assumption ] Then, with probability - 6, we have
π T(x)
C. = max (//2) VT E A(X); Ja(a) (p(a) < 2 2 Cr x [
Pos (x)
Per step regret
Star
where Noil = fo E X Pott (x) > 0}. As an immediate corollary,
Ex-x(r(x)) Ezap. [r(x)] S aKL((T) + 2/C, Exppon [
In the theorem above, establish that the per-step regret against generative model aim to compete with is small as long as the generative model π falls within Xai and the learned model is calibrated in Assumption T] First, the term (Stat) corresponds with over the offline data distribution Poft- When the model is well-specified, it is upper-bounded
the statistical error associated
by vd/n, n. where d represents the effective dimension of F, we will discuss shortly. Secondly, the term C₂ corresponds to the coverage between comparator generative model and our generative model Pa. Hence, it indicates that the performance of our learned Pa. is at least as good as that of comparator generative model π covered by Pott- While this original coverage term C₂ diverges when goes outside of X,#, we can refine it using the extrapolation capabilities of function class F, as we will discuss shortly. This refined version ensures that we can achieve high-quality designs that outperform designs in the offline data (i.c., best designs in X
7

---

Example 3. We consider a scenario where an RKHS is use F. Let F be a model represented by
an Sefinitedimmeniioaall c(-). Let d denote the effective dimension of F /Valko et al. 2013 3)
Corollary 1 (Informal: Formal characterization is in Section D ) Assuming that the model is well-specified, with probability 1 - 8, we have:
R. En (E (x)]
Ja(T) Ja(Pa) S Vxx
Cx:= =
sup K: [x||2=1 K T (0(x))))))))))(() (x)]K²
The refinement of the coverage term in C. is characterized as the relative condition number between covariance matrices on generative model π and an offline data distribution Pott, which is smaller than C.. This CN could still be finite even if C' is infinite. In this regard, Corollary illustrates that the trained generative model can outperform the best design in the offline data by harnessing the extrapolation capabilities of reward models.
7 Experiments
We perform experiments to evaluate (a) the effectiveness of conservative methods for fine-tuning diffusion models and (b) the comparison of our approach between existing methods for MBO with diffusion models (Krishnamoorthy et al., 2023 Yuan et al. 2023). We will start by outlining the baselines and explaining the experimental setups. Regarding more detailed setups, hyperparameters, architecture of neural networks, and ablation studies, refer to Appendix
Methods compare. We compare the following methods in our evaluation. For fair comparison, we always use the same a in BRAID and STRL.
BRAID (proposed method): We consider two approaches: (1) Bonus, as in Example by setting a last layer as a feature map and constructing kernel, (2) Bootstrap, as in Example 2
Standard RL (STRL): EL-fine-tuning that optimizes the standard without any conservative term, following existing works on tee-tunnng (Clark et al. [023 Prabhudesai et al., 2023).
DDOM (Krishnamoorthy et al. 2023): We train with weighted classifier-free guidance (Ho and Salimans 2022) using offline data, conditioning on a class with high y values (top 5%) during inference. Note that this method is training from scratch rather than fine-tuming.
Offline Guidance (Yuan et al., 2023): After training a classifier using offline data, we use guidance (conditional diffusion models) (Dhariwal and Nichol 202T on top of pre-trained diffusion models and condition on classes with high y values (top 5%) at inference time.
Evaluation. We assess the performance of each generative model primarily by visualizing the histogram of true rewards r(z) obtained from the generated samples. For completeness, we include similar histograms for both the pre-trained model (Pretrained) and the offline dataset (Offline).
As for hyperparameter selection, such as determining the strengths of conservative terms/epochs, we adhere to conventional practice in offline RI. (e.g., Rigter et al. (2022); Kidambi et al. (2020); Matsushima et al. (2020) and choose the best one through limited number of online interactions.
We omit comparisons with pure MBO methods for two reasons: (i) DDOM, which we compare against, already demonstrates a good performance across multiple datasets, and (ii) these methods are unable to model complex valid spaces since they do not incorporate state-of-the-an generative models (e.g., stable diffusion)) thereby lacking the capability to generate valid designs (e.g., natural images) as we show in Section 7.2
Remark 2.
7.1 Design of Regulatory DNA/RNA Sequences
We examine two publicly available large datasets consisting of enhancers (n 700k) (Gosai et al.. 2023) and UTRs (n z 300k) (Sample et al. 2019) with activity levels collected by massively parallel
reporter assays (MPRA) (Inoue al. 2019). These datasets have been extensively used in sequence
'Regarding the effectiveness of KL- KL-regularization, it has been discussed in Fan et al. 2023) Uchara et al. 2024). Hence, in work, we focus on the effectiveness of conservatism in reward modeling.

---

Hebiol
( S'URS
b Enhancers
(c) Images
Figure 2: Barplots of the rewards r(x) for samples generated by each algorithm. It reveals that proposals consistently outperform baselines.
offine
STRL SRAID Bucl SRADD doeen
don 826
Reweard Charrics
10000
Benus
honee 19.10
montey 1.26
(b) Training curves in terms of f
Undesirable scenarios
(but not r))
(c) Generated images
Figure 3: Results on Image Generation
optimization for DNA and RNA engineering, particularly for the advancement of cell and RNA therapy (Castillo-Hair and Seelig, 2021 Ghari et al., [2233] Lal al. 2024 Ferreira DaSilva et al.) 2024). In the Enhancers dataset, each x is DNA sequence with length of 200, while E R is the measured activity in cell lines. For the UTRs dataset, 50 while E IR is the mean ribosomal load (MRL) measured by polysome profiling.
5'UTR RNA sequence with length of
Setting of oracles and offline data. We aim to explore a scenario where we have a pre-trained model and offline dataset. Since the true reward function r(-) is typically unknown, we initially divide the original dataset D (x) (f) randomly into two subsets: Don and D'. Then, from D', we select datasets below 95% quantiles for enhancers and 60% quantiles for UTRs and define them as offline datasets Dont. Subsequently, we construct an oracle r(.) by training neural network Don and use it for testing purposes. Here, we use an Enformer-basee model, which is a state-of-the-ee model for DNA sequences (Avsec al. 2021). Regarding pre-trained diffusion models, we customized for sequences over simplex space (Avdeyey et al. (023). In the subsequent analysis, each algorithm solely has access the offline data Dorff and pre-trained diffusion model, but not r(-).
Results. The performance results in terms of r(.) are depicted in Fig 2 and b. It is seen that fine-tuned generative models via RL outperform conditioning-baat methods: DDOM and Guidance. This is expected because conditional models themselves are not originally intended to surpass the conditioned value (Rs best value in the offline data). Conversely, fine-tuned generative models via RL capable of exceeding the best value in offline data by harnessing the extrapolation capabilities of reward modeling, as also theoretically supported Corollary I Secondly, both BRAID-boot and BRAID-bonie demonstrate superior performance compared to STRL suggests that con- servatism aids in achieving fine-tuned generative models with enhanced rewards while mitigating overoptimization.
7.2
Image Generation
We consider the task of generating aesthetically pleasing images, following prior works (Fan et al. 2023 Black al. 2023). We use Stable Diffusion v1.5 as our pretrained diffusion model, which generate high-quality images conditioned prompts such as "cat" and "dog". ". We use the AVA

---

dataset MMurry et al. 2012) as our offline data and employ linear MLP on top of CLIP embeddings train reward models and f 9) from offline data for fine-tuning.
Setting of oracles. To construct r(æ), following existing works, we use the LAION Aesthetic Predictor V2 (Schuhmann] 2022), already pre-trained on a large-scale image dataset. However, this LAION predictor gives high scores even if generated images are almost identical regardless of prompts, as Figure 3b] These situations are undesirable because it means fine-tuned models are too far away from pre-trained models. Hence, for our evaluation, we define r(z) as follows: (1) asking vision language models (c.g.. LLaVA (Liu et al., 2024)) whether images contain objects in the original prompts (c.g., dog, cat), (2) if Yes, outputting the LAION predictor, and (3) if No, assigning This evaluation ensures that high r(z) still indicates capturing the space the original stable diffusion.
We show that our proposed approach outperforms the baselines in terms of r(x), as in 2 $ We also show the generated images in [ Additionally, plot the training curve during the fine-tuning process in terms of the mean of r(x) of generated samples in Fig Beb. The results indicate that in STRL, while the learning curve based on the learned reward quickly grows, fine-tuned models no longer necessarily remain within the space of pre-trained models (STRL in Fig 3 9. In contrast, in our proposal, by carefully regularizing on regions outside of the offline data, we can generate more aesthetically pleasing images than STRL, which remain within the space of pre-trained models. For more images/ ablation studies, refer to Appendix
Results.
Summary
For the purpose of fine-tuning from offline data, we introduced a conservative fine-tuning approach by optimizing a conservative reward model, which includes additional penalization outside of offline data distributions. Through empirical and theoretical analysis, we demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs through pre-trained diffusion models.
References
Angermueller, C., D. Dohan, D. Belanger, R. Deshpande, K. Murphy, and L. Colwell (2019). Model- based reinforcement learning for biological sequence design. In International conference learning representations.
Avdeyev, P., C. Shi, Y. Tan, K. Dudnyk, and J. Zhou (2023). Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, pp. 1276-1301. PMLR.
Avsec, 2., V. Agarwal, D. Visentin, J. R. Ledsam, A. Grabska-Barwinske K. R. Taylor, Y. Assael, Jumper, Kohli, and D. R. Kelley (2021). Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods 18(10), 1196-1203.
Bansal, A., H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein (2023). Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 843-852.
Black, K., M. Janner, Y. Du, I. Kostrikov, and S. Levine (2023). Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.
Brandfonbrener, D., Bietti, J. Buckman, R. Laroche, and J. Bruna (2022). When does return- conditioned supervised learning work for offfine reinforcement learning? Advances in Neural Information Processing Systems 35, 1542-1553.
Brookes, D., H. Park, and J. Listgarten (2019). Conditioning by adaptive sampling for robust design. In International conference machine learning, pp. 773-782. PMLR.
'The score LLaVA for detecting objects was L.0 as detailed Appendix E.2.3 Hence, this part considered to be accurate.
Note omitted DDOM because it is fine-tuning method.
10

---

Campbell, A., J. Yim, R. Barzilay, T. Rainforth, and T. Jaakkola (2024). Generative flows on discrete state-spaces: Enabling multimodal flows with applications protein co-design. arXiv preprint arXiv:2402.04997.
Castillo-Hair, S. M. and G. Seelig (2021). Machine learning for designing next-generation mrna therapeutics. Accounts of Chemical Research 55(1), 24-34.
Chang. J. D., M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun (2021). Mitigating covariate shift in imitation learning via offline data without great coverage. arXiv preprint arXiv:2106.03207.
Chen, C., Y. Zhang, J. Fu, X. S. Liu, and M. Coates (2022). Bidirectional learning for offline infinite-width model-based optimization. Advances in Neural Information Processing Systems 35, 29454 29467.
Chua, K., R. Calandra, R. McAllister, and S. Levine (2018). Deep reinforcement learning handful of trials using probabilistic dynamics models. Advances in neural information processing systems 31.
Clark, K., P. Vicol, K. Swersky, and D. J. Fleet (2023). Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.1 17400.
Dhariwal, and A. Nichol (2021). Diffusion models beat gans image synthesis. Advances in neural information processing systems 34, $780-8794,
Fakoor, R., J. W. Mueller, K. Asadi, P. Chaudhari, and A. J. Smola (2021). Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems 34, 11260-11273.
Fan, Y., O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee (2023). Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint ar3iv:2305.16381
Fannjiang, C. and Listgarten (2020). Autofocused oracles for model-based design. Advances in Neural Information Processing Systems 33, 12945-12956.
Ferreira DaSilva, L.. S. Senan, Z. M. Patel, A. J. Reddy, S. Gabbita, Z. Nussbaum, C. M. V. Cordova, A. Wenteler, N. Weber, Tunjic, et (2024). Dna-diffusion: Leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements. bioRxiv, 2024-02.
Geist, M., B. Scherrer, and O. Pietquin (2019). theory of regularized markov decision processes. In International Conference on Machine Learning, pp. 2160-2169, PMLR.
Ghari, P. M., A. Tseng, G. Eraslan, R. Lopez, T. Biancalani, G. Scalia, and E. Hajiramezanali (2023). Generative flow networks assisted biological sequence editing. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop.
Gómez-Bone Bombarelli, R., J. Wei, Duvenaud, J. M. Heránndez-Lobato B. Sänchez-Lengeling, D. Sheberla, J. Aguilela--apppe Iparraguiree, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzit (2018) 8). Automatic chemical design using a data-driven continuous representation of molecules. ACS central science 4(2), 268-276.
Gosai, J., R. I. Castro, N. Fuentes, J. C. Butts, S. Kales, R. R. Noche, K. Mouri, P. C. Sabeti, S. K. Reilly, and R. Tewhey (2023). Machine -uiieed design of synthetic cell type-specific cis-regulatory elements. bioRxiv.
Ho, J., A. Jain, and P. Abbeel (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840-6851.
Ho, J. and T. Salimans (2022). Classifier-frre diffusion guidance. arXiv preprint arXiv:2207.12525
Inoue, F., A. Kreimer, T. Ashuach, N. Ahituv, and N. Yosef (2019). Identification and massively parallel characterization of regulatory elements driving neural induction. Cell stem cell 25(5), 713-727.

---

Kidambi, R., A. Rajeswaran, P. Netrapalli, and T. Joachims (2020). Morel: Model- based offline reinforcement learning. Advances in neural information processing systems 33, 21810-21825
Kong, L., Y. Du, W. Mu, K. Neklyudov, V. De Bortol, H. Wang, D. Wu, A. Ferber, Y.-A. Ma, C. P. Gomes, et al. (2024). Diffusion models as constrained samplers for optimization with unknown constraints. arXiv preprint arXiv:2402. 18012.
Krishnamoorthy, S., S. M. Mashkaria, and A. Grover (2023). Diffusion models for black-box optimization. arXiv preprint arXlv:2306.07180
Kumar, A., A. Zhou, G. Tucker, and S. Levine (2020). Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems 33, 1179-1191.
Kveton, B., C. Szepesvari, S. Vaswani, Z. Wen, T. Lattimore, and M. Ghavamzadeh (2019). Garhage in, reward out: Bootstrapping exploration in multi-armed bandits. In International Conference on Machine Learning, pp. 3601-3610. PMLR.
Lal, A., Garfield, T. Biancalani. and Eraslan (2024). regim: Designing realistic regulatory dna with autoregressive language models. bioRxiv, 2024-02.
Lee, K., H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, Abbeel, M. Ghavamzadeh, and S. Gu (2023). Aligning text-to-image models using human feedback. arXiv preprint 22w:2202.///99.
Levine, S. (2018). Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint
Levine, S., A. Kumar, G. Tucker, and J. Fu (2020). Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint wwXiv:2005.01644
Li, Z., Y. Ni, W. A. Beardall, G. Xia, A. Das, G.-B. Stan, and Y. Zhao (2024). Discdiff: Latent diffusion model for dna sequence generation. arXiv preprint arXiv:2402.06077)
Linder, J. and G. Seelig (2021). Fast activation maximization for molecular sequence design. BMC bioinformatics 22, 1-20.
Liu, H., C. Li, Q. Wu, and Y. J. Lee (2024). Visual instruction tuning. Advances in neural information processing systems 36.
Matsushima, T. H. Furuta, Y. Matsuo, O. Nachum, and S. Gu (2020). Deployment-efficient rein-
forcement learning via model- based offline optimization. arXiv preprint arXiv:2006.036677
Murray, N., L. Marchesotti, and F. Perronnin (2012). Ava: A large-scale database for aesthetic visual analysis. 2012 IEEE conference on computer vision and pattern recognition, pp. 2408-2415. IEEE.
Neu, G., A. Jonsson, and V. Gómez (2017). A unified view of entropy-regularizzd markov decision processes. arXiv preprint 1rXiv:170..07799
Notin, P., J. M. Heránndez-Lobato, and Y. Gal (2021). Improving black-box optimization vae latent space using decoder uncertainty. Advances in Neural Information Processing Systems 34, 802-814.
Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, 27730-27744.
Paine, L., C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Fre- itas (2020). Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055.
Prabhudesai, M., A. Goyal, D. Pathak, and K. Fragkiadaki (2023). Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arX1v:2310.03739.
12

---

Qiu, S., L. Wang, C. Bai, Z. Yang, and Z. Wang (2022). Contrastive ucb: Provably efficient contrastive self-supervised learning in online reinforcement learning. International Conference on Machine Learning, pp. 18168-18210. PMLR.
Rigter, M., B. Lacerda, and N. Hawes (2022). Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems 35, 16082-16097.
Rombach, R., A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022, June). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 10684-10695.
Sample, P.J., B. Wang, D. W. Reid, V. Presnyak, I. J. McFadyen, D. R. Morris, and G. Seelig (2019). Human 5 utr design and variant effect prediction from massively parallel translation assay. Nature biotechnology 37(7), 803-809.
Sarkar, A., Z. Tang, C. Zhao, and P. Koo (2024). Designing dna with tunable regulatory activity using discrete diffusion. bioRxiv, 2024-05.
Schuhmann, C. (2022, Aug). Laion aesthetics.
Sohl-Dickstein, Weiss, N. Maheswaranathan, and S. Ganguli (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR.
Song. J., C. Meng, and S. Ermon (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
Srinivas, N., A. Krause, S. M. Kakade, and M. Seeger (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995.
Stark, H., B. Jing, Wang. G. Corso, Berger, R. Barzilay, and T. Jaakkola (2024). Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.05841.
Touvron, H., L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099: 0092880.
Trabucco, B., A. Kumar, X. Geng, and S. Levine (2021). Conservative objective models for effective offline model-based optimization. International Conference on Machine Learning, pp. 10358- 10368. PMLR.
Uehara, M. and W. Sun (2021). Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226.
Uehara, M., Y. Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, T. Bian- calani, and S. Levine (2024). Fine-tuning continuous-time diffusion models as entropy- regularized control. arXiv preprint 15X1v:2402.15194.
Valko, M., N. Korda, Munos, I. Flaounas, and N. Cristianini (2013). Finite-time analysis of kernelised contextual bandits. arXiv preprint 1309.663000680
Vargas, F., W. Grathwohl, and A. Doucet (2023), Denoising diffusion samplers. arXiv preprint arXiv:2302.1383.
Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint, Volume 48.
Cambridge university press.
Wallace, B., M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik (2023). Diffusson model alignment using direct preference optimization. arXiv preprint ar arXiv:2311.12900
Wu, K. E., Yang, R. van den Berg, S. Alamdari, J. Zou, A. X. Lu, and A. Amini (2024).
Protein structure generation via folding diffusion. Nature Communications 15(1), 1059.
13

---

Wu, X., K. Sun, F. Zhu, R. Zhao, and H. Li (2023). Better aligning text-to-iman lo-image models with human preference, arXiv preprint 2343:22303.14400.
Wu, Y., G. Tucker, and O. Nachum (2019). Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911:11911.111111
Xie, T., C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal (2021). Bellmann-conistent pessimism for offline reinforcement learning, Advances in neural information processing systems 34.
Xiong, W. H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang (2023). Iterative preference learning from human feedback: Bridging theory and practice for rihí under kl-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models.
Xu, J., X. Liu, Y. Wu, Y. Tong, Q. Li. M. Ding, J. Tang, and Y. Dong (2023). Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977.
Yang, K., J. Tao, J. Lyu, C. Ge, J. Chen, Q. Li, W. Shen, X. Zhu, and (2023). Using human feed- back to fine-tune diffusion models without any reward model. arXiv preprint 231.122311.13231.
Yu, T., G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma (2020). Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems 33, 14122..
Yuan, H., K. Huang, C. Ni, M. Chen, and M. Wang (2023). Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. arXiv preprint urXiv:2307.07055
Zhan, W., M. Uehara, N. Kallus, J. D. Lee, and W. Sun (2023). Provable offline preference-hased reinforcement learning. The Twelfth International Conference on Learning Representations.
Zhang, Q and Chen (2021). Path integral sampler: a stochastic control approach for sampling. arXiv preprint arX14:2111.15141.
Zhang, T,, T. Ren, M. Yang. J. Gonzalez, D. Schuurmans, and B. Dai (2022). Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning, pp. 26447-26466. PMLR.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In abstract and intro, we explained our algorithm in details.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We summarize it in Section 8)
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide theoretical results in Section 5 and Due to the page limit, we add more detailed results in the Appendix well.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have added details as much as possible.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: We have added codes.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have added the details in the main text and sup material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have added details as much as possible. In Figure 2, we show the distribu- tion. In Table 1, show the confidence interval. In Figure 3(b), also plot the std of for generated samples.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have added details much possible.
15

---

9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpah/t..pppp./a.a..../o/////tt
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed potential positive societal impact in the introduction. Regarding negative impact, we have regonized unintended is possible.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Due to the page limit, we haven't included However, we recognize diffusion models could be misused to make fake images. We aim to address this problem in the future.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
16

---

A Additional Related Works
In this section, we summarize additional related works.
Conservative approaches in offline RL/offline contextual bandits. Conservative approaches have been explored in offline RL and contextual bandits. Firstly, in both model-free and model-based RL, one prevalent method involves incorporating an additional penalty on top of the reward functions (Yu et al. 2020: Chang et al. 2021). Secondly, in model-based RL, common strategy is to train transition dynamics in a conservative manner (Kidambi et al.) 2020 Rigter et al.) 2022 Uchara and Sun 202I). Thirdly, in model-free RL, typical approaches include conservative learning of
q-functions (Kumar et al. 2020 Xie et al. (2021) or the inclusion of KI. penalties against behavioral
policies (Wu et al. 2019 Fakoor al. (0221)
However, these works are not designed to incorporate a diffusion model, unlike our approach. Hence, it remains unclear how their works can generate designs that remain within intricate valid design space, such as high-quality images using stable diffusion.
Design with generative models. Many works are focusing on design problems with generative models. However, these works are typically limited to the usage of VAEs. (Notin et al. 2021 Gómez Bombarelli et al. 2018) Our work is still significantly different because we focus on generative models.
Fine-tuning in LLMs from human feedbacks. A closely related area of research involves fine- tuning LLMs through the optimization of reward functions using human feedback (Touvron et al.. 2023 Ouyang et al. 2022). Especially, following works such as Zhan et al. (2023), from theoretical viewpoint, Xiong et al. (2023) explores the effectiveness of pessimism in offline scenarios and its theoretical aspect. However, our theoretical findings are more specifically tailored to diffusion models. Indeed, our main result in Theorem [ is novel, and our algorithm differs significantly, as fine-tuning methods in offline settings in the literature on LLMs typically rely policy gradient or PPO, whereas we use more direct backpropagation approaches. Furthermore, the meaning of step size is diffcrent as well. Hence, in (Xiong et al.) 2023), they do not use soft entropy regularized MDPs.
Typically, in the above works, human feedback is considered to given in the form of preferences. Similarly, in the context of diffusion models, Yang et al.] (2023); Wallace et al.] (2023) discusses fine-tuning of diffusion models using preference-bases feedback. However, these works focus on online settings but not offline settings.
Sampling from unnormalized distributions. our approach, we use an RL method to sample from our target distribution that is proportional to exp((f ((x) (z). While MCMC has traditionally been prevalent in sampling from unnormalized Boltzmann distributions exp(r(z)). recent discussions (Zhang and Chen [ Vargas et al., 2023). have explored RI. approaches similar to our approach. However, their focus differs from ours, as they do not address the fine-tuning of diffusion models (i.e., no Ppre) or sample efficiency in offline settings.
Another relevant literature discusses sampling from unnormalized Boltzmann distributions when pre-trained diffusion models are available (Kong et al., (202)). However, their algorithm closely resembles blassifier-based guidance (Dhariwal and Nichol (022). rather than fine-tuning algorithm. Additionally, they do not examine conservatism in an offline setting.
B Direct Back Propagation
Our planning algorithm has been summarized in Algorithm 2 Here, we parametrize each policy by neural networks.
For the sake of explanation, also add typical cases where the domain is Euclidean in Algorithm 3
17

---

Algorithm 2 Direct Back Propagation (General case)
1: Require: Set diffusion-model p(-( 0), pre-trained model (Ppro 11) batch size m, a parameter IR+.
2: Initialize: 0, = Pre
3: for [1, ... S do
4: 0et
5: Collect samples (0)) from current diffusion model (i.e., generating by sequen-
tially running polices {pe(- ( ())=0) from = T + 1 to t 1)
6: Update 0, to 0s+1 by adding the gradient of the following loss I((0) with respect to 0 at 0,:
((O))
m Σ
1
f( (o) -0)=)
(lpppc (10)
7: end for
8: Output: Policy (pe(1l;5s))
Algorithm 3 Direct Back Propagation (in Euclidean space)
1: Require: Set a diffusion-model (p; E 0)i-+++ pre-trained model
(W(ptt, zy; xy; of) batch size m, parameter a E IR+.
2: Initialize: 01 = 8 pre
3: for E [1,...... do
4: 0
5: Collect samples {a, (0)) from current diffusion model (i.e., generating by sequen-
tially running polices (N(ptt, zz; 0), of) H=) from + to = 1)
6: Update 8, to ss+1 by adding the gradient of the following loss L(0) with respect to 0 0,:
m
l 2 (0). t:
p(x) (0), 6 pre ||
((
F(a) ((=)) - a Σ
171
20²
7: end for
8: Output: Policy (s)((l:ss))
C All Proofs
C.1 Proof of Theorem
Here, we actually prove a stronger statement.
Theorem (Marginal and Posterior distributions). Let Pc(xe) and pt (x,(xx=1) be marginal distri-
butions at t or posterior distributions of x, given It-1, respectively, induced by optimal policies 11 Then,
pe(x.)
(x:)/ (0 p.
(ot (x-1).
Proof. To simplify the notation, let f() f(x) ((x). As first step. by using induction, aim to obtain an analytical form of the optimal policy
First, we define the soft-optimal value function as follows:
Uz- 0t-1 (x)- = Egg,) ((x) - a Σ
k=
Then, by induction, we have
pr (e(3)_110)) argmax (pp)) (12- (x1-1) (
(x)
18

---

With simple algebra, we obtain
(1-1(21-1) Pi (x+-1
pe( exp
(12)
a
Here, noting
max v((xr)
(()-)))- aKL(p (1))))
(E)((t)
we get the soft Bellman equation:
(e((xt)
Uz-1 (t-1((1-1)) Pi pre (20-112reddae--
exp
exp
(13)
a
Therefore, by plugging
13 into
12
we actually have
(1-1(11-1)
Pt pro (xt-1 (x))
exp
De (((
(14)
exp
Finally, with the above preparation, we calculate the marginal distribution:
/ II Pe( dx++1:+--
(x))
L==++1
Now, by using induction, we aim to prove
De(xx)) = exp
Dt pro
(xt))
a
Indeed, when / - 7 + 1, from (4). this hold as follows:
pr+ (r+i(eT41) exp (
pre (Pr+))))))
Now, suppose the above holds at t. Then, this also holds for t - 1:
/ pe( (x:))xz
P. (
/ exp
Ut (1-1((0-11)
(p. (p50o(xx (x=
(Use Equation 14
Ut- 01-1((1-11
exp
pp (II-1).
By invoking the above when the statement concluded.
C.2 Proof of Theorem
In the following, we condition on the event where
Væ E Xprei |r(x) f(x)) < g(z).
holds.
First, we define
(x( = E [(=)] cKL(Tl)p.n) J. (T) = E [r(x)] aKL( lpmm).
note that, 7. maximizes Ja (T). Therefore, we have
(T) - (fta) = (T) - Ja(TT + ja(T) - ja( + Ja((aa) Ja (fta
S JaTT) j.( + f.((ta) Ja(
(Definition of f.)
$ J.( - (T)
(Pessimism)
19

---

Here, in the step (i), we use
Vx E X prei (r(x) F(x)) - g(x).
Then.
Ja(T) - Ja(faa
S (T) - Ja(T) S 2B [xx
1 2 E ER [9(x)}}
(Jensen's 's inequality)
π
Pa [})))))
(Importance sampling)
Ppre
Hence, the statement is concluded.
D Theoretical Guarantees with Gaussian Processes
We explain the theoretical guarantee when using Gaussian processes. In this section, we suppose the model well-specified.
Assumption 2. r(æ) + e where e ~ ( 1) where r belongs to an RKHS in Hy.
D.1 Preparation
We introduce the notation to state our guarantee. For details, see Srinivas et al. (2009 Appendix B),
Uehara and Sun (202T) Chapter 6.2). Chang et al. (202I) Chapter C.3).
For simplicity, we first suppose the following. Assumption 3. The space X' is compact, and vz E (2 1.
We introduce the following definition. Regarding details, refer to Wainwright 22199 Chapter 12). Definition 1. Let Hy be the RKHS with the kernel k(:,-). We denote the associated norm and inner product by || [lx. {.))k. respectively. We introduce analogous notations for
k(x,,') k(x, n²) - k(x)) (K + +1)-11k(a)))
and denote the norm and inner product by || 11 (:))
Note explained in Srinivas et al. (2009] Appendix B) and Chang et al. 2222 Chapter C.3), actually, we have H4 = Hg-
In the following, We use the feature mapping associated with an RKHS H4. To define this, from Mercer's theorem, note we can ensure the existence of orthonormal eigenfunctions and eigenvalues [1/,, 4s} such that
{ (x) (x)dz K(:0) Σ (vss))
( (x 0(i + j)
Then, we define the feature mapping: Definition 2 (Feature mapping).
<(z) == I (T),)()), (PVV1 ...
Assuming eigenvalues are in non-increasing order, we can also define the effective dimension
following Srinivas al. (2009 Appendix B), Uehara and Sun 2021 Chapter 6.2), [hang et al. 2021 Chapter C.3): Definition 3 (Effective dimension).
6.2),(Chang
d' min jeN:j2n
The effective dimension is commonly used and caleulated in many kernels (aakko al. 2013). In
finite-ding dinee--ammnsional linear kernels {a + a ((x)) E R2d such that k(x, 2) Φ (x)0(2 (2),, letting
d' rank (o0 ((()))))o)))(((000)))) (())), we have
d' < ddd
because there exists M4+1 0, 1442 0....
20

---

D.2
Calibrated oracle
Using result in Srinivas et al. 2009). we show
f(z) - -r()) - c k(x
where
C()) 11 1 log (n/8)Inn In log(det(I K)).
Then, with probability - 6, we have
f(x) ra)) (f(-) r(-), k(,, x))
(Reproducing property)
< [1((-) r(-)lı x +1
(CS inequality)
= [[((-) -70-1 V (0,
- (0) / ()),
(Use Theorem 6 in Srinivas et al. 2009
D.3
Regret Guarantee (Proof of Corollary
1
Recall from the proof Theorem
Ja (a Ja S 2F (px=~10(x)) =c0 IV x)),
Now, first, to seper-bound Exwa /v k(2z, x)], we borrow Theorem 25 in Chang et al. 2021 which shows
Crd' (d' + log(e2/8))
Exwa Iv [((2,x))] s ai
n
where
Be>=<((2)) (x)]
C sup
<:ll x Expo., (x)) (r)]
Next, in order upper-bound C(8), we borrow Theorem in Chang al. (021). which shows
In S cd + log(c2/5)) log(1 + n).
The statement Corollary D is immediately concluded.
E
Additional Details of Experiments
E.1
DNA/RNA sequences
In this subsection, add the details of experiments in Section 7
E.1.1 Architecture of Neural Networks
Diffusion models. Regarding diffusion models for sequences, we adopt the architecture and
algorithm tailored to biological sequences over the simple space (Avdeyev et 2023) Its architecture is described in [able
We the architecture in Avsec et al. (2021), which is state-of-the-ar aar model in sequence modeling. We just change the last layer so that it is tailored to a regression problem as in Lal et al. (2024).
Oracles.
E.1.2 Hyperparameters
In all experiments, we use A100 GPUs. The important hyperparameters are summarized in the following table (Table on page 22
21

---

Table 1: Basic architecture of networks for diffusion models
Layer
Input dimension
Output dimension
Explanation
200 x 4
256
Linear + ReLU
256
256
ConviD ReLU
10
256
256
ConviD ReLU
11
256
256
ReLU
Table Important hyperparameters for fine-tuning. For all methods, we use Adam as an optimizer.
Method
Type
Value
Batch size
128
KL parameter
0.001
LCB parameter (bonus) c
0.1 (UTRs), 0.1 (Enhancers)
Number of bootstrap heads
3
BRAID
Sampling to neural SDE
Euler Maruyama
Step size (fine-tuning)
50
Guidance level
10
Guidance
Guidance target
Top 5%
Hyperparameter selection. The process of selecting hyperparameters in offline RL is known to be a challenging task (Rigter et al., 2022; Paine et al. 2020 in general. A common practice in existing literature is determining crucial hyperparameters with a limited number of online interactions. In our case, the key hyperparameters include the strength of the LCB parameter (utilized online in BRAID-Bonus and the termination criteria during training (applied to all fine-tuning algorithms such as STRL). To ensure a fair comparison, we operate within a framework where we can utilize 120 x 20 online samples. This implies that, for instance, in STRL and TRAID-Bonus, an online evaluation using 120 samples for 20 pre-delined epochs. However, in FRAID-Bonus, given the additional hyperparameters be tested (strengths of the bonus term 0.01, 0.1, 1.0), we 40 samples for each 20 pre-defined epoch.
conduct
E.1.3 Ablation Studies
We performed ablation studies by varying the strength of the bonus parameter in Figure We chose the one with the best performance in the main text (See the previous section to see the validity of this procedure).
1.0
0.5
-A5
-1.0
mftime
Treineeeed
Guidence
STRL
BRAID-Beot
BRAID-lonus BRAID-Bonud DRAD--louud
Figure 4: UTRs
E.2 Images
In this section, we describe the additional experiment regarding image generation in Section 7.2
22

---

Mathod
Figure 5: Enhancers
E.2.1 Description of Offline Data
utilize images from the AVA dataset (Murray et al. 2012) as samples I, containing over 250, 000 image aesthetic evaluations. Rather than using the raw scores directly from the dataset, we derive the labels by utilizing the pre-trained LAION Aesthetic Predictor V2 Schuhmann (2022) built on top of CLIP embeddings. This choice is made because employ the LAION Aesthetic Predictor the ground truth scorer to assess both our methods and baselines. In total, we have curated an offline dataset comprising 255490 image-score pairs: (a((t) (0)
E.2.2 Architecture of Neural Networks
We adopt the standard StableDiffusion v1.5 (Rombach et al. 2022) as the pre-traine trainrdieed model with the DDIM scheduler (Song al. 2020). Note this pre-trained model is conditional diffusion model.
Using the offline dataset {(x (اا we train the reward oracle f by an MLP on the top of CLIP embeddings. The detailed MLP structure listed in Table Note that, compared to the true LAION Aesthetic Score Predictor V2 (Schuhmann 2022), our reward oracle proxy has simpler structure with fewer hidden dimensions and fewer layers. We aim impose the hardness of fitting the true reward model, which is typically infeasible in many applications. In such scenarios, a pessimistic reward oracle is especially beneficial to mitigate overoptimization.
Table 3: Architecture of reward oracle for aesthetic scores
Layer
Input dimension
Output dimension
Explanation
768
256
Linear + ReLU
256
64
Linear ReLU
64
16
Linear + ReLU
16
Linear
E.2.3 LLM-aided evaluation
As stated in the main text, the original LAION Aesthetic Predictor V2 (Schuhmann) 2022) tends to assign higher scores even to images that disregard the original prompts, which is undesirable. To effectively identify such problematic scenarios, we employ a pre-trained multi-modal language model to verify whether the original prompt is present in the image or not. For each generated image, we provide the following prompt to LLaVA (Liu et al. 2024) along with the image:
<image>
USER: Does this image include (prompt } Answer with Yes or No ASSISTANT:
evaluated its accuracy and precision with human evaluators by generating images using Stable Diffusion with animal prompts (such as dog cat). The achieved Fl score was 1.0.
23

---

E.2.4 Hyperparameters
In all image experiments, we use four A100 GPUs for fine-tuning StableDiffusion v1.5
Rombach
al. 2022). The set of training hyperparameters listed in Table 4
Table 4: Important hyperparameters for fine-tuning Aesthetic Scores.
Method
Parameters
Values
Guidance weight
7.5
DDIM Steps
50
Batch size
128
BRAID
KL parameter
LCB bonus parameter C'
0.001
Number of bootstrap heads
4
Guidance weight
7.5
DDIM Steps
50
STRL
Batch size
128
KL parameter
Guidance level
100
Offline guidance
Guidance target
10
Optimizer Learning rate (41, (2)
AdamW
0.001
(0.9. 0.999)
Optimization
Weight decay
0.1
Clip grad norm
5
Truncated back-propagation step K
K Uniform 50)
E.2.5 Effectiveness of LLa LLaVA-aided evaluation
In our evaluation, we utilize a large multi-modal model like LLa As previously mentioned, relying solely on the raw score fails to detect scenarios where generated images ignore the given prompts.
Table(]illustree the outcomes of LLaVA--ssister evaluations for the pre-trained model and four checkpoints of the STRL baseline. It is evident that LLaVA successfully identifies all samples generated by the pre-trained model and the first two checkpoints. However, despite seemingly high-reward samples, many samples from checkpoints 3 and do not align correctly with their prompts, resulting in reduced mean reward. Figure @Jshowaaess five failure examples from each of checkpoints and Thus, we can validate our quantitative evaluation of reward overoptimization.
Table 5: Statistics of LLLa VA-Vu-adeeeeedd scores.
method
mean
min
max
invalid/total samples
pre-trained model
5.789
4.666
6.990
0/400
STRL-ckpt-I
6.228
4.769
7.193
0/400
STRL-ckpt-2
6.870
5.892
7.602
0/400
STRL-ckpt-3
6.484
0.0
7.944
50/400
STRL-ckpt-4
0.200
0.0
7.620
389/400
E.2.6 Ablation Studies
Ablation on BRAID- -AADDBBonus hyperparameter
We provide the boxplots for different Bonus hyper- parameters Figure indicating our method's robustness to hyperparameter tuning.
Additional qualitative results More image visualizations for BRAID and baselines can be found
in Figure
24

---

22
rabbit 7.02
6.72
monkey 7.63
butbarlyy 7.48
7.80
monkey|
panda 7.94
dog
Figure 6: Image-prompt alignment failures detected by LLaVA.
Roward
Offline
Pretrained
Guidance
STRL
Borus--Se-4
Bonus-le-3
Bonus-Se-
Benus-le-2
Bomus-5e-2
Method
Figure 7: Ablation study for BRAID-Bonus... By adjusting the pessimism strength C1 while keeping À 0.1, show that RRA--BBouus outperforms all baselines for wide range of hyperparameter selection.
cat
dog
horse
monkey
rabbit
butterfly
peacock
panda
Offiine Guidance
6.14
6.44
6:38
STRL
6.81
7.06
7.20
6.63
6.59
6.43
6.67
6.68
BRAID-Boot
7,82
7.45
7.33
7.39
7.32
7.18
7.80
7.74
BRAID-Bonus
7.48
7.84
7.49
7.54
7.54
7.15
7.45
7,50
Figure 8: More images generated by BRAID and basclines. All algorithms choose the best checkpoint according to our LLaVA-aided evaluation. The visualization demonstrates the benefits of introducing pessimistic terms that can help to achieve high scores while mitigating reward overoptimization.
25


Reviews:
[{"review_id": "w2ThrUBKm5", "reviewer": "Reviewer_wUpc", "summary": "1) This paper analyzed the two mainstream angles of computational design.\n2) Proposed a hybrid one that offline fine-tunes generative models.\n3) Conduct experiments on two tasks to show the performance of their method.", "questions": "N/A", "limitations": "1) Lack of gradual guidance from analyzing the advantages and disadvantages of existing methods to proposing the hybrid method.\n2) Lack of multi-metric quantitative results analysis. (e.g. in the image generation task, the fidelity and diversity should also be reported by like LPIPS Score/CLIP score/FID/IS, etc.)", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Sufficient theoretical analysis and detailed preliminaries.\n2) The idea is straightforward.\n3) The method is comprehensive.", "weaknesses": "1) In the introduction, the advantages and disadvantages of the two mainstream issues are not fully analyzed.\n2) Insufficient metrics evaluation for image generation task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback! We have addressed your concerns by providing additional explanations on (1) the disadvantages and advantages of pure generative model/MBO approaches, and (2) experimental results on diversity metrics based on CLIP scores. \n\n**Weaknesses: In the introduction, the advantages and disadvantages of the two mainstream issues are not fully analyzed. Limitations: Lack of gradual guidance from analyzing the advantages and disadvantages of existing methods to proposing the hybrid method.**\n\nWe have tried explaining well; however, due to space constraints, it may come across as somewhat terse. Assuming we have more space in the final version, we will elaborate further in the introduction as follows.\n\n1. Pure generative approach: We primarily refer to conditional diffusion models. \n- Disadvantages: It may be unclear whether we can learn designs with properties superior to those in the offline data. We will explicitly connect the introduction with Section 2. This connection will clarify that we indeed compare our approach with conditional diffusion models as a pure generative approach in Section 7.\n- Advantages: We can generate valid designs that lie within the sample space (e.g., natural image space, natural DNA, RNA, molecular space), which is challenging to achieve with a pure MBO approach.\n\n2. Pure MBO approach\n- Disadvantages: We will add the following example in the introduction. It can be challenging to incorporate the natural constraints of the valid space. For example, when the design space is an image space and the reward function is compressibility, optimizing the just learned reward model may result in highly non-natural images with high compressibility. However, our typical goal is to obtain a natural image with high compressibility. To achieve this, it is essential to incorporate a (pre-trained) generative model that characterizes the natural image space. \n- Advantages: Our approach learns a reward model, granting us the extrpolation capabilities so that we can generate designs beyond the offline data distribution by leveraging the extrapolation capabilities of the reward model. \n\n**Weakness: Lack of multi-metric quantitative results analysis. (e.g. in the image generation task, the fidelity and diversity should also be reported by like LPIPS Score/CLIP score/FID/IS, etc.)**\n\nThank you for the suggestion. As the primary goal of the experiments is to demonstrate that our method can mitigate overoptimization, we focus on reporting the performance metrics needed to show this in Section 7.2. However, we recognize that general synthesizing metrics are frequently reported in papers on diffusion models (though they may not be directly helpful in measuring overoptimization). The following is our attempt at a similarity score.\n\n**Our attempt**: While achieving diversity is not our stated promise, we experimented with CLIP cosine similarities for completeness to measure the diversity of generated samples. However, we observed that CLIP similarities are influenced largely by semantic information. For example, two images generated with the same prompt (e.g., cat) are likely to have a high CLIP cosine similarity (>0.8), even if they are visually very different. We agree that LPIPS might be more reliable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZeCXOYk38w", "reviewer": "Reviewer_dLuW", "summary": "This paper proposes a conservative approach for fine-tuning diffusion models with a reward model learned from offline data. Specifically, the ideas are two-fold: The first idea is to replace the reward model with a conservative estimate based on classical generalization bounds. The second idea is to leverage the KL divergence to force the optimized distribution to not deviate too far from the pretrained model. Experiments and theoretical results show the efficacy of the proposed method in fine-tuning diffusion models without over-optimization.", "questions": "STRL methods like AlignProp and DRaFT can work with ODE samplers, which are more commonly used in practice than SDE samplers. However, the method proposed in this work, due to the use of entropy regularization, can only adopt SDE samplers. I wonder if it is possible to design a regularization term for ODE samplers. Could the authors share some insights on this point?", "limitations": "See my comments on the weakness above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is well-presented, and the motivation behind the algorithm is interesting. The over-optimization problem is indeed critical when fine-tuning diffusion models with learned rewards.\n\n2. Extensive experimental results show the efficacy of the proposed method in improving the reward model while avoiding reward over-optimization.", "weaknesses": "1. Leveraging generalization bounds via kernel RKHS and bootstrap is interesting, but I doubt their practicality for real applications. Firstly, the RKHS bound is usually too conservative to be useful, while the computational cost for the bootstrap method is pretty high since one has to train the model from scratch multiple times. As far as I can tell, the reward models used in the experiments are mainly single-layer MLPs, and it is doubtful whether this approach is useful when the reward model needs to be a larger model.\n\n2. Another problem with the conservative estimator of the reward models is that it is unclear whether it is useful given the current experimental results. On one hand, KL regularization is a widely-known technique for preventing over-optimization in diffusion models and is thoroughly studied in existing works, so it is certain that the KL regularization term will help. On the other hand, the proposed algorithm mixes both the conservative reward estimator and the KL regularization term together, making it unclear which part is playing the role in avoiding over-optimization. My guess is that, for the most part, only the KL regularization term is effective in the end.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback. We have addressed your concern by explaining (1) the bootstrap/RKHS bonus term's practical usefulness in our scenario and its widespread use in various real applications/papers, and (2) how our experiments are intentionally designed to demonstrate that conservatism (rather than KL) helps mitigate overoptimization. Please let us know if this addresses your concerns, or if there are any other issues we should discuss or address!\n\n**Weakness: RKHS bound is usually too conservative to be useful. I doubt their practicality for real applications.**\n\nWe would like to convey that it is still practically useful from several aspects. \n \n1. **Negative penalty term (Example 1) based on RKHS theory is widely used in the literature on Gaussian processes (GPs)**. For example, as noted in Wikipedia of GPs, many widely used software tools are built around GPs. They have been proven to be effective in real-world applications, e.g., robotics/material science/drug discovery (Deringer et al., Gramacy et al.). \n\n2. **Our examples with DNA and RNA designs are real applications**, as the tasks, data, and models are all proposed by established scientific journals, as we have cited Sample et al., 19, Inoue et.al, 19. There is more literature on solving this task, e.g., Taskiran et al.   \n\n3. We believe there may be a misunderstanding: **We have combined it with deep learning models.** After extracting features from deep learning models, we used them as a negative penalty term to avoid overoptimization in Example 1. We did not use a pure RKHS model with a fixed feature as our reward model. This approach is used by many famous RL papers, such as Yu et al. 19. \n\n4. While we acknowledge that the generalization bound in Theorem 2, derived under the RKHS assumption, might not be tight, **Our goal is to propose an algorithm that offers both superior experimental performances and fundamental guarantees that formalize the algorithm’s intuition.** We believe that our upper bound suffices for this purpose. Specifically, our intuitive message is that fine-tuned generative models outperform the best designs in offline data by leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs. Corollary 1 formalizes this intuition, as mentioned in Section 6.\n\n5. **Statistical guarantees in most of the literature are not completely tight**, as proving the lower bound is extremely challenging in many scenarios (Wainwright, 19). Therefore, the claim mentioned by the reviewer applies not only to our work but also to nearly all upper bounds in the literature in RKHS/GPs. We think showing tightness is beyond our scope. \n\n**Weakness: the computational cost for the bootstrap method is high since one has to train the model from scratch multiple times. The reward models used in the experiments are mainly single-layer MLPs, and it is doubtful whether this approach is useful when the reward model needs to be a larger model.**\n\nWe acknowledge the reviewer’s point; however, we would like to emphasize that (1) computational efficiency is not the primary focus of our paper, (2) it can be addressed straightforwardly by using variants if we want to further accelerate bootstrap part, and (3) the computational cost of bootstrap is not necessarily high in many real applications.\n\n1. **In scenarios with limited offline data, sample efficiency (the cost of obtaining data with reward feedback) is more critical than computational efficiency.** Our work always prioritizes the former, which differs from online settings where computational efficiency may be more relevant. We have motivated such emphasis in the Introduction by clarifying the hardness of wet lab data collection in scientific domains.\n\n2. **To increase speed, we can utilize various bootstrap variants (Lakshminarayanan et al. Al, Osband et al., Chua et al.) without training from scratch, such as sharing backbone models, Bayesian deep learning way, etc.** In this way, the bootstrap has been practically and widely used in the ML community. We will add such an explanation. \n\n3. The computational cost of running pure bootstrap in our experiments is also not so high in many real applications. Indeed, while the reward models in Section 7.1 are not single-layer MLPs (they include transformers and are representative models in genomics proposed in Avsec et.al 21), the training takes several hours. In scenarios where sample efficiency is crucial, training multiple times (or in a parallel manner) won’t be a significant concern. In fact, for both tasks we only employ 3,4 bootstrapped models, so it does not bring too much computational concerns.\n\n**It is unclear whether it is useful given the current experimental results. On one hand, KL regularization is a widely-known technique for preventing over-optimization in diffusion models so it is certain that the KL regularization term will help. On the other hand, the proposed algorithm mixes both the conservative reward estimator and the KL regularization term together, making it unclear which part is playing the role in avoiding over-optimization. My guess is that, for the most part, only the KL regularization term is effective in the end.**\n\nWe believe there may be a misunderstanding regarding the interpretation of our experimental results. **In our experiments, we have compared our proposal with the baseline that has the KL regularization, as noted in Section 7 ( Lines 177-185 in our paper), but do not have conservatism in the reward model.** For example, in images, STRL and our proposals BRAID-Boot, BRAID-Bonus) share exactly the same KL strength, and its value ($\\alpha=1$) can be found in Table 5, Appendix  F.2.4. Therefore, our current experiments have directly addressed your concern by showing the effectiveness of conservative reward modeling. \n\n**Q: Can be adapted to ODE samplers?**\n\nWe address this in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVNShYNhh3", "reviewer": "Reviewer_hthV", "summary": "The paper tackles the task of black box optimization in an offline setting. Given a pretrained diffusion model, they first train a surrogate model on the offline data and use it to tilt the diffusion model distribution via finetuning it. The authors distinctly focus on an uncertainty quantification based procedure to bias the diffusion model tilting toward regions where the reward is high and the reward uncertainty is low while not tilting toward regions with high uncertainty. Experiments are carried out on a reasonable set of diverse tasks.\n\nThe paper introduces a small specific challenge and address it well with a reasonable approach and good motivations. The potential impact of the method may be small but it is neat, educational, and should be useful in many cases. I recommend acceptance.", "questions": "1. Do I understand correctly that theorem 2 only states that the finetuned diffusion model will incur lower regret than the pretrained diffusion model? Why is that useful and would we not rather be interested in relations between using your additional uncertainty quantification bias for the reward model based finetuning and not using it?\n2. It seems to me that reward models are often very poor in scientific applications (more so than the generative models that can be trained on a lot more data). Does this mean that their uncertainty estimates are also bad and your method might not provide any improvements in these cases?", "limitations": "The authors point out the key inherent limitation of their work in the fact that the reward models that are often trained on limited data are likely suboptimal instead of just mentioning useless small limitations that are beside the point (which is the more common practice it seems).", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Identifying a crucial non-obvious overlooked challenge and specifically pinpointing it. The authors identify that previous work on tilting diffusion models with reward models misses to incorporate tilting less toward regions of the reward function in which it has high uncertainty but high reward. Instead, we should only tilt to the regions that have high reward and high certainty to avoid optimizing toward adversarial examples. (The fact that the finetuned diffusion model will steer away from the pretraining distribution seems like a less relevant insight)\n2. The authors identify a relevant overlooked problem in finetuning diffusion models and bring standard techniques from uncertainty quantification into the field to address it in a reasonable fashion. They do not overcomplicate things and their technique could be valuable to several researchers in the area. \n3. The authors prove that the training procedure yields the desired distribution. I have no comments regarding the value/insightfulness of the proof. Maybe other reviewers have a stronger opinion about the relevance.\n4. The authors evaluate their method on a very diverse set of experiments that includes discrete DNA sequence generation, and image generation. The results are convincing and demonstrate the central empirical claim that out of distribution generation is a problem and is effectively avoided with the proposed conservative reward model fine tuning.\nMinor:\n1. Interesting snippets of insights. The authors point out interesting relationships and connections along the way which are non-obvious and well placed for putting their motivations into context.\n2. Exceptional clarity in writing. The paper lays out the task in its precise specification and covers required concepts and related work in equal clarity.", "weaknesses": "1. I would say that the insights in terms of methodological novelty are on the moderate side. The ideas are simple and good, which is appreciated, but the level on which the conceptural changes operate are low level (a tweak to diffusion model tilting) and thus limited in impact. However, it is certainly a good thing to have.\nVery hard to address and not a must have for ML conferences:\n1. Evaluations are inherently limited in their computational nature and the conclusions that can be drawn for the procedures effectiveness in biological sequence optimization is small. Do you authors disagree with this in any way?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback. We have addressed your concern by explaining a more detailed evaluation plan for biological tasks.  \n\n**Weakness: Evaluations are inherently limited in their computational nature and the conclusions that can be drawn for the procedures effectiveness in biological sequence optimization is small. Do you authors disagree with this in any way?**\n\nWe thank the reviewer for pointing out that our computational evaluation did not provide enough confidence in our generated biological sequences. While these are complex systems, several of the main factors contributing to the activity of these biological sequences are well-studied and understood. Therefore, there are several analyses we can perform to gain deeper confidence.\n\n* Of the two functions we optimize, enhancer activity in HepG2 is controlled to a large extent by the binding of transcription factors to the DNA sequence, particularly HepG2 specific transcription factors such as HNF4A, HNF4G, HNF1A and HNF1B. 5’UTR activity is, to a large extent, dependent on the sequence of the translation initiation site. Features such as A or G at position −3 relative to AUG and a G at +4 are understood to increase sequence activity, as well as the absence of upstream start codons. These features are understood from many biology studies, including those on the original datasets (Sample et al. 2019, Gosai et al. 2023)\n\n* For the generated HepG2 sequences, we will use the JASPAR database (https://jaspar.elixir.no/) to scan the generated sequences for motifs known to bind to human transcription factors. We expect to see a higher frequency of motifs for activating transcription factors, particularly HepG2- specific ones, and a reduced frequency of motifs known to bind to transcriptional repressors. For the 5’UTR sequences, we will examine the sequences for upstream start codons and also examine base frequencies in the translation initiation sites. \n\nThis analysis will clarify whether our model has learned well-established features of biological sequence activity and allow us to draw stronger conclusions regarding the validity of the generated sequences.\n\n**Questions: Do I understand correctly that theorem 2 only states that the finetuned diffusion model will incur lower regret than the pretrained diffusion model? Why is that useful and would we not rather be interested in relations between using your additional uncertainty quantification bias for the reward model based finetuning and not using it?**\n\nLet me clarify as follows. \n\n* Roughly speaking, our theorem (Theorem 2 and Corollary 1) states that our fine-tuned model performs better in terms of $J_{\\alpha}$ than the best design in the reward data distribution. Since $J_{\\alpha}$  includes a metric related to proximity to pre-trained models, this theorem conveys our key message: 'fine-tuned generative models outperform the best designs in the offline data by leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs’. \n\n* The uncertainty quantification term is used to construct $\\hat p_{\\alpha}$. When evaluating the actual performance in Theorem 2, we have used the true reward $r$. \n\n**Q: It seems to me that reward models are often very poor in scientific applications (more so than the generative models that can be trained on a lot more data). Does this mean that their uncertainty estimates are also bad and your method might not provide any improvements in these cases?**\n\nYes. While we did not intend to obscure this information, as it is briefly mentioned in the Limitations section, we will make it more explicit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dy0r2GPpf1", "reviewer": "Reviewer_HnkA", "summary": "This paper presents a conservative fine-tuning method called BRAID, which integrates the strengths of diffusion models and model-based optimization (MBO) to improve the performance of pre-trained diffusion models on offline datasets. BRAID optimizes a conservative reward model that includes penalties outside the offline data distribution to prevent overoptimization and generate valid designs. The approach is validated through empirical and theoretical analyses, demonstrating its ability to outperform the best designs in offline data while avoiding the generation of invalid designs. The paper also discusses the method's effectiveness compared to existing conditional diffusion models and traditional MBO techniques, with experiments showcasing its superiority in biological sequence and image generation. The authors acknowledge the limitations of their study, particularly in model selection and hyperparameter tuning, and suggest future research directions.", "questions": "* The methods will need more samples, as shown in the pseudo-code of the algorithms 2 Direct Back Propagation (General case), which may bring more computational burden. Meanwhile, the methods train use two diffusion model to obtain the policy. Can you give me some experiments to show the computational burden? \n* Have the authors considered alternative generative models such as GANs or VAEs, and can they provide a comparative analysis of performance and resource usage?\n* How to obtain the $\\hat{g}$ in the pseudo-code of Algorithm 1 BRAID? \n* This methods is related to offline reinforcement learning. So can you give me some experiment comparisions with the SOTA offline RL methods ？", "limitations": "* The method requires careful selection of hyperparameters, which can be challenging in a purely offline setting without access to additional online data.\n* The pseudo-code of the paper does not illustates the algorithm explicitly. The authors can improve it further.\n* This paper lack strong baselines comparisions and enough related works, which is related to some offline reinforcement learning methods. So you can add more related offline reinforcement learning works.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* BRAID incorporates a conservative approach to fine-tuning diffusion models, which includes penalization terms that discourage the model from generating designs outside the distribution of the offline data. This conservative strategy is effective in preventing overoptimization and ensuring the validity of the generated designs.\n\n* The method is supported by both theoretical analysis and empirical results. Theoretically, it provides a regret guarantee, ensuring that the fine-tuned models can outperform the best designs in the offline data. Empirically, it has been validated through experiments across various domains, such as biological sequences and images, demonstrating its ability to generate high-quality designs.", "weaknesses": "* Difficulty in tuning hyperparameters without online data interaction.\n* Reliance on accurate reward and diffusion models for effective performance.\n* Theoretical results depend on certain idealized assumptions that may not hold in all cases.\n* Can you compare the methods with other SOTA offline RL methods to illustrate your proposed augmented methods more effective than the SOTA offline RL methods? I think this paper is very relevant to some offline RL methods, such as ReDS[1], A2PR[2], CPED[3], SCQ[4]. It is not required that experimental comparisons must be given, but at least add some discussion with these methods to the paper.\n\nReferences：\n\n[1] Singh, Anikait, et al. \"ReDS: offline reinforcement learning with heteroskedastic datasets via support constraints.\" Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023.\n\n[2] Liu, Tenglong, et al. \"Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning.\" In International Conference on Machine Learning (ICML). PMLR, 2024.\n\n[3] Zhang, Jing, et al. \"Constrained policy optimization with explicit behavior density for offline reinforcement learning.\" Advances in Neural Information Processing Systems. 2023\n\n[4] Shimizu, Yutaka, et al. \"Strategically Conservative Q-Learning.\" arXiv preprint arXiv:2406.04534 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and insightful feedback. We have addressed the reviewer's concern by clarifying that (1) our goals differ significantly from those in standard offline RL works,  (2) we have compared our method with recent works that align with our objectives, such as Yuan et al. (2023) and Krishnamoorthy et al. (2023).\n\n**W: Comparison with SOTA offline RL methods, IReDS[1], A2PR[2], CPED[3], SCQ[4].**\n\nThanks for raising this point. We would like to emphasize that, in general, our originally intended goals are quite different, though we acknowledge that the ideas in the papers you cited are very interesting and potentially helpful in developing new algorithms for our tasks. \n\n1. **The goals of our paper and most offline RL works are different (as briefly mentioned in Appendix A) because our paper aims not to address standard offline RL tasks but our paper aims to generate high-quality designs in extremely high-dimensional action spaces.** Given the high dimensionality of these spaces (e.g., images, chemicals, biological sequences), our focus is on incorporating SOTA pre-trained generative models (i.e., diffusion models) to constrain the search space to valid domains (natural image space, natural chemical space, natural biological space). While we appreciate the relevance of the works you cited in offline RL, these papers were not designed to address our specific tasks. It is unclear how their methods, which do not include experimental results in our scenarios (such as generating images, biological sequences, or molecules), can be directly applied to our work. \n\n2. We have compared relevant existing methods in the most pertinent field: Goal-wise, our work aligns more closely with offline (contextual) bandits in extremely high-dimensional action spaces rather than with offline RL. However, the RL aspect appears in our work because diffusion models can be viewed as MDPs, and we have utilized this observation to address our problem as detailed in Section 5. **We have compared several recent baselines, such as Krishnamoorthy et al. (2023) and Yuan et al. (2023), that tackle the same problem as us in Section 7.**\n\n3. Let me explain whether offline RL works you cited naively address our problem. **While the offline contextual bandit problem is an offline RL problem with horizon 1, it is not obvious that the papers you cited are directly applicable to our task because they do not focus on incorporating expressive pre-trained generative models.** We acknowledge that some insights from these papers could be relevant to our problem. However, translating these insights into our context may not be straightforward and could require more than simply adapting the original algorithms.\n\n- For instance, replacing $\\rho$ with diffusion models in Algorithm 1 of ReDS[1] might be valid, but its practical utility is uncertain since the log-likelihood estimation of diffusion models lacks an explicit form and is not differentiable.\n- Similarly, using diffusion models as policies in Algorithm 1 of A2PR[3] might not be practical for our task as well because it is hard to get the explicit form for the log-likelihood. \n\n**W. Difficulty in tuning hyperparameters without online data interaction/Reliance on accurate reward and diffusion models for effective performance**\n\nWe agree with the reviewer and have acknowledged our limitations. However, since this limitation is common across many works that use purely offline data (e.g., Rigter et al. (2022); Kidambi et al. (2020)), we have adhered to current conventions in the literature by assuming limited online interaction in actual experimental settings.\n\n**Q: The methods will need more samples, as shown in the pseudo-code of the algorithm 2. Can you give me some experiments to show the computational burden?**\n\nTo ensure clarity, let us first distinguish between two types of samples: (a) offline data samples ($x, r(x))$) and (b) artificial samples generated by diffusion models. In the offline setting, sample efficiency in terms of offline data (a) (e.g., lab feedback data) is crucial, so the computational burden (how many samples we use in (b)) is less of a concern as long as it is not excessively high.\n\n**With this in mind, yes, our Algorithm 2 requires artificial samples like (b) above. However, since we are not using offline data, the number of samples used in Algorithm 2 is less concern. We have not discussed the computational complexity as it has already been discussed in  existing works in detail (Black et al. (2023) ,Prabhudesai et al., 2023), and our focus is on sample efficiency.** The actual computational cost varies by domain; for example, image generation might take several hours, while sequence generation might take several minutes. We will address this info in the next version. \n\n**Q: Have the authors considered alternative generative models such as GANs or VAEs, and can they provide a comparative analysis of performance and resource usage?**\n\nThis is an interesting point. We will include a more detailed discussion in the Appendix.\n\n- **Reasons for using diffusion models**: In many domains (e.g., images, molecules), diffusion models have demonstrated SOTA performance as generative models, as evidenced by numerous papers (Rombach, Robin, et al; Avdeyev, Pavel, et al). This motivates us to use diffusion models as SOTA generative models to capture the valid space (natural image, chemical, biological spaces). \n\n- **Advantages and disadvantages of GANs and (pure) VAEs:** We acknowledge that GANs and VAEs remain useful as generative models. In particular, fine-tuning to optimize downstream reward functions is computationally faster if we use them. However, as demonstrated in many papers on generative models for images, molecules, the performance as generative models tends to decrease.\n\n**Q: How to obtain the $g$ in the pseudo-code of Algorithm 1 BRAID?**\n\nWe have discussed it in Section 4.3. We will add a pointer to make it more accessible to readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate feedback from all reviewers. We respond to raised weaknesses/questions as much as possible. 

**Papers we cite:** We have added the papers we cited in our response here.

* Deringer, V. L., Bartók, A. P., Bernstein, N., Wilkins, D. M., Ceriotti, M., & Csányi, G. (2021). Gaussian process regression for materials and molecules. Chemical Reviews, 121(16), 10073-10141.

* Gramacy, Robert B. Surrogates: Gaussian process modeling, design, and optimization for the applied sciences. Chapman and Hall/CRC, 2020.

* Taskiran, Ibrahim I., et al. "Cell-type-directed design of synthetic enhancers." Nature 626.7997 (2024): 212-220.

* Yu, Tianhe, et al. "Mopo: Model-based offline policy optimization." Advances in Neural Information Processing Systems 33 (2020): 14129-14142.

* Wainwright, Martin J. High-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge university press, 2019.

* Chen, Ricky TQ, et al. "Neural ordinary differential equations." Advances in neural information processing systems 31 (2018).

* Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

* Chua, Kurtland, et al. "Deep reinforcement learning in a handful of trials using probabilistic dynamics models." Advances in neural information processing systems 31 (2018).

* Osband, Ian, et al. "Deep exploration via randomized value functions." Journal of Machine Learning Research 20.124 (2019): 1-62.

* Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

* Avdeyev, Pavel, et al. "Dirichlet diffusion score model for biological sequence generation." International Conference on Machine Learning. PMLR, 2023.

* Xu, Minkai, et al. "Geodiff: A geometric diffusion model for molecular conformation generation." International Conference on Learning Representations. 2022

**Additional rebuttal for a reviewer dLuW** 

That’s an interesting point. Here are several ideas. We will incorporate the following into the main text:

- While somewhat heuristic, we can still use the ODE sampler after fine-tuning. Specifically, when generating samples for fine-tuning, we use the SDE sampler with a small variance term; however, after fine-tuning, we can still use it as the ODE sampler by setting the variance term to 0. 

- Another idea is to explicitly calculate (or estimate) the log-likelihood ($log p_{\text{pre}}$) using the well-known formula from the neural ODE paper (Theorem 1 in Chen et.al) and incorporate it as a penalty in the reward term. However, the caveat of this approach is that estimating this log-likelihood is often challenging and non-differentiable.

Meta-review: The paper proposes a conservative fine-tuning approach to tackle overoptimization in diffusion models, which is common in settings where task-specific reward models are unknown and offline data is used instead for fine-tuning diffusion models. The reviewers highlighted the originality of the proposed method and found that the empirical evaluation satisfactorily demonstrated the method's usefulness. All reviewers agree that the paper should be presented at NeurIPS, and I agree that the paper would be of interest to the NeurIPS community and would be a valuable addition to the program.
Decision: Accept (poster)