Provable Maximum Entropy Manifold Exploration via Diffusion Models
Authors: Riccardo De Santi
Marin Vlastelica
Ya-Ping Hsieh
Zebang Shen
Niao He
Andreas Krause
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
exploration
fine-tuning
maximum state entropy reinforcement learning
Primary area: general_machine_learning
TL;DR: A scalable algorithmic principle for provable manifold exploration via diffusion model fine-tuning.

Abstract: Exploration is critical for solving real-world decision-making problems such as scientific discovery, where the objective is to generate truly novel designs rather than mimic existing data distributions.  In this work, we address the challenge of leveraging the representational power of generative models for exploration without relying on explicit uncertainty quantification. We introduce a novel framework that casts exploration as entropy maximization over the approximate data manifold implicitly defined by a pre-trained diffusion model. Then, we present a novel principle for exploration based on density estimation, a problem well-known to be challenging in practice. To overcome this issue and render this method truly scalable, we leverage a fundamental connection between the entropy of the density induced by a diffusion model and its score function. Building on this, we develop an algorithm based on mirror descent that solves the exploration problem as sequential fine-tuning of a pre-trained diffusion model. We prove its convergence to the optimal exploratory diffusion model under realistic assumptions by leveraging recent understanding of mirror flows. Finally, we empirically evaluate our approach on both synthetic and high-dimensional text-to-image diffusion, demonstrating promising results.

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Riccardo De Santi" 12
Marin Vlastelica 12
Ya-Ping Hsieh
Zebang Shen
Niao He
Andreas Krause 12
Abstract
R
M RD
Exploration is critical for solving real-world decision-making problems such as scientific discovery, where the objective is to generate truly novel designs rather than mimic existing data distributions. In this work, we address the challenge of leveraging the representational power of generative models for exploration without relying explicit uncertainty quantifi- cation. We introduce novel framework that casts exploration as entropy maximization over the approximate data manifold implicitly defined by pre-trained diffusion model. Then, we present novel principle for exploration based on density estimation, problem well-known be challenging in practice. To overcome this issue and render this method truly scalable, we leverage fundamental connection between the entropy of the density induced by a diffusion model and its score function. Building on this, we develop an algorithm based on mirror descent that solves the exploration problem as sequential fine-tuning of pre-trained diffusion model. We prove its convergence to the optimal exploratory diffusion model under realistic assumptions by leveraging recent understanding mirror flows. Finally, we empirically evaluate our approach on both synthetic and high-dimensional text-to-image diffusion, demonstrating promising results.
Ppre
Figure 1: A diffusion model The pre-trained on set of points (white) implicitly learns a set Spre (orange) approx- imating the true low-dimensional data manifold M C R (black) with m < d. The approximate data manifold Qpre can be significantly smaller than Rd (yellow).
et al., 2023). Traditionally, generative models have been employed to capture the underlying data distribution in high- dimensional spaces, facilitating processes such as molecule generation or material synthesis (Bilodeau et al., 2022; Zeni et al., 2023). However, simply approximating the data dis- tribution is insufficient for real-world discovery, where ex- ploration beyond high (data) density regions is essential.
Nonetheless, illustrated in Figure 1, these models excel at capturing complex data manifolds that are often signifi- cantly wwerrdiiennsooal than the ambient space (Stanczuk et al., 2024; Kamkari et al., 2024; Chen et al., 2023), and can synthesize realistic novel samples that satisfy intricate constraints (e-g.. valid drug molecules or materials). Yet, when the goal shifts to exploring novel regions within that manifold, fundamental question remains:
1. Introduction
How can we leverage the representational power of generative models to guide exploration?
Recent progress in generative modeling, particularly the emergence of diffusion models (Sohl-Dickstein al., 2015; Song Ermon, 2019; Ho al., 2020), has achieved un- precedented success in generating high-quality samples across diverse domains, including chemistry (Hoogeboom et al., 2022), biology (Corso et al., 2022), and robotics (Chi
Our approach In this work, we tackle this challenge by first introducing the maximum entropy manifold exploration problem (Section 3). This involves learning a continuous- time reinforcement learning policy (Doya, 2000; Zhao et al., 2024) that governs new diffusion model optimally plore the approximate data manifold implicitly captured by pre-trained model. To this end, we present a theoretically grounded algorithmic principle that enables self- seff-guieed ex- ploration via a diffusion model's own representational power of the density it induces (Section 4). This turns exploration
'Equal contribution 'ETH Zurich, 8092 Zurich, Switzerland "ETH Al Center, Zurich, Switzerland. Correspondence to: Ric- cardo De Santi <rdesanti eth..ch.>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
into density estimation, task well-known to be challenging in high-dimensional real-world settings (Song et al., 2020; Kingma et al., 2021; Skreta et al., 2024). To overcome this obstacle and render the method proposed truly scalable, we leverage a fundamental connection between the entropy of the density induced by a diffission model and its score func- tion. Building on this, we propose a practical algorithm that performs manifold exploration through sequential fine- tuning of the pre-trained model (Section 6). We provide theoretical convergence guarantees for optimal exploration in simplified illustrative setting by interpreting the algo- rithm proposed as a mirror descent scheme (Nemirovskij & Yudin, 1983; Lu et al., 2018) (Section 5), and then generalize the analysis to realistic settings building on recent under- standing of mirror flows (Hsich et al., 2019) (Section 7). Finally, we provide an experimental evaluation of the pro- posed method, demonstrating its practical relevance on both synthetic and high-dimensional image data, where we lever- age pre-trained text-to-image diffusion model (Section 8).
(DMs) are deep generative models that approximately sam- ple complex data distribution by learning from observa- tions dynamical system to map noise to novel valid data points (Song & Ermon, 2019). First, we introduce a forward stochastic differential equation (SDE) transforming to noise data points sampled from the data distribution Pdata :
dX, = [(x,, (ddt + 9()dB, with Xo ~ Pdata (1)
where X, E Rd represents d-dimensional point, (Br, 2 0) is d-dimensional Brownian motion, f R. x R + Rd is a drift coefficient, and g R - R, is a diffusion coefficient. We denote with Pr the marginal density at time t. Given a time horizon > 0, one can sample X, ~ Pt by running the forward SDE in Equation (1). We denote the time-reversal process by Xm = Xx- for 0 S t t T, following the backward SDE:
dxpc =f (X. T t)dt + ng(T t)dB, (2)
with free(Xpa)) T t) corresponding to:
Our contributions To sum up, in this work we present the following contributions:
1+ 2 g tt (
-f -[(xpe -T)
The maximum entropy manifold exploration problem, that captures the goal of exploration over the approximate data manifold implicitly represented by pre-trained diffusion model (Section 3)
where v log (xr) is the score function, and 17 E [0, 1]. By following the backward SDE in Equation (2) from XT Pr, after steps one obtains X,- = XTT ~ po = Pdata- In practice, pr is replaced by a fixed and data-independent noise distribution Pso R Pr for large T, typically Gaus- sian', motivated by an asymptotic analysis of certain diffu- sion dynamics (Tang & Zhao, 2024).
A scalable algorithmic principle for manifold exploration that leverages the representational power of pre-trained diffusion model (Section 4), and a theoretically grounded algorithm based on sequential fine-tuning (Section 6).
Score matching and generation. Since the score function Vz Per)) is unknown, it is typically approximated by a neural network so (zx t) Icarned by minimizing the MSE at points sampled according to the forward process, namely:
Convergence guarantees for the algorithm presented both under simplified and realistic assumptions leveraging re- cent understanding of mirror flows (Sections 5 and 7).
An experimental evaluation of the proposed method show- casing its practical relevance on both an illustrative task and a high-dimensional text-to-image setting (Section 8).
E J(0) = [- xopp,
[w(t)]ls(xctt) t) - V log
where w : [0, + Rso is a weighting function. Crucially, this is equivalent to the denoising score matching objective (Vincent, 2011) consisting in estimating a minimizer 0* of:
2. Background and Notation
General Notation. We denote with X C Rd an arbitrary set. Then, we indicate the set of Borel probability measures on X with P(X),, and the set of functionals over the set of probabil- ity measures PP(2)) as (A)).. We write dy = pdx to express that the density function of μ E P(Y) with respect to the Lebesgue measure is p. Along this work, all integrals with- out an explicit measure are interpreted w.r.t. the Lebesgue measure. Given integer N. we define [N] (1. (1...., N). Moreover, for two densities μ,, / E PP))., we denote with DKL. (H,, v) the forward tween and p. Ultimately, we denote by 7d[0, a] the uniform density over the bounded set [0, a] with a E R.
w(t) E E [- J0--po 1r0
E
Va, log po (x:(2))))))) [1)
where Pel- | ro) is the conditional distribution of xa given an initial sample xo ~ Po. which has a closed-form for typical diffusion dynamics. Once an approximate score function 8g- is learned, it can generate novel points approximately sampled from the data distribution. This is achieved by sampling an initial noise sample X5- Pso and following the backward SDE in Equation (2), replacing
Beilback-Leiblee divergence be-
IIn the following, we will choose Poo to be a truncated Gaussian for the sake of theoretical analysis.
Continuous-time diffusion models. Diffusion models
2

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Ppr
FT
Pre
noise distribution
PT
PT
π
Pu
PT
(a) Pre-trained and fine-tuned diffusion model processes.
(b) Regularized surprise maximization
Figure (2a) Diffusion processes and marginal densities corresponding to the pre-trained model œ (green), and maximally explorative fine-tuned model x² (violet). (2b) Fine-tuning pre-trained diffusion model (green) via the surprise maximization principle in Eq. (9) one obtains diffusion model (violet) able to sample low-density regions.
the true score V (c()) with sæ-, leading to the process {X; he Next, we introduce a framework that we will leverage to fine-tune a pre-trained diffusion model.
3. Problem Setting: Maximum Entropy Manifold Exploration
In this work, we aim to fine-tune a pre-trained diffusion model pre to obtain new model x², induuiing process:
Continuous-time reinforcement learning. We formulate knite-horizon vontinuous-time reinforcement learning (RL) specific class of stochastic control problems (Wang et al., 2020; Jia & Zhou, 2022; Zhao et al., 2024). Given a state space X' and an action space A, we consider the transition dynamics governed by the following diffusion process, where we invert the direction of the time variable:
dX, b(Xt, t, at )t + no(td with of ~ ... (5)
that rather than imitating the data distribution Pdota aims to induce marginal state distribution pt that maximally explores the approximate data manifold Spre defined as:
Opre= = supp(ppF"
(6)
dx, = b(x,, (, a,)d + (()dB, with No ~
(3)
Formally, we pose the exploration problem as optimization of an entropy functional over the space of marginal distributions p7 supported over the approximate data
where μ E P(X) is an initial state distribution, (Br, 2 0) is d-dimensional Brownian motion, X x A + Rd the drift coefficient, a [0, - RR is the diffusion coefficient, and a, E a selected action. In the following, we consider state space X' == R2 x [0. T). and denote by (Markovian) policy function (X,,, t) (PA) mapping state (x, t) E X' to density over the action space A, and denote with P. the marginal density at time /. induced by policy π. In particular, we will consider deterministic policies so that o, T (X.,
manifold, as shown in Figure 2a. Crucially, Upres which is typically complex set, c.g., molecular space, is defined only implicitly via the pre-trained diffusion model pre as expressed Equation (6). Formally, we state the exploration problem as follows.
Maximum Entropy Manifold Exploration
arg max
H (pT))
Pre-trained diffusion model as an RL policy. A pre- trained diffusion model with score function goee can be interpreted as action process at thee = gre (X , T l), where af is sampled from a continuous-time RL policy ar pre As consequence, we can express the back- ward SDE induced by the pre-trained score sre as follows:
s.t. pf E P(Pppr)
In this formulation, H. F(Qpre) denotes the differential entropy functional quantifying exploration, expressed
H(a) = - d du du H(p)=
E P(Pppe))
(8)
dx
dx; = bbx;, 1t )dt (t)dB
(4)
For this objective to well defined, i.e., the maximum is achieved by some measure μ P(Qppe)) sufficient condi- tion is stated in the following and proved in Appendix A. Proposition 1 (Sepre is compact). Suppose that sre is Lipschitz and the noise distribution po is chosen as the truncated Gaussian. Then Opre spanned by an ODE sampler is compact.
where we define b(rr, L. a. = - -xx - 1+2² (T t) and att) = (T' - t) (Zhao et al., 2024). In the following, we denote the pre-trained diffusion model by its (implicit) policy Thee, which induces a marginal density pre pTp approximating the data distribution Pdttt-
3

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Notice that the assumptions in Proposition are standard for analysis of diffusion processes, (e-g., Lee et al., 2022; Pidstrigach, 2022), and not limiting in practice. Proposition I implies that Spre is a bounded subset of IR,, which according to the manifold hypothesis approximates a lower-dimensional data manifold (Li et al., 2024; Chen et al., 2023; Stanezuk et al., 2024; Kamkari et al., 2024), as illustrated in Figure
F has low density due limited pre-training samples.
Moreover, the KL. regularization in Equation (9) implicitly enforces pf to lie on the approximate data manifold Qpre: Formally, we have that:
2g supp(p7) S supp(p?") Spre Va >> 0 (11)
The entropy first variation in Equation (10) can be interpreted as measure of surprise, while the entropy functional in Equation (7) as expected surprise (Achiam & Sastry, 2017).
Crucially, both the constraint set P(Ppre) and the marginal density P7 in Problem 7 are never represented explicitly, but only implicitly as functions of the pre-trained policy Tere and of the new policy æ respectively.
4.2. Easy to optimize, but hard to estimate density
In the rest of this work, we show that Problem (7) can be solved by fine-tuning the initial pre-trained model with respect to rewards obtained by sequentially linearizing the entropy functional. Towards this goal, in the next section, we introduce scalable algorithmic principle that guides exploration by leveraging the representational capacity of the pre-trained diffusion model.
Existing fine-tuning methods for diffusion models can only optimize linear functionals of př. namely L(²) = (f. E F(X), since they can be represented as classic (reward) functions ʃ : X - R. defined over the design space X, e.g., space of molecules. Although the entropy functional H in Equation (7) is non-linear with respect to p7. its first variation is linear functional. As consequence, by rewrit- ing it as shown in Equation (10), it can be optimized using existing fine-tuning methods for classic reward functions via stochastic optimal control schemes (e.g., Uehara et al., 2024b; Eomingo-Enrice et al., 2024; Zhao et al., 2024), where the fine-tuning objective is:
4. A Principle for Scalable Exploration
As first step towards tackling the maximum entropy manifold exploration problem in Equation (7), we introduce principle for exploration corresponding to a specific (intrinsic) reward function for fine-tuning.
To this end, we define the first variation of a functional over a space of probability measures (Hsieh et al., 2019). A functional F E (F))), where F TP(()) - R. has first variation μ P(X) if there exists function (F()) E (XX) such that for all y E IP()) it holds that:
' E arg max E
201 -lo log (pT (x) a -DDKLPPT.PT
We have shown that exploration can be self-guided by generative model using its representational power of the density it induces. But unfortunately, estimating this quantity (i.e., 1 is well-known to be a challenging task in real-world high-dimensional settings (Song et al., 2020; Kingma et al., 2021; Skreta et al., 2024).
F(u+ qp') = F(4) + e 8F())) + o(c).
where the inner product is interpreted as an expectation.
We can now present the following exploration principle as KL-regularized fine-tuning of the pre-trained model to maximize the entropy first variation evaluated at pf"
4.3. Generative exploration without density estimation
In the following, we show that in the case of diffusion mod- els, the entropy's 's first variation at the marginal density pT induced by n, as in Equation (10) with π = xP"e can be optimized fully bypassing density estimation. This can be achieved by leveraging the following fundamental con- nection between the gradient of the entropy first variation V v.8 (p7)) and the score s² (:,T))
Regularized Entropy First Variation Maximization
" (8H (FT pre p - aDKI(PT.K pre) (9)
arg max x
4.1. Generative exploration via density estimation
Crucially, this algorithmic principle does not rely on explicit uncertainty quantification and uses the generative model's ability to represent the density po direct exploration. By introducing function f X R defined for all X' as:
Gradient of entropy first variation Negative score
v (pT) V. log (.,T)
(12)
Using Equation (12), it is possible to solve the maximiza- tion problem in Equation (9) by leveraging a first-order
f(x) = 87 (PT (x) log (PTT)
(10)
the exploration principle in Equation (9) computes a policy x* inducing pf with high density in regions where
fine-tuning method such as Adjoint Matching (Domingo- Enrich et al., 2024) with Vzf(x)= = - T) as reward
4

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
gradient, where shre is the known (neural) score model approximating the true score function. This realization over- comes the limitation of density estimation and renders the method scalable for high-dimensiona real-world problems. For the sake of completeness, we report a detailed pseu- docode of its implementation in Appendix D.
functional. We say that F is L-smooth relative to e F(2)
over P(X)) if L scalar s.t. for all μ., E IP(x)):
F(v) - F(u) + (6F(1), p) + (D)
(13)
and we say that F I-strongly convex relative to e F(X)
over P(X) if 2 0 scalar s.t. for all μ. E P()):
4.4. Beyond maximum entropy exploration
F(V) 2 F(u) (8F()). - n1) + +Doov, (14)
As shown in Section 8, beyond the goal of maximum (entropy) exploration, the principle in Equation (9) can be used to achieve the desired trade-off between exploration and validity by controlling the regularization coefficient o. Higher values lead to fine-tuned model that conservatively aligns with the validity encoded in the pre-trained model g."C. In contrast, low a values enable exploration of low density regions within the approximate data manifold Ppre. as illustrated in Figure 2b. The latter modality is particularly relevant when a validity checker is available, e.g,, synthetic accessibility (SA) scores for molecules (Ertl & Schuffenhauer, 2009), formal verifiers for logic circuits (Coudert & Madre, 1990), allowing the discovery of new valid designs that expand the current manifold or dataset, effectively performing a guided data augmentation process (Zheng et al., 2023).
In the following, we view the principle in Equation (9) as step of mirror descent (Nemirovskij & Yudin, 1983) and the KL divergence term the Bregman divergence induced by an entropic mirror map Q
-, i.e.
DRI D) = D *(,, We can now state the following lemma regarding F = H.
Lemma 5.1 (Relative smoothness and strong convexity for F = Q = H). For F = Q = - as in Equation (9), we have that F is I-smooth (i.e., L = 1) and I-strongly convex (i.e., 1 1) relative to Q
We can finally state the following set of idealized assump- tions as well as the one-step convergence guarantee.
Assumption 5.1 (Exact estimation and optimization). We consider the following assumptions:
In particular, one might wonder if there exists a value of a such that the obtained fine-tuned model can provably solve the maximum entropy exploration problem in Equation (7). In the following section, we present a theoretical framework that answers this question positively under the idealized as- sumptions of exact score estimation and optimization oracle.
1. Exact score estimation: gare (pre(T,)) V log p
2. The optimization problem in Equation (9) is solved exactly.
Theorem 5.2 (One-step convergence). Given Assump- tions 5.1, fine-tuinn a pre-trained model Tre according to Equation (9) with a = L = 1, leads to a policy x inducing a marginal distribution p7 E PP(2pre) such that:
5. Provably Optimal Exploration in One Step
In this section, we show that under the assumptions of ex- act optimization and estimation of the entropy first vari- ation 8H (, single fine-tuning step using Equation (9) yields an optimally explorative policy π for entropy maximization over Ppree Complete proofs are reported in Appendices and C.
H(pr) H(PT) S DD-DRIPPT
L L- K
"T0) = (15)
where p1== PT. is the marginal distribution induced by the optimal exploratory policy x' E arg max H(VT) with A= {T compatible with the approximate data manifold Dywee
We start by recalling the notion of Bregman divergence induced by functional Q E (XX) between two densities U E IP()),, namely:
pf E P(opre)} being the set of policies
Delu, v) = 2(x) - 2(v) - (8Q(v).
Theorem 5.2 hints promising performances of using Equa- tion (9) as fine-tuning objective for maximum entropy exploration, as under Assumptions 5.1, it provably leads to an optimally explorative policy in one step. However, As- sumptions 5.1 clearly do not hold in most realistic settings as:
Next, we introduce two structural properties for our analysis.?
Definition 1 (Relative smoothness and relative strong con- vexity (Lu et al., 2018). Let F P()) - R a convex
IIn line with standard notations in the optimization literature, we present the framework as minp e((Ppe)) - H(Pr), which is clearly equivalent to (7).
1. The estimation quality of the score (,T") learned from data is approximate.
5

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
2. The high-dimensional stochastic optimal control meth- ods used to solve Equation (9) are approximate in prac- tice.
Eq. (12) (line 4). Ultimately, it returns a final policy π == TK- We report possible implementation of LINEARFINETUNINGSOLVER in Appendix D.
Crucially, S-MEME controls the distributional behavior of the final diffusion model я, which is essential to optimize the entropy as it is a non-linear functional over P(Qppe)).
Therefore, optimizing the entropy first variation as in Equation (9) is actually unlikely to lead to an optimally explorative diffusion model, as shown experimentally in Section 8. To address this issue, in the next section we propose an exploration algorithm by building on the exploration principle in Equation (9).
However, it is still unclear whether the algorithm provably converges to the optimally explorative diffusion model xt" In the next section, we answer affirmatively this question by developing theoretical analysis under general assumptions based on recent results for mirror flows (Hsich et al., 2019),
6. Algorithm: Score-based Maximum Entropy Manifold Exploration
7. Manifold Exploration Guarantees
In the following, we present Score-based Maximum Entropy Manifold Exploration (S-MEME), see Algorithm 1, which reduces manifold exploration to sequential fine-tuning of the pre-trained diffusion model The by following a mirror descent (MD) scheme (Nemirovskij & Yudin, 1983). Crucially, each iteration & of S-MEME corresponds to a fine-tuning step according to Equation (9), where the pre-trained model pPre =: Ty is then replaced by the model at the previous iteration, namely Wk-1- Concretely, this makes it possible to reduce the optimization of the entropy functional, which is non-linear w.r.t. pf. to a sequence of optimization problems of linear functionals.
The purpose of this section is to establish a realistic frame- work under which Algorithm 1 is guaranteed to solve the maximum entropy manifold exploration Problem (7).
7.1. Key Assumptions
We now present all the assumptions and provide an explana- tion of why they are realistic. Conceptually, these assump- tions align with the stochastic approximation framework of Benaïm (2006); Mertikopoulos et al. (2024); Hsieh et al. (2021). Specifically, recall that p f* represents the (stochastic) density produced by the LINEARFINETUNING SOLVER oracle at the k-th step of S-MEME, and consider the following mirror descent iterates:
Algorithm 1 score-based Maximum Entropy Manifold Exploration (S-MEMF)
1 DKI (P.PT= Tt-t)
p = arg max (Rpre)))))
(dH) (p77-)).1
.P)
input K number of iterations, xee : pre-trained diffusion, regularization coefficients
The
{a 1
(MD,)
1: Init: Tu= ;= Pre
where 1/7h- = ag in Algorithm 1.
2: for k= 2==11 1, K do
As explained in Section 5, the maximum entropy manifold exploration problem (7) can be solved in a single step using (MD+). However, in realistic settings where only noisy and biased approximations of (MD;) are available, it becomes essential quantify the deviations due to these approximations from the idealized iterates in (MD;). Additionally, the step sizes " must be carefully designed to account for such deviations. This section aims to achieve precisely this goal. To this end, we first require:
3: Set: Vafa= = - 1-1 with s sh -T--1
4: Compute Ta via first-order linear fine-tuning:
TTk + LINERRFNETUNNNGSOVERR V 2kk Ogee T8-1)
5: end for
output policy क TK
Algorithm 1 requires as inputs a pre-trained diffusion model The the number of iterations K, and a schedule of regular- ization coefficients {a K At each iteration, S S-MEME sets the gradient of the entropy first variation evaluated at the previous policy - namely V.5H (p--)) to be the score s = -t associated to the diffusion model - obtained at the previous iteration (line 3). Then, it computes policy T& by solving the following fine-tuning problem
Assumption 7.1 (Support Compatibility). We assume that supp(p7*) C f.ror all k; and supp(p"*) foror some j.
Next, we require purely technical assumption that is typically satisfied in practice:
Assumption 7.2. The sequence (8H(P7+)) )b is precompact in the topology induced by the Taxe norm.
Now, denote by GR the filtration up to step k. and consider the decomposition of the oracle into its noise and bias parts:
log (p+ 1) (s)) on KLLF- + -
arg max
E
by = E (8H((p*)) SHH(PE) |
(16)
via a ordst solver such as Adjoint Matching (Domingo- Enrich et al., 2024), using Vfk == - (1, T) as in
Uk 8H((Tt) 57((ot) be
(17)
6

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Observe that, conditioned on Gj. Ug has zero mean, while by captures the systematic error. We then impose the following: Assumption 7.3 (Noise and Bias). The following events happen almost surely:
the common setting where the density PT induced by pre- trained model ::"r presents high-density region (yellow area in Figure 3a) and a low-density region (green area in in Figure 3a). As illustrated in Figure 3a, the pre-trained model प induces an unbalanced density, where N = 80000 samples are obtained mostly from the high-density area. For quantitative evaluation, we compute a Monte Carlo estimate (19) of H(()). Crucially, Figure 3 shows that S-MEME can induce a highly explorative density in terms of entropy (see Figure 3d), compared with the pre-trained model, after only K = 4 iterations. One can notice that the density induced by the fine-tuned model (see Figure 3c) is significantly more uniform and higher in low- proobabayllityy regions for the pre-trained model (see right region of Figure 3c), while preserving the support of the data distribution.
| 1xc + 0
(18)
Σ
Ti (1ball20 + |[[rrl22@]
<00
Σ 7k 11bxlloo0 00
(20)
Two important remarks are worth noting. First, (18) repre- sents necessary condition for convergence: if this condi- tion is violated, it becomes straightforward to construct ex- amples where no practical algorithm can successfully solve the maximum entropy problem. Second, (19) and (20) ad- dress the trade-off between the accuracy of the approximate oracle LINEARFINET
pr
S-MEME
S-MEME
S-MEME 3
and the aggressiveness of the step sizes, 7/k- Intuitively, smaller noise and bias allows for the use of larger step sizes. In this context, (19) and (20) establish a concrete criterion ensuring that the task of finding the optimally explorative policy succeeds with probability 1.
FID P PT CLIP H(p, P
0.0
10.25
9.83
19.15
22.27
20.79
20.88
19.86
-1916.47
564.72
482,81
843.88
Table 1: FID, CLIP and cross-entropy evaluation of p/ and p. For k = 1, S-MEME achieves larger distance to PT while preserving high CLIP score.
We are now finally ready to state the following result.
are
Theorem 7.1 (Convergence under general assumptions).
(2) Text-to-image manifold exploration. We consider the problem of exploring the data manifold of creative ar- chitecture designs given pre-trained text-to- so--to--mmge diffu- sion model. For this we utilize the stable diffusion (SD) 1.5 (Rombach et al., 2021) checkpoint pre-trained on the LAION-5B dataset (Schuhmann et al., 2022). Since SD-1.5 uses classifier-free guidance (Ho & Salimans, 2022), we fine-tune the velocity resulting from applying the classifier- free guidance with a guidance scale of 8 which is stan- dard for SD-1 Similarly, we also use the same guidance scheme for the fine-tuned model. We fine-tuned the check- point with K = 3 iterations of S-MEME on a single Nvidia H100 GPU for the prompt A creative architecture.". In Fig- ure 4, we show images generated from qPre, mi and T3, re- sulting from the same initial noise samples. One can notice an increase in the complexity and originality of the respec- tive images, likely hinting at higher probability of the fine- tuned model to sample from lower-density region for ,Pre, Moreover, less conservative architectures are sampled with more steps of S S-MEME while preserving semantic faithful- ness. We measure this in Table 1 by computing the Fréchet inception distance (FID) (Heusel et al., 2017) and Gaussian cross- cross-entropy in feature space of Inception-v3 between p. and pf for k 1, 2, 3 as well as the CLIP score (Hes- sel et al., 2021) for the distribution induced by the specific prompt. The main reason for these proxy metrics is the intractability of computing log 1T (x)((x)) in high- lige-diimnsional spaces, such as that of images generated by a large diffusion model. One can notice from Table an increase in FID and cross-entropy between the distributions as increases,
Consider the standard Robbins-Monro step-size rule:
Σ&nh -===0a =0, Ex Th < oc. Then under Assumptions 7.1
to 7.3, the sequence of marginal densities p induced by
the iterates Th of Algorithm I converges weakly to p4
almost surely. Formally, we have that:
pT
pT a.s.
(21)
where p4 E arg
H(pr) is the maximum entropy marginal density compatible with Opre-
Remark. It is possible to derive an explicit convergence rate for Theorem 7.1 corresponding to ((log log ")) which is, in general, the best achievable without additional assumptions (Karimi et al., 2024). We omit the technical details, they are rather involved and offer limited practical relevance.
8. Experimental Evaluation
In this section, we analyze the ability of S-MEME to in- duce explorative policies on two tasks: (1) An illustrative example to showcase visually interpretable exploration and ability to sample from low-density regions (see Figure 3), and (2) A text-to-image task aiming explore the approxi- mate manifold of creative architecture designs (see Figure 4). Additional details on experiments are provided in Ap- pendix E.
(1) Illustrative setting. In this experiment, we consider
7

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
2.0
0.5
Entropy
1.8
0.0
0.0
0.0
1.6
Pre-trained
S-MEME
-0.
05
-05
1.4
-1.0
-10
-2
S S-MEME Iteratioon
True data support
(b) Pre-trained model sample
Finc-tuned model sample
Entropy evaluation
Figure 3: Illustrative example with unbalanced pre-trained model g/"e. (3a) Support of true data distribution composed of small high-densite region (yellow) and a wide low-density area (green). (3b) Sample from pre-trained model ,rre. (3c) Sample from T4 obtained after 4 steps of S- S-MEME. (3d) Entropy estimation of densities {p& R==1 obtained via K = 4 steps. Notice that S- S-MEME returns fine-tuned model with significantly higher entropy than g (see 3c and 3d), and higher density in lew-density regions for the pre-traines model (compare (3a) and (3c)), while preserving the data support shown in 3a.
Figure 4: Generated images from pre (top) and т3 (bottom) for fixed set of initial noisy samples using the prompt "A creative architecture.". We observe increase in complexity and originality of the S-MEME generated images while preserving semantic faithfulness, likely hinting at higher probability of sampling from a lower-density region of nPre
while the fine-tuned model preserves the CLIP score of the pre-trained model. We provide further results for text-to- image in Appendix G, and in Appendix F, we evaluate the Vendi score (Friedman & Dieng, 2022) diversity metric.
in this area (Liu Abbeel, 2021; Seo et al., 2021; Mutti et al., 2021). Recent studies have tackled maximum entropy exploration with finite sample budgets (e.g., Mutti al., 2022b;a; 2023; Prajapat et al., 2023; De Santi et al., 2024b).
We believe several ideas presented in this work can extend to such settings. Ultimately, to the best of our knowledge, this is the first work providing a rigorous theoretical analysis of maximum state entropy exploration over continuous state spaces, albeit for specific sub-case, as well as leveraging this formulation for fine-tuning of diffusion models.
9. Related Work
In the following, we present relevant work in related areas.
Maximum State Entropy Exploration. Maximum state entropy exploration, introduced by Hazan et al. (2019), addresses the pure-exploration RL problem of maximizing the entropy of the state distribution induced by a policy over dynamical system's 's state space (e.g., Lee et al., 2019; Mutti et al., 2021; Guo et al., 2021). The presented manifold exploration problem is closely related, with pf representing the state distribution induced by policy æ over subset of the state space. Nonetheless, in Problem (7) the admissible state distributions P((Ppre) are represented only implicitly via pre-trained generative model Trre, able to capture complex design spaces, e.g., valid molecules. Moreover, exploration is guided by the diffusion model's score function via Eq. 12, overcoming the need of explicit entropy or density estimation, fundamental challenge
Continuous-time RL. Continuous-time RL extends stochastic optimal control (Fleming & Rishel, 2012) to handle unknown rewards or dynamics (c.g.. Doya, 2000; Wang et al., 2020). Problem (7) represents the pure exploration case of continuous-time RL, were the goal to compute (purely) exploratory policy л over subset of the state space Sxrre S X' implicitly defined by pre-trained generative model nrte, Moreover, Problem (7) can be further motivated learning setting (c.g., Lindner et al., 2021; Mutny et al., 2023; De Santi et al., 2024a), where an agent aims to learn an unknown homoscedastic reward function such as toxicity over a molecular space (Yang et al., 2022). To our
continuous-time RL reward
8

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
knowledge, this is the first work that tackles maximum entropy exploration in continuous-time RL setting.
This idea of optimization with implicit constraints captured by generative models is novel and absent in earlier work.
Diffusion models fine-tuning via optimal control. Recent works have framed diffusion models fine-tuning with respect to a reward function f X + IR as an entropy- regularized stochastic optimal control problem (e.g., Uehara et al., 2024a; Tang, 2024; Uehara al., 2024b; Domingo-Entin EnrochErrchhhh et al.. 2024). In this work, we introduce scalable fine-tuning scheme, based first-order solvers for classic rewards (e.g., Domingo-Enrich et al., 2024), that optimizes a broader class of functionals requiring information about the full density pl- such as entropy and alternative exploration measures (De Santi et al., 2024b; Hazan et al., 2019). This paves the way to using diffusion models for optimization of distributional objectives, rather than simple scalar rewards. Beyond classic optimization, our framework is particularly relevant for Bayesian optimization, or bandit, problems (e.g., Uehara et al., 2024b), where the reward function to be optimized over the manifold is unknown and therefore exploration is essential.
10. Conclusion
This work tackles the fundamental challenge of leveraging the representational power of generative models for exploration. We first introduce a formal framework for exploration as entropy maximization over the approximate data manifold implicitly captured by a pre-trained diffusion model. Then, we present an algorithmic principle that guides exploration via density estimation, a challenging task in real-world settings. By exploiting a fundamental connection between entropy and diffusion model's 's score function, we overcome this problem and ensure scalability of the proposed principle for exploration. Building on this, introduce S-MEME, sequential fine-tuming algorithm that provably solves the exploration problem, with convergence guarantees grounded in recent advances in mirror flows. Finally, we validate the proposed method on both a conceptual benchmark and a high-dimensional text-to-image task, demonstrating its practical relevance.
Sample diversity in diffusion models generation. The lack of sample diversity in diffusion model generation is a key challenge tackled by various works (c.g.. Corso et al., 2023; Um et al., 2023; Kirchhof et al., 2024; Sadat et al., 2024; Um & Ye, 2025). These methods complement ours by enabling diverse sampling from the fine-tuned explorative model ob- tained via S-MEME. While prior works focus on generating diverse samples from a fixed diffusion model, ours provides a framework for manifold exploration as policy optimization via reinforcement learning. This enables scalable and prov- able maximization of typical exploration measures in RL, such as state entropy (Hazan et al., 2019). Among related works, Miao et al. (2024) shares the closest intent, but lacks a formal setting with exploration guarantees, and the dif- fusion model's s exploration process relies on computing an external metric for exploration, rather than being self-guided via its own score function S-MEME achieves via Eq. (12).
A Acknowledgements
This publication was made possible by the ETH AI Center doctoral fellowship to Riccardo De Santi, and postdoctoral fellowship to Marin Vlastelica. The project has received funding from the Swiss National Science Foundation under NCCR Catalysis grant number 180544 and NCCR Automa- tion grant agreement 51NF40 180545.
Impact Statement
In this work we provide a method and theoretical analysis of maximum-entrop exploration for diffusion generative models as well as an implemented algorithm. To the best of our knowledge, this will enable further more applied research in this areas with exciting applications.
Optimization over probability measures via mirror flows. Recently, there has been a growing interest in analyzing op- timization problems over spaces of probability measures. Existing works have explored applications including GANs (Hsieh et al., 2019), optimal transport (Aubin-Frankee Frankowski et al., 2022; Léger, 2021; Karimi al., 2024), and kernel- ized methods (Dvurechensky & Zhu, 2024). However, to the best of our knowledge, the case of the entropy-based objective in Problem 7, along with its associated relative smoothness analysis framework, has not been previously ad- dressed. Moreover, prior approaches do not leverage a key aspect of our framework: the admissible space of probability measures, P(Qppre), is represented only implicitly via a pre- trained generative model, which can approximate complex data manifolds learned from data, such as molecular spaces.
References
Achiam, J. and Sastry, S. Surprise-based intrinsic moti- vation for deep reinforcement learning. arXiv preprint 170301770,, 2017.
hubin-Frankowski, P.-C., Korba, A., and Léger, F. Mirror descent with relative smoothness in measure spaces, with application to sinkhorn and em. Advances in Neural Information Processing Systems, 15:17263-17275 2022.
Benaim, M. Dynamics of stochastic approximation algo- rithms. In Seminaire de probabilites XXXIII, pp. 1-68. Springer, 2006.

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Benaïm, M. and Hirsch, M. W. Asymptotic pseudotrajec- tories and chain recurvent flows, with applications. Jour- nal of Dynamics and Differential Equations, 8:141-176, 1996.
Ertl, P. and Schuffenhauer, A. Estimation of synthetic acces- sibility score of drug-like molecules based molecular complexity and fragment contributions. Journal of chem- informatics, 1:1-11, 2009.
Fleming, W. H. and Rishel, R. W. Deterministic and stochas- lic optimal control, volume 1. Springer Science & Busi- ness Media, 2012.
Bilodeau, C., Jin, W., Jaakkola, T., Barzilay, R., and Jensen, K. F. Generative models for molecular discovery: Recent advances and challenges. Wiley Interdisciplimary Reviews: Computational Molecular Science, 1668) 2022.
Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210022110. 2022.
Chen, M., Huang, K., Zhao, T., and Wang, M. Score approx- imation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Con- ference on Machine Learning, pp. 4672-4712. PMLR, 2023.
Guo, Z. D., Azar, M. G., Saade, A., Thakoor, S., Piot, B., Pires, B. A., Valko, M., Mesnard, T., Lattimore, T., and Munos, R. Geometric entropic exploration. arXiv preprint arXiv:2101.02055 2021.
Chi, C., Feng. S., Du, Y., Xu, Z., Cousineau, E., Burch- fiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Halmos, P. R. Measure theory, volume 18. Springer, 2013.
Hazan, E., Kakade, S., Singh, K., and Van Soest, A. Prov-
ably efficient maximum entropy exploration. In Interna- tional Conference on Machine Learning, 2019.
Corso, G., Stärk, H., Jing, B.. Barzilay, R., and Jaakkola, T. Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776. 2022.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Corso, G., Xu, Y., De Bortoli, V., Barzilay, R., and Jaakkola, T. Particle guidance: non-iid diverse sampling with diffu- sion models. arXiv preprint 1..1:23310.13002 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter. S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Coudert, O. and Madre, J. C. A unified framework for the formal verification of sequential circuits. In The Best of ICCAD: 20 Years of Excellence in Computer-Aided Design, pp. 39-50. Springer, 1990.
Hiriart- Prriart-Urruty,. J.-B. and Lemaréchal, C. Fundamentals of convex analysis. Springer Science & Business Media, 2004.
De Santi, R., Joseph, F. A., Liniger, N., Mutti, M., and Krause, A. Geometric active exploration markov deci- sion processes: the benefit of abstraction. arXiv preprint arXiv:2407. 13364, 2024a.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
De Santi, R., Prajapat, M., and Krause, A. Global rein- forcement learning: Beyond linear and convex rewards via submodular semi- eeniggradient methods. arXiv preprint arXiv:2407.09905, 2024b.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Hoogeboom. E., Satorras, V. G., Vignac, C., and Welling, M. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867- 8887. PMLR, 2022.
Domingo-Enrich. C., Drozdzal, M., Karrer, B., and Chen, R. T. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.0888) 2024.
Hsieh, Y.-P., Liu, C., and Cevher, V. Finding mixed nash equilibria of generative adversarial networks. In Interna- tional Conference on Machine Learning, pp. 2810-2819, PMLR, 2019.
Doya, K. Reinforcement learning in continuous time and space. Neural computation, 12(1):219-245. 2000.
Dvurechensky, P. and Zhu, J.-J. Analysis of kernel mirror prox for measure optimization. International Confer- ence on Artificial Intelligence and Statistics, pp. 2350- 2358. PMLR, 2024.
Hsich, Y.PP., Mertikopoulos, P., and Cevher, V. The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In International Conference on Machine Learning, pp. 4337-4348 PMLR, 2021.
10

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Jia, Y. and Zhou, X. Y. Policy evaluation and temporal- difference learning in continuous time and space: A mar- tingale approach. Journal of Machine Learning Research, 23 23(154):1-55, 2022.
Mertikopoulos, P., Hsieh, Y.-P., and Cevher, V. A uni- fied stochastic approximation framework for learning in games. Mathematical Programming, 203(():559-609, 2024.
Kamkari, H., Ross, B. L., Hosseinzadeh, R., Cresswell, C., and Loaiza-Ganena G. A geometric view data complexity: Efficient local intrinsic dimension estimation with diffusion models. arXiv preprint arXiv:2406.03537, 2024.
Miao, Z., Wang, J., Wang, Z., Yang. Z., Wang. L., Qiu, Q. and Liu, Z. Training diffusion models towards di- verse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10844-10853, 2024.
Karimi, M. R., Hsieh, Y.-P., and Krause, A. Sinkhorn flow as mirror flow: A continuous-time framework for general- izing the sinkhorn algorithm. In International Conference on Artificial Intelligence and Statistics, pp. 4186-4194. PMLR, 2024.
Mutny, M., Janik, T., and Krause, Active exploration via experiment design in Markov chains. In International Conference on Artificial Intelligence and Statistics, 2023.
Mutti, M., Pratissoli, L., and Restelli, M. Task-agnostic exploration via policy gradient of non para-meettticc state entropy estimate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9028-9999 1028-9036, 2021.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 14:21696-21707, 2021.
Kirchhof, M., Thornton, J., Ablin, P., Béthune, I.., Ndiaye, E., and Cuturi, M. Sparse repellency for shielded gener- ation in text- -magee diffusion models. arXiv preprint arXiv:2410.06025 2024.
Mutti, M., De Santi, R., De Bartolomeis, P., and Restelli, M. Challenging common assumptions in convex rein- forcement learning. Advances in Neural Information Processing Systems, 35:4489-4502, 2022a.
Mutti, M., De Santi, R., and Restelli, M. The importance of non-markovianity in maximum state entropy exploration. International Conference on Machine Learning, pp. 16223-16233 PMLR, 2022b.
Lec, H., Lu, J., and Tan, Y. Convergence for score-based generative modeling with polynomial complexity. Ad- vances in Neural Information Processing Systems, 35: 22870-22882, 2022.
Mutti, M., De Santi, R., De Bartolomeis, P., and Restelli, M. Convex reinforcement learning in finite trials. Journal of Machine Learning Research, 24(250):1 2023.
Lee, I.,, Bysenbach, B., Parisotto, E., Xing, E., Levine, S., and Salakhutdinov, R. Efficient exploration via state marginal matching. arXiv preprint arXiv: urXiv:1906.055 1906.0527, 2019.
Nemirovskij. A. S. and Yudin, D. B. Problem complexity and method efficiency in optimization. 1983.
Léger, F. A gradient descent perspective on sinkhorn. Ap- plied Mathematics & Optimization, 84(2):1843-1855, 2021.
Pidstrigach, J. Score-based generative models detect mani- folds. Advances in Neural Information Processing Sys- tems, 35:35852-3555 2022.
Li, Z., Yuan, H., Huang, K., Ni, C., Ye, Y., Chen, M., and Wang, M. Diffusion model for data-driven black-box optimization. arXiv preprint arXh::4403. 2024.
Prajapat, M., Mutný, M., Zeilinger, M. N., and Krause, A. Submodular reinforcement learning. arXiv preprint arXiv:2307. 13372, 2023.
Lindner, D., Turchetta, M., Tschiatschek, S., Ciosek, K., and Krause, A. Information directed reward learning for reinforcement learning. Advances in Neural Information Processing Systems, 34:3850-3862, 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021.
Liu, H. and Abbeel, P. Behavior from the void: Unsuper- vised active pre- training Advances in Neural Information Processing Systems, 34:18459-18477 2021.
Sadat, S., Buhmann, J., Bradley, D., Hilliges, O., and We- ber, R. M. Cads: Unleashing the diversity of diffusion models through condition sampling, 2024. URL https: / www.ariivvv///..../aaaaaaaaaaava wtwpaawwwwww.a.aaaaaaaaa hat.aa////311.11..
Lu, H., Freund, R. M., and Nesterov, Y. Relatively smooth convex optimization by first-order methods, and appli- cations. SIAM Journal on Optimization, 28(1):333-355 2018.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: open large-scale

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294, 2022.
Vincent, P. A connection between score matching and de- noising autoencoders. Neural computation, 23(7): 1661 1674, 2011.
Seo, Y. Chen, L., Shin, J., Lee, H., Abbeel, P., and Lee, K. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pp.
Wang. H., Zariphopoulou, T., and Zhou, X. Y. Reinforce- ment learning continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1-34, 2020.
PMLR, 2021.
Skreta, M., Atanackovic, L.., Bose, A. J., Tong, A., and Neklyudov, K. The superposition of diffusion mod- using the it o density estimator. arXiv preprint arXiv:2412.17762. 2024.
Yang, Y., Wu, 7.,, Yao, X., Kang. Y., Hou, T., Hsieh, C.- Y., and Liu, H. Exploring low-toxicity chemical space with deep learning for molecular generation. Journal of Chemical Information and Modeling, 62(13):311-3319, 2022.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Zeni, C., Pinsler, R., Zügner, D., Fowler, A., Horton, M.. Fu, X., Shysheya, S., Crabbé, J., Sun, L., Smith, J., et al. Mattergen: generative model for inorganic materials design. arXiv preprini arXiv:23 22.0v:::33120088.... 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Zhao, H., Chen, H., Zhang. J., Yao, D. D., and Tang, W. framework of fine-tuning diffusion models by continuous-time reinforcement learning. arXiv 2024.
Scores as actions:
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
preprint
Zheng, C., Wu, G., and Li, C. Toward understanding gener- ative data augmentation. Advances in neural information processing systems, 36:54046-540000 2023.
Stanezuk, P., Batzolis, G., Deveney, T., and Schönlieb, C.-B. Diffusion models encode the intrinsic dimension of data manifolds. In Forty-first International Conference on Machine Learning. 2024.
Tang, W. Fine-tuning of diffusion models via stochastic con- trol: entropy regularization and beyond. arXiv preprint arXiv:2403.06679 2024.
Tang, W. and Zhao, H. Contractive diffusion probabilistic models. arXiv preprint 13Xiv:2401.13/ 15, 2024.
Uehara, M., Zhao, Y., Black, K., Hajiramezanali, E., Scalia, G., Diamant, N. L., Tseng, A. M., Biancalani, T., and Levine, S. Fine-tuning of continuous-time diffusion models entropy regulaieee control. arXiv preprint arXiv:2402. 1519 2024a.
Uehara, M., Zhao, Y.. Black, K., Hajiramezanali, E., Scalia, G., Diamant, N. L.,, Tseng, A. M., Levine, S., and Bian- calani, T. Feedback efficient online fine-tuning of diffu- sion models. arXiv preprint 16355,,, 2024b.
Um, and Ye, J. C. Self-guided generation of minority samples using diffusion models. In European Conference Computer Vision, pp. 414 430. Springer, 2025.
Um, S., Lee, S., and Ye, C. Don't play favorites: Mi- nority guidance for diffusion models. arXiv preprint arXiv:2301. 12334, 2023.
12

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
A. Proof for Proposition 1
Recall the probability flow ODE in (Song et al., 2020, eq. (13)), which is what we use to generate pf (this also common practice in the literature). We know that the generative ODE corresponding the backward SDE Equation (4) is written (suppose 1) for simplicity)
dxp = - 29 (T - -T
dt.
x +))
Here f is defined in the forward process Equation (1). Clearly the velocity field u(x, t) is Lipschitz w.r.t due the assumption that sere is Lipschitz and the fact that f is linear w.r.t Consequently, the flow map induced by the above ODE is Lipschitz. As result, since Xo is sampled from truncated Gaussian distribution which has compact support, x = supp(pT ) is also compact for any finite T.
13

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
B. Proof for Section 5
Theorem 5.2 (One-step convergence), Given Assumptions 5.1, fine-tuning a pre-trained model qære according to Equation (9) with a L leads to a policy æ inducing a marginal distribution pr. E pSpppr)) such that:
L L- - K
H(pf)) H(pT)
S DKL
=0
(15)
where pi- == Pl- is the marginal distribution induced by the optimal exploratory policy xt" E arg m H(p)) with A { p7. E [P(Spre)} being the set of policies compatible with the approximate data manifold Ppre.
Proof. Towards proving this result, we interpret Eq. (9) as the first iteration of Algorithm 1. Hence, to prove the statement, it is sufficient to show that Algorithm after one iteration computes тi inducing density P7 such that H(ph) = H(p7). We prove this result by leveraging the properties of relative smoothness and relative strong convexity introduced in Sec.
The analysis bases on a classic analysis for mirror descent via relative properties (Lu al., 2018) First, we show the following, where for the sake of using simple notation, we denote PT by Ph,, and consider arbitrary density P(2lpre)
H(m)) S (((x-1))+++ + (((()-)))+((((()))))))))) S M((x=11) + (6H(( - LDo(U, 1x-1) - LDQ(U,,))
1k-1) + LDQ(Pk: Pk-1)
(22)
(23)
where in the first inequality we have used the L-smoothness of H relative to Q H in Definition 1, while in the last inequality we have used the three-point property of the Bregman divergence (Lu et al., 2018, Lemma 3.1) with ((4)= = (88H((s-1). pk-1), z= p4-1, and x Then, we can derive:
H(px) - H(() + (L D)(H Pk-1) - LDQ(H,
(24)
by using the I-strong convexity of H relative to Q H. in Definition 1. By induction, using monotonicity of the iterates and non-negativity of the Bregman divergence as in (Lu et al., 2018), one obtains:
K
(H(())) - H(m) & LDo(U, mo) - T, (T,-7) Delu, S I.De(1,1))
L
(25)
L 1-1
k=
Defining:
K
1
Σ
(26)
C₁
L L-1
k=1
and rearrenging the terms leads to:
p.DQ(u, (1+ + )
H(ux) H(m) S CaLDolu,po)
(27)
Given Eq. 27, the convergence in the statement can be derived using Lemma 5.1, and the fact that (1+
1 z
1+ ky T. Ultimately, pT E P(D > 0 is trivially due to the fact that Opre is the support of the right element of the Kullback-Liibler divergence in Eq. 9.
14

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
C. Proof for Section 7
C.1. Proof of Theorem 7.1
We restate the theorem for reader's convenience:
Theorem 7.1 (Convergence under general assumptions). Consider the standard Robbins-Monro step-size rule: En 00. Ex>h Rh < oc. Then under Assumptions 7.1 to 7.3, the sequence of marginal densities pi induced by the iterates Te of Algorithm I converges weakly to Pr almost surely. Formally, we have that:
PT -r Pr a.s.
(21)
where p7 E arg max px H(PT) is the maximum entropy marginal density compatible with Spre.
Proof. To enhance the readability of our proof, we begin by outlining the key steps.
Proof Outline. The main idea is to analyze the convergence of the iterates {or JeEN generated by Algorithm by relating them to corresponding continuous-lie dynamical system. Specifically, define the initial dual variable
ho 5HH(ppre) - log Ppre.
and consider the following system:
{ h4 8H(p)) Pt = 8( ((-)))(he)
h4 log ahe
=
(MF)
Pr Ja ehr
Here, (h) = log Jo ex is the Fenchel dual of the entropy function (Hsieh et al., 2019; Hiriart-Urruty & Lemaréchal, 2004).
To bridge the gap between discrete and continuous-time time analysis, we construct a continuous-time interpolation of the discrete iterates )EN, Let (At == 8H((T)))EN be the sequence of the corresponding dual variables. We introduce the notion of an "effective time" 7h, defined as:
Yn
which represents the cumulative time elapsed up the k-th iteration of the discrete-time process h* using step-size The Using 7", we define the continuous-tine interpolation h (t) of h follows:
h(t) = h
(h++1 h*)..
(Int)
---
Intuitively, the convergence of our algorithm follows if the following two conditions hold:
Informal Assumption (Closeness of discrete and continuous times). The interpolated process (Int) asymptotically
approaches the continuous-time dynamics in (MF) as k + oc.
Informal Assumption 2 (Convergence of continuous- dynamics). The trajectory of (MF) converges to the optimal solution of the maximum entropy problem (7).
To formalize the above intuition, we leverage the stochastic approximation framework of Benaim (2006); Mertikopoulos
al. (2024); Karimi et al. (2024), outlined follows.
First, to precisely state Informal Assumption 1, we introduce a measure of "closeness" between continuous orbits. Let 2 denote the space of integrable functions on Ω (viewed the dual space of probability measures; see (Halmos, 2013)), and define the flow θ: 14 x Z + Z associated with (MF). That is, for an initial condition ho = hh E Z, the function θ describes the orbit of (MF) at time E R..
We then define the notion of "asymptotic closeness" as follows:
15

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Definition 2. We say that (()) is asymptotic pseudotrajectory (APT) of (MF) for all T > have:
sup - (.()(t))l = 0. to 0S
lim
(28)
This comparison criterion, introduced by Benaïm & Hirsch (1996), plays a central role in our analysis. Intuitively, states that (t) eventually tracks the flow of (MF) with arbitrary accuracy over arbitrarily long time windows. Consequently, (Int) is an APT of (MF), we can reasonably expect its behavior and thus that of [h*]keN JkeN to closely follow (MF).
The precise connection is established by Benaïm & Hirsch (1996) through the concept of internally chain-transifive (ICT) sets:
Definition 3 (Benaüm & Hirsch, 1996; Benaïm, 2006). Let S be a nonempty compact subset of Z. Then:
1. S is invariant (S) S for all t E R.
2. S is attracting if it is invariant and there exists compuct neighborhood K. of S such that lim aoe dist(O,(h), uniformly for all h E K.
3. S is an internally chain-transitive (ICT) set it is invariant and O admits no proper attractors within S.
The significance of ICT sets lies in (Benaïm, 2006, Theorem 5.7):
Theorem C.I (APTs converge to ICT sets). Let futt) be a precompact asymptotic pseudotrajectory generated by {hr JkeN for the flow associated with the continuous-tine system (MF). Then, almost surely, ht - S, where S is an ICT set of (MF).
By Theorem C.I, establishing Theorem 7.1 reduces to proving the following two statements:
1. The iterates (h keN of Algorithm generate a precompact APT of (MF).
2. The unique ICT set of (MF) the solution to the optimization problem (7).
These results provide the rigorous counterpart to Informal Assumptions 1 to 2. The proof below proceeds by formally establishing each of these points.
The ICT set of (MF) is the solution to (7). By the definition (MF), can easily see that:
ehe
So he ehe
p: = pehe In So ehte
(29)
Pt (he - Bp.lie)
(30)
We then compute:
d
H(pi)
= (8H())))
(31)
dt
= (loggr. (11 - Epp Bep.he))
by (30)
(32)
I pr log pehe - -f pr logg: [ pihe
(33)
plogp, (1, Pt logo
)²
by (MF)
(34)
E Exiopi (log (X,)) (Ex, (Exe~pe log (x.))
(35)
< 0
(36)
by Jensen's inequality. Also, note that the inequality is strict if h4 is not constant, i.e., if pt is not uniform on Ω.
In short, we established (36) that H((-) serves as a Lyapunov function for the continuous-timg time system (MF). Since H((-) is strictly concave, the only ICT set is the singleton {,r), where p represents the uniform (and hence mntropy-maximinn measure on Ω (Benaïm, 2006, Proposition. 6.4).
16

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Algorithm 1 generates APT. Let (p/r) kEN be the sequence of measures Ωt generated by Algorithm 1 with the oracle LANIAAEFNNTTTTTNNNSSSNVEEE and recall that its dual variables given by (h* = 8'H(TT )aEN. Also, recall the corresponding continuous-time interpolation (Int).
Assumption 7.1 ensures that each dual variable h is a well-defined function on S2 after some iteration j. while Assumption 7.2 guarantees the precompactness of h.).. Furthermore, under Assumption 7.3, standard arguments (see, e.g., Proposition 4.1 of (Benaim, 2006) or (Karimi et al., 2024)) establish that h(-) generates an APT of the continuous-time flow defined by (MF). Finally, Theorem C.I ensures that [h* kKEN converges almost surely an ICT of (MF), which have already shown contain only {pr}.
Therefore, applying the theory of (Hsieh et al., 2021; Karimi al., 2024), we conclude that, almost surely,
lim h* = lim 8H(pf))
lim - log 8H((PT) in Loo. k-+00
(37)
Since Q is compact, (37) implies that, for any smooth test function vi on 92 ( v) + ()) which completes the proof.
17

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
D. Detailed Example of Algorithm Implementation
D.I. Pseudocode for implementation of Eq. (9)
For the sake of completeness, the following we present the pseudocode for possible implementation of LINEARFINE TUNINGSOLVER via first-order optimization method, used solve (9), as well within S-MEME. In particular, we present the same implementation the linear fine-tuning via stochastic optimal control problem and solves it via regression.
in Sec. 8, based Adjoint Matching (Domingo-Enrich et al., 2024), which captures
In the following, adopt the notation from the Adjoint Matching paper (Domingo-Enrich denote the pre-trained noise predictor by e the fine-tumed one as finctuned and with ā the cumulative noise schedule, as used by Ho et al. (2020). The complete algorithm is presented in Algorithm 2. First, notice that given noise predictor (as Defined Sec. 2) and a cumulative noise schedule a, one can define the score s as follows (Song & Ermon, 2019);
al., 2024, Apx E.4). We
((x,t) v1 Oz
s(x,
(38)
Algorithm 2 LINEARFINETUNINGSSN (Implementation based on Adjoint Matching (Domingo-Enrich et al., 2024))
input N number of iterations, e pre-trained noise predictor, regularization coefficient, m trajectories batch size,
ff: reward function gradient
1: Init: fiuetuned = ere with parameter 0
2: for = 0, 2,...,
do
Sample m trajectories {X1] according DDPM (Song et al., 2020), e.g., sample 6, I),, Xo 10
X
āc+1 āi
X;
8r Bet1
finetuned
(X,, +
1 āc-1
11 αt ax+1
C
v1
Use reward gradient:
vx(x))
For each trajectory, solve the lean adjoint ODE, (Domingo-Eniie Enrich et al., 2024, Eq. 38-39), from t T 0:
0R a 2.41 Vx,
(V
0't1 Oz
Xx
aett
á
pro
Xx
v1 āt
Where X, and å, are computed without gradients, i.e., X, sstoggrad(x))... stopgrad((.). For each trajectory compute the Adjoint Matching objective (Domingo-Enrich et al., 2024, Eq. 37):
[())
Σ = 1=0
áet1
(1
Cee
X finctaned( (X2,) c (PTe(x))))
11 1 u2+1 (1
C
āe (1 xr+1)
a1+1
1
1 āi
ac+1
Compute the gradient V VoC(A) and update 0.
4: end for output Fine-tuned noise predictor finetuned
18

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
E. Experiment Details
In this section we provide further details on the experiments.
Illustrative setting. Pre-training was performed by standard denoising score- sooremmatching and uniform samples, namely 10K, from the two distributions in Fig. 3a. For fine-tuning, in this experiment
S-MEME for 6000 gradient steps in total,
for K = 1, 2, 3, 4. Notably, k=1 amounts to having a fixed reward for fine-tuning. - lo " Each round of S S-MEME performs 6000/At K for the particular experiment. In this way we observe the effect of having more rounds of the mirror descent scheme, with same number of gradient updates. Since we utilize Adjoint Matching (Domingo-Enries Ennich et al., 2024) for the linear solver in Algorithm 1, we perform an iteration of Algorithm 2 by first sampling 20 trajectories via DDPM of length 400 that are used for solving the lean adjoint ODE with the reward -XV pr()) and 0.1. Subsequently we perform stochastic gradient steps by the Adam optimizer with batch size 2048, initialized with learning rate 4 10 For the density plots in Figure 3 we sampled 80000 points with 100 DDPM steps. To obtain Figure 3d, computed Monte-Carke estimate of H((p7) with an approximation of log PT (z) resulting from the instantaneous change of variables and divergence flow equation,
log p(x) = log))(z) + /-" V [at, ()dt,
(39)
where / is the velocity of the probability-flow ODE for the variance-preserving forward process of the diffusion.
Text-to-image architecture design. For obtaining Figure 4, similarly in the illustrative example used Algorithm 2 as the linear solver. At each iteration of Algorithm 2 we sample 4 trajectories of length 60 by DDPM, conditioned on the prompt on top of which we perform 10 Adam steps with initial learning rate 3 x 10-7 and batch size 8. Each iteration of Algorithm entails 20 iterations of Algorithm 2. We ran Algorithm 1 for K = 3. For this experiment, we used X = 0.1.
Text-to-image evaluation. Evaluating the entropy of PT is computationally prohibitive for the case of the high- dimensional latent of SD-1.5, Consequently, we opted for proxy metries to quantify how much does the distribution change with increase of Tg. the FID score for distributional distance and CLIP score for semantic alignment. In addition, computed the cross-entropy in Table 1 between the Gaussians in the Inception-v3 feature space, where the Gaussians were fitted the same way as in computing the FID score. The FID score, cross-entropy and CLIP score have been computed on 3000 samples from respective conditional distributions.
F. Diversity Measures
Picking a proper diversity measure for the space of images is non-trivial. For completeness, we provide here additional results for the Vendi score (Friedman & Dieng, 2022), which is kernel diversity metric. This however again comes with particular choice of kernel and feature map @. We picked the standard RBF kernel,
k(x, x²) = exp(->lló(z) - <b(2))l),
(40)
where set 7 50 and pick the Inception-v3 model feature space. Given computation constraints, re-ran another experiment with the prompt "A creative architecture." and evaluated the sampled for each MD iteration with 300 images (samples are given in Figure 5) with x = 0.1 and initial learning rate for Adam being 10-6 with 2 MD steps and 100 gradient steps per MD iteration. The reported numbers for Vendi and CLIP scores in Table are computed over 3 seeds.
pT
S-MEME 1
S-MEME
Vendi
1.53
1.7 + 0.08
1.63 + 0.07
CLIP
0.22
0.22 + 0.01
0.22 + 0.01
Table 2: FID, CLIP and cross-entropy evaluation of p and pr For = 1,2, S-MEME achieves larger distance to p pre while preserving high CLIP score.
G. Additional Text-to-Image Results
In the following, we present additional experimental results obtained via the same text-to- text-to-image pre-traine pre-trained diffusion model introduced in Sec. 8, and with experimental details as presented within Sec. E.
19

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Figure 5: Generated images from pre (top) and T3 (bottom) for a fixed set of initial noisy samples using the prompt "A creative architecture.".
Figure 6: Generated images from the with prompt "A creative impressionist painting."
20

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Figure 7: Generated images obtained via fine-tuning of Tree via S S-MEME with prompt "A creative impressionist painting."
Figure 8: Generated images from pre with prompt "Creative furniture."
21

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Figure 9: Generated images obtained via fine-tuning of TTe via S-MEME with prompt "Creative furniture."
Figure 10: Generated images from The with prompt "An innovative car design."
22

---

Provable Maximum Entropy Manifold Exploration via Diffusion Models
Figure 11: Generated images obtained via fine-tuning of pre via S-MEME with prompt "An innovative car design." "
23


Reviews:
[{"review_id": "y3eqpzGKBi", "reviewer": "Reviewer_WBXm", "summary": "This paper considers the problem of exploring the underlying data manifold learned with a diffusion model. \n\nIt formulates the problem as maximizing the entropy of the probability distribution, and proposes to solve it with a KL-regularized optimization of the first variation of the entropy. It proves that the optimal is obtained given perfect training optimization.\n\nFurthermore, it considers the fact that in practice the training and optimization have errors, and propose an iterative fine-tuning algorithm. It proves that this algorithm converges to the optimal solution under general and realistic assumptions.\n\nThe method is theoretically connected to continuous-time RL.\n\nThe method is empirically validated with an illustrative example and fine tuning a Stable Diffusion model for text-to-image task.\n\n## update after rebuttal\nThank you for providing the theoretical results and rationales for the evaluation metrics. I agree with these rationales. I think it would be interesting to explore the application of this method such as drug discovery, as future directions.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for recognizing our work as very well written, theoretically solid, and practical. In the following, we address several points and questions mentioned within the review.\n\n**Inception Score (IS) and Vendi Score (VS).**\n\nIS: To the best of our understanding, the Fréchet Inception Distance (FID) has been introduced as a way to mitigate the inability of the IS score to recognize intra-class mode collapse [1], which is related to what we are aiming to measure in our setting. In particular, in our setup with images, the generated points would all be part of one class. As a consequence, IS might not be a better evaluation measure than FID. \n\nVS: We agree with the Reviewer that this measure of diversity might be relevant in this context. Nonetheless, choosing a relevant kernel in practice might be non-trivial, and if the feature space is obtained via a non-linear map, which is typically the case, it is unclear how well this measure aligns with entropy. In any case, we are grateful for this suggestion and will certainly explore it in this context.\n\n**Relevance to de-bias pre-trained generative models**\nAlthough not strictly related to the motivation presented in this work, we agree that the presented method is very relevant to de-bias generative models, and further research could focus on its potential impact on this important problem.\n\n**Question 1.**\nThe presented method can run using any state-of-the-art (linear) diffusion model fine-tuning scheme, such as Adjoint Matching [2] or alternative methods, e.g., [3]. To the best of our knowledge, these methods are very scalable and they have been shown to successfully fine-tune large-scale pre-trained models for images [2,3], molecules [3], and proteins [3], among others. Crucially, due to Eq. (12) within Sec. 4.3, our algorithm becomes effectively as scalable as the state-of-the-art methods mentioned above, which have already proved successful in relevant real-world applications with larger pre-trained models [2,3].\n\n**Question 2.**\nWe can establish convergence guarantees under three different assumptions, ordered by increasing generality:\n-  Perfect fine-tuning: If fine-tuning is exact, our algorithm terminates in a single step (see Section 5).  \n\n- Unbiased noise oracle: When the noise oracle is unbiased (i.e., $\\( b_k = 0 \\)$ in (16)), standard mirror descent analysis yields a convergence rate of $\\( O(k^{-1/2}) \\)$ (see [7]).  \n\n- General bias term: If the noise oracle in Eq. (16) includes a general bias term, then under the arbitrary slow decay assumption in Eq. (18), a polynomial-time guarantee is no longer feasible [6]. In this case, stochastic approximation techniques are required, and the best achievable rate is  $\\tilde{\\mathcal{O}}((\\log\\log k)^{-1})$, which follows from our proof. We chose to present an asymptotic result rather than explicitly stating this rate, as the difference is negligible and in this case it is conventional to present the asymptotic result [6,7].\n\nAmong these, the third setting is the most practical, which is why we focused on it in Section 7. Notably, our results show that convergence does not require running $\\( k \\to \\infty \\)$ even with the most general assumptions (i.e., case 3 above).\n\n\n**Question 3.**\nThe presented version of S-MEME relies on optimal control methods as discussed in Sec. 4.2. Nonetheless, it is standard to refer to this recent problem of optimization over the space of admissible state distribution to maximize a known functional as Convex RL [4] or General Utilities RL [5]. This is the main reason why we use the RL notation. Moreover, to the best of our understanding, nothing prevents to replace the control oracle currently used with classic MDP planning methods used in RL. In conclusion, we agree with the Reviewer that the choice of introducing the problem with RL notation is mostly a notational convention.\n\nUltimately, we want to thank the Reviewer for reporting a typo, which we have corrected, and mentioning several relevant related references of which we were not aware.\n\n**References**\n\n[1] Lucic et al., Are GANs Created Equal? A Large-Scale Study. NeurIPS 2018.\n\n[2] Domingo-Enrich et al., Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control. ICLR 2025.\n\n[3] Uehara et al., Feedback Efficient Online Fine-Tuning of Diffusion Models. ICML 2024.\n\n[4] Mutti et al., Challenging common assumptions in convex reinforcement learning. NeurIPS 2022.\n\n[5] Zhang et al., Variational policy gradient method for reinforcement learning with general utilities. NeurIPS 2020.\n\n[6] Karimi et al., Sinkhorn flow as mirror flow: A continuous-time framework for generalizing the sinkhorn algorithm, AISTATS 2024.\n\n[7] Borkar et al., The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization 2000.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The method makes sense, and is practical.\n\nThe toy example is evaluated with entropy, the target, and illustration, which makes sense and is intuitive. \n\nFor the text-to-image experiment, the CLIP score shows that the p.d.f. still has the same support, and the FID and cross entropy show it deviates from the pretrained model. For the latter, it is implied that the p.d.f. might have higher entropy because if deviates from the original one with low entropy. I understand that entropy is hard to estimate on this space, but would other diversity metrics such as Inception Score or Vendi score be a better surrogate? \n\nAnother interesting experiment would be first fine tune a stable diffusion model on some biased dataset to introduce imbalance, and see if the proposed method can recover it.", "theoretical_claims": "The theoretical claims make sense to me. I skimmed through the proofs in the appendix and I think they are correct.", "experimental_designs_or_analyses": "The experimental designs are sound. See **Methods And Evaluation Criteria.**\n\nBesides, for future work, it would showcase the utility of this model if tested on broader contexts, such as molecule generation problems mentioned in the paper.", "relation_to_broader_scientific_literature": "The paper deals with the data manifold exploration problem, a common problem in real data.\n\nThe paper is related to diffusion models, and can be applied as fine tuning steps to existing models.\n\nAs potential future work, the method might be useful for improving the exploration of drug discovery and other generative-discovery tasks.", "essential_references_not_discussed": "This is not directly comparable, but an interesting line of work that might be related, for the problem of manifold exploration/sample bias reduction: Geometry-Based Data Generation (Lindenbaum, et al., 2018, Neurips), Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds (Sun, et al., 2025, AISTATS). These methods solve the problem from a geometrical perspective.", "other_strengths_and_weaknesses": "## Strengths:\n\n1. The paper is very well written and theoretically solid, with proofs of convergence to the proposed objectives.\n2. The paper did not stop at the ideal case of perfect pretraining and optimization, but admitting those cases are not attained in practice, and further proposes a solution for this, making the method more practical. The theoretical assumptions for the method are justified.\n3. The method is directly usable on pretrained diffusion models and has potential applications in tasks requiring exploration.\n\n## Weaknesses:\n\n1. The evaluation metric for the image generation task can be improved (see **Methods And Evaluation Criteria)**\n2. The method could be showcased on tasks with a stronger motivation for manifold exploration (such as drug discovery). The current example of image generation is not very intuitive.", "other_comments_or_suggestions": "typo in the formula in line 95, right column: $x$ should be $x_t$; also, should be minimizer, not maximizer.", "questions_for_authors": "1. Are there empirical or theoretical results on the scalability of this method, especially on the larger pretrained models?\n2. In assumption 7.3 and thm 7.1 do we assume inifite steps? k, infinite sum? Then is there an estimation of convergence rate?\n3. In terms of connection to RL, is it just notational? It seems that the problem is formulated and solved without relying on properties or algorithms of RL.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNACCXVqiw", "reviewer": "Reviewer_Q9Dh", "summary": "The paper presents a framework to perform optimal exploration of the data manifold defined by a pre-trained diffusion model. This can be useful whenever one wants to sample using diffusion models and explore the full data region within the learned data manifold. The approach that the paper proposes is based on self-guided exploration using the density estimated by the diffusion model itself. Using a connection between the entropy of the density learned by the diffusion model and its score function, they propose a sequential fine-tuning procedure that results in a fine-tuned version that should lead to optimal exploration. The fine-tuning procedure is derived using the connection of diffusion model and reinforcement learning and by using mirror descent theory. They evaluate the proposed method on a toy dataset and an image-to-text task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the interesting questions. In the following, we address several points that can hopefully let the Reviewer appreciate more this paper, and which will be included in a revised version.\n\n**Uncertainty quantification**\nWe agree, our method does not employ *explicit* uncertainty quantification, but arguably relies on a form of *implicit* uncertainty quantification, although not the classic Bayesian one.\n\n**Toy experiment**  \n- The y-range of plot 2.a should be the same as plots 2.b and 2.c.\n- Since Fig. 2.b and 2.c are histogram plots, for a smooth cmap, points with low probability have a similar color to points with zero probability. We will try to overcome this issue by testing non-smooth cmaps.\n- The original density is in Fig. 2.a, namely two uniform distributions.\n- Pre-training was performed by standard denoising score-matching and uniform samples (namely 10K) from the two distributions in Fig. 2.a. \n\n\n**Text-to-Image experiment** \n- The references to Fig. 9 in Sec. 8 should be references to Fig. 3. \n- We agree with the Reviewer that the method presented in this work is relevant for molecular spaces. Nonetheless, applications in scientific discovery go beyond the scope of this paper, which captures mathematically a novel and relevant problem, proposes fundamental algorithmic advances, and provides a novel type of theoretical analysis for diffusion fine-tuning. We believe that exploration in the visual domain is relevant for multiple applications, and particularly amenable for evaluation as it does not require specific background knowledge.\n- Since the method proposed directly maximizes a gradient of entropy, one can show that for $\\alpha \\geq 1$ (in Eq. 9), the entropy value increases monotonically. One could attempt at increasing entropy by reducing guidance strength, but this would result in being closer to the unconditional distribution thus changing the support significantly. On the contrary, our algorithm (provably) explores within the support of the conditional distribution via entropy maximization, which is substantially different.\n\n**Computational complexity and hyperparameter selection**\nUnder exact fine-tuning one iteration is sufficient (see Sec. 5). In practice, we show in Sec. 8 that very few iterations (e.g., $K=3$) can lead to useful fine-tuned models. Therefore, the computational cost aligns with typical fine-tuning, e.g. [3]. The parameter $N$ is significantly problem dependent, and as in the case of Adjoint Matching [3], it requires experimental tuning.\n\n**Question 1**\nIt is common to express the reverse process of a diffusion process via a SDE parametrized by a parameter $\\eta \\in [0,1]$ (e.g., in [1]). The processes obtainable for any valid $\\eta$ induce the same marginal distributions. \n\n**Question 2**\n- The truncation in Prop. 1 is merely a technical aspect to build a theoretical formulation that matches the real sampling process, which always gives finite values. The bounding interval can be any as long as it is finite. Therefore it can fully capture any real scenario.\n- Diffusion notation uses $0$ for the time-step corresponding to data (e.g., [1]), while fine-tuning works denote by $0$ the noise level (see, e.g., [2]). This is formally not fully correct, since to compute closed-form the noise distribution one has to take the limit to infinity. A recent rigorous solution (see, e.g., [1]) is the one used in our paper. Nonetheless, we agree with the Reviewer that the sign flip might create confusion and have already added a drawing that clarifies this in a new version.\n\n**Question 3**\nSec. 4 shows that given exact fine-tuning, the algorithm convergences in $1$ iteration with $\\alpha_1 = 1$. It should be set to lower values to prioritize the surprise term in Eq. 9, and to higher values to prioritize regularization, as discussed in Sec. 4.4. When considering apx. fine-tuning oracles, Theorem 7.1 gives necessary conditions on $\\alpha_k = 1/\\gamma_k$ for the guarantees to hold. In practice, $\\alpha_k$ should be experimentally tuned for the specific application.\n\n\n**Question 4**\nAss. 7.1 could be difficult to verify in some cases. However, it can be relaxed as $\nsupp(p_T^{\\pi_k}) \\subset \\tilde{\\Omega} \\text{ for all } k, $ and $ supp(p_j^{\\pi_k}) = \\tilde{\\Omega} $ for some $ j.\n$ Using the same analysis, we can show that the algorithm can solve the exploration problem on $\\tilde{\\Omega}$. In particular, if $\\tilde{\\Omega}$ approximates the true support, as in Fig. 2, our analysis guarantees exploration of the approximate support.\n\n**References**\n\n[1] Zhao et al., Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning.\n\n[2] Uehara et al., Feedback Efficient Online Fine-Tuning of Diffusion Models. ICML 2024.\n\n[3] Domingo-Enrich et al., Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control. ICLR 2025.", "claims_and_evidence": "The theoretical claims done in the paper seem supported by proofs. Claims regarding the scalability and effectiveness of the proposed method are supported by two (a toy and a text-to-image) experiments. (More on the experiments below).\nI have just one minor comment, and I  might be wrong here, so I ask the authors to correct me if I am wrong. There is a constant repetition that the method proposed in the paper does not rely on explicit uncertainty quantification. I agree it's not explicit, but as the method is framed in terms of density estimation it strongly relies on a sort of uncertainty of the score function it seems. Because the constant repetition seems to give the message that the method wants to be completely detached from the concept of uncertainty in any of the forms, which might be a bit misleading.", "methods_and_evaluation_criteria": "As mentioned above, to show the effectiveness of the proposed framework, they consider two different experiments: a 2D-toy experiment and a text-to-image experiment involving stable diffusion.\n\n- The first toy experiment is a nice and intuitive way to present the problem the paper wants to tackle and what are the results of the proposed method. It would be helpful to have all the plots with the same x-range and maybe a different `cmap` as in (b) some samples are rarely visible. I think it would also be curious to have a plot of the true density and the samples used to train the pre-trained model. Also, there are no details on how the authors trained the diffusion model in the appendix for that specific example.\n- Regarding the second experiment: the authors consider a pre-trained text-to-image diffusion based on based on stable diffusion and perform experiments for two different prompts. They measure results in terms of FID, CLIP, and distance between the two marginal distributions. You mention that you are showing samples for every iteration of the fine-tuning in Figure 9, but it is not clear if that is the case. Also, while the evaluation makes sense, finding a task where the exploration is needed is difficult and I am not fully convinced that the image domain is the best application. Also by comparing Fig. 4 and Fig.5 it is difficult to understand that the fine-tuned model is exploring the manifold more as all the samples look similar, and none of them resemble the one of the pre-trained model. Those results might be similar to getting samples by using two different guidance strengths, but I might be wrong. Maybe having an additional experiment on molecules with specific properties (as the authors mentioned in the paper) would make the paper stronger.", "theoretical_claims": "I checked the theoretical claims the authors have in the paper, but I cannot guarantee that the proofs are completely correct.", "experimental_designs_or_analyses": "I have checked the experimental details provided in the appendix. As mentioned above, there are not so many details on the training set and how the score was trained in the first experiment. It would be also interesting to know how expensive it is to perform the fine-tuning procedure the authors describe in algorithm 1 as it seems to consist of a nested loop. Also how did you usually choose the number of $K$ refinement iterations and the number of $N$ iterations in the inner loop? Additionally, in line 844 you mention 4 trajectories and then you mention a batch size of 8. Are these two related?", "relation_to_broader_scientific_literature": "They present connections to broader scientific literature in the related works section, which is nicely written and very detailed.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "- Line 831 Wince instead of Since\n- There might be a possible mistake in the pseudocode in the appendix I guess. In the adjoint ODE, there is a $k$ subscript that it's not clear where it comes from (if it comes from the outer loop then you might have to include it as input). Also if the initial $t=T$, then the first $\\bar{a}$ considered is $T+1$ which does not exist.", "questions_for_authors": "- Where does the $\\eta \\in [0,1]$ come from in the reverse process in Eq. 2? It's not justified and it does not appear in the forward process. Can the authors comment on this?\n- Why in proposition 1 the noise distribution $p_0$ is a truncated Gaussian? What's the bounding interval in this case? For standard diffusion models, the noise distribution at time T and not 0 approximately converges to a standard Gaussian for variance-preserving processes. Also in the section \"Score matching and generation\" the author is using $p_0$ for the data distribution. Indeed, at some point the author starts describing everything in terms of the time going backward, and when $t=T$ we get that the backward marginal $p_T$ corresponds to the data distribution. Therefore, the notation used in the paper can be improved to make the paper more clear.\n- It seems that the algorithm depends at each iteration on a different ${\\alpha_k}_1^K$ that weighs exploration and exploitation. How should one decide the values of ${\\alpha_k}_1^K$?\n- I might have a naive question: is Assumption 7.1 always satisfied? I understand that if we assume that both models have support on $\\mathbb{R}^d$ this is true, but can it happen that by maybe forcing the model to explore too much, due to approximations and noise, the algorithm will end up sampling from regions of the space where there were no data initially and were the pre-trained model was not able to sample from? Similar to the case of using a too strong guidance strength? Like in Fig. 2 it seems that the tuned model gets samples on the true data support but where it seems that the pre-trained model gets no samples.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WOUSLK8RPX", "reviewer": "Reviewer_kBVm", "summary": "This paper introduces a maximum entropy manifold exploration problem. They proposes a modification to the pretrained diffusion model to maximize an entropy objective function. They also proposed an algorithm to solve this optimization problem. They supported their results with numerical experiments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for reading our work. In the following, we address several fundamental points and questions mentioned within the review that can hopefully let the Reviewer appreciate more the content of this work.\n\n**Motivation of the work**\nIn the following, we aim to make clear the main motivation of this work. Pre-trained generative models can generate plausible objects of a certain data type, e.g. valid images or molecules. Nonetheless, the probability of sampling rare, novel, objects will be very low. This is because the model has been trained on a dataset of existing objects to approximate the data distribution. Therefore rare objects in the data will be rare to sample. In this work, we adapt the generative model so that its induced distribution is not proportional to that of existing data, but is shifted towards lower-probability regions where novel designs can be sampled, while preserving plausibility (i.e., staying within the approximate data manifold). To the best of our knowledge, this is a fundamental (open) problem for discovery via generative models.\n\n**Question 1**\nYes, it is. In fact, if $\\Omega_{pre}$ would be a known set representable in a computer one could simply define a uniform distribution on such a set and then sample from it. But what if $\\Omega_{pre}$ is an unknown and possibly very complex set, e.g., space of valid molecules or images, that cannot be represented in a computer if not implicitly via a generative model? How can we explore such a set? This work tackles exactly this problem, which is both fundamental and non-trivial. We thank the Reviewer for asking this question as we believe it is of central importance, and we will make sure to further clarify this point in a revised version of the work.\n\n**Question 2**\nEquation (7) describes an optimization problem over the set of densities supported over a set $\\Omega_{pre}$ represented only implicitly by a pre-trained diffusion model. This fully captures the manifold exploration problem. On the other hand, Eq. (9) is an unconstrained KL-regularized optimization problem of a reward function obtained by linearizing the entropy, as explained in details within Sec. 4. As explained in Sec. 6, a concave function like entropy can be maximized via a Mirror Descent scheme [1], which relies on a sequence of simpler optimization problems. In this context, Eq. (7) represents the concave optimization problem, while Eq. (9) captures each simpler optimization problem. Crucially, via this viewpoint, we can use scalable methods to solve each smaller problem to solve the complex exploration problem. We will make sure to make this more explicit within an updated version of the work.\n\n**Question 3**\nWithin this work, $s^{pre}$ is the score of a pre-trained diffusion model that one wishes to fine-tune by optimizing a regularized measure of surprise (Eq. (9)).This quantity has to be estimated with respect to the previous model, which at the first iteration of S-MEME corresponds to the pre-trained model with score $s^{pre}$. In practice, as mentioned within Sec. 4.3, the score of the fine-tuned model $s^\\pi$ can be initialized as $s^\\pi = s^{pre}$. After this step, there is no difference in using $s^\\pi$ or $s^{pre}$ to estimate the induced marginal density as these score networks are equal. We will make sure to make this more clear in order to prevent any doubt.\n\n**Question 4**\nReinforcement learning spans several problems, including pure-exploration ones, e.g., [2,3], where the goal is to explore a certain space, typically via maximization of an intrinsic reward, which in the case of this work would correspond to surpise (i.e., the entropy first variation) in Eq. (9) and (10). Pure exploration problems often are relevant on their own. In this case, surprise maximization is clearly relevant for discovery applications where it can make it possible to sample surprising designs, or, as mentioned by Reviewer WBXm, to de-bias a given pre-trained model by inducing a more balanced distribution than the data distribution. Nonetheless, ideas from pure exploration can be used for exploration schemes in exploration-exploitation settings where there is an unknown quantity (e.g., a reward function) to be learned and optimized. In this case, the exploration principle introduced within Eq. (9), which makes it possible to scalably maximize a measure of surprise, could be used as a way to regularize an exploratory scheme for reward-learning in black-box optimization settings, e.g., [4].\n\n**References**\n\n[1] Nemirovski et al., Problem complexity and method efficiency in optimization, 1983.\n\n[2] Hazan et al., Provably efficient maximum entropy exploration. ICML 2019.\n\n[3] Mutny et al., Active exploration via experiment design in markov chains. AISTATS 2023.\n\n[4] Uehara et al., Feedback Efficient Online Fine-Tuning of Diffusion Models. ICML 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Their problem is new, so I feel this point is not applicable here...", "theoretical_claims": "No.", "experimental_designs_or_analyses": "I didn't check the code.", "relation_to_broader_scientific_literature": "It is related to diffusion guidance, an important technique used in the diffusion model commmunity. The paper also mentioned applications in molecular generation.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "The problem is novel, while the motivation is not that clear to me", "other_comments_or_suggestions": "1. Isn't the maximizer of the entropy function just the uniform distribution over $\\Omega^{pre}$? \n2. How related are the two objectives (7) and (9)? \n3. I feel Section 4.3 is not presented very clearly: What is the motivation for using $s^{pre}$ in the actual implementation instead of $s^{\\pi}$? \n4. I am not very familiar with RL, but I feel in general we not only want to do exploration but also want to do exploitation and maximize a reward function. This is also the case for diffusion guidance: practioners want to guide the sample generation towards a direction that maximizes certain reward function. Can the authors discuss extension in this aspect?", "questions_for_authors": "1. Isn't the maximizer of the entropy function just the uniform distribution over $\\Omega^{pre}$? \n2. How related are the two objectives (7) and (9)? \n3. I feel Section 4.3 is not presented very clearly: What is the motivation for using $s^{pre}$ in the actual implementation instead of $s^{\\pi}$? \n4. I am not very familiar with RL, but I feel in general we not only want to do exploration but also want to do exploitation and maximize a reward function. This is also the case for diffusion guidance: practioners want to guide the sample generation towards a direction that maximizes certain reward function. Can the authors discuss extension in this aspect?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eZs41RZ8Pm", "reviewer": "Reviewer_FUKh", "summary": "The authors consider the problem of exploration in planning and decision-making problems.  This problem has many applications including to the exploration-exploitation paradigm in reinfocement learning.  While in most applications the exploration step is performed by sampling from a Gaussian process, the authors consider more general exploration distributions modeled by neural networks.  Specifically, for datasets which lie near lower-dimensional manifolds, the authors aim to sample from entropy-maximizing distributions, which maximize entropy aver the lower-dimension manifold.\n\n The main challenge encountered by the authors is that while generative models are able to generate points, the authors wish to explore “atypical” subsets of this distribution which may have convenient properties for the given application.  In order to compute this entropy-maximizing distribution and sample from it, the authors use the fact that the score function in diffusion models is (in the limit as t —> 0) given by log density of the distribution from which the data was sampled.  This allows the authors to explore the space of distributions in an entropy-maximizing manner.  Next, the authors apply their exploration method to design an algorithm based on mirror descent for the exploring the manifold by sampling from a maximum-entropy distribution on the manifold.\n\n## update after rebuttal\n\nThank you for the helpful clarifications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for appreciating our work and asking interesting questions. In the following, we address several important points mentioned within the review that can hopefully let the Reviewer appreciate more the content of this work.\n\n**Asymptotic convergence guarantees**\nWe thank the Reviewer for raising these concerns and would like to provide a clarification: We can in fact establish convergence guarantees under three different assumptions, ordered by increasing generality:\n- Perfect fine-tuning: If fine-tuning is exact, our algorithm terminates in a single step (see Section 5).  \n- Unbiased noise oracle: When the noise oracle is unbiased (i.e., $ b_k = 0$ in Eq. (16)), standard mirror descent analysis yields a convergence rate of $\\mathcal{O}(k^{-1/2})$ (see [3]).  \n- General bias term: If the noise oracle in Eq. (16) includes a general bias term, then under the arbitrary slow decay assumption in Eq. (18), a polynomial-time guarantee is no longer feasible [3]. In this case, stochastic approximation techniques are required, and the best achievable rate is  $\\tilde{\\mathcal{O}}((\\log\\log k)^{-1})$, which follows from our proof. We chose to present an asymptotic result rather than explicitly stating this rate, as the difference is negligible and in this case it is conventional to present the asymptotic result [3,4].\n\nAmong these, the third setting is the most practical, which is why we focused on it in Section 7. We will incorporate these clarifications into the revision.\n\n\n**Empirical results clarification**\nWithin the Experimental Evaluation section, we evaluate qualitatively (i.e., visually) and quantitatively (i.e., via the metrics within Fig. 2-d and Table 1) the performance of models $\\pi_k$ obtained by fine-tuning a pre-trained model $\\pi^{pre}$ for $k$ iterations of S-MEME. In the case of text-to-image experiments, Fig. 3 shows on the top row a set of images obtained by sampling the pre-trained model $\\pi^{pre}$, which in this case corresponds to Stable Diffusion 1.5 [1] trained on the LAION-5B dataset [2], while the bottom row shows images sampled via $\\pi_3$, which is the diffusion model obtained after $3$ iterations of our algorithm. Qualitatively, one can visually notice that the images within the top row appear mostly gray and similar to each other, while the images from the bottom row show more diversity both among them and compared with the ones above, while preserving semantic meaning. Quantitatively, ideally we would want to estimate the entropy of the marginal density induced by the fine-tuned model, but this is hard in practice, as explained within Sec. 8.2. As a consequence, within Table 1 we show several proxy metrics to give numerical insights about the performances of the fine-tuned models. Within this table, the label 'S-MEME 1' refers to the model obtained after one single iteration of S-MEME, 'S-MEME 2' after two iterations, and 'S-MEME 3' after three iterations. Crucially, we show that the FID and cross-entropy scores, which aim to capture the degree of diversity of the marginal density induced by a model $\\pi_k$ from the pre-trained model, increases over the iterations of S-MEME, meanwhile the CLIP score, which assesses the naturalness, or image quality, of the generated samples, is kept high across increasing iterations of S-MEME. Clearly, the trade-off between surprise maximization (i.e. diversity from pre-trained model) and naturalness due to regularization with the pre-trained model, can be chosen arbitrarily by tuning the $\\{\\alpha_k\\}$ parameters of the algorithm, which manage this trade-off as shown In Eq. (9). The experimental results in Sec. 8 aim to evaluate a specific parameter choice to show practical relevance of the proposed scheme. \n\nWe thank the Reviewer for these question sand hope that the explanations given can help the Reviewer better appreciate our work. We will update a revised version of the paper with better clarification of these aspects.\n\n**References**\n\n[1] Rombach et al., High-resolution image synthesis with latent diffusion models. CVPR 2022.\n\n[2] Schuhmann et al., An open large-scale dataset for training next generation image-text models. NeurIPS 2022.\n\n[3] Karimi et al., Sinkhorn flow as mirror flow: A continuous-time framework for generalizing the sinkhorn algorithm. AISTATS 2024.\n\n[4] Borkar et al., The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization 2000.", "claims_and_evidence": "The authors provide theoretical guarantees which show (i) that their entropy maximizing exploration step is optimal and (ii) that their algorithm (Algorithm 1) converges to the maximum entropy density after infinitely many iterations of their algorithm. Visually, the designs from their method appear (subjectively) to be more unusual/less conventional than the baseline model, but possibly at the cost of being less realistic and of lower quality.  Empirically, the authors implement their algorithm on an architecture dataset where the goal is to explore atypical architectural designs.", "methods_and_evaluation_criteria": "The theoretical results prove the optimality of the authors exploration method (Theorem 5.2), and the \"asymptotic correctness\" of their algorithm (Theorem 7.1). However, they do not provide any non-asymptotic guarantees for their algorithm.\n\nFor empirical criteria, the architecture dataset used by the authors provides a good illustration of the exploration problem, and allows the authors to compare their algorithm to baseline methods on a compelling application.   Visually, the designs from their method appear (subjectively) to be more unusual/less conventional than images sampled without any iterations of their mirror descent algorithm distribution (which serves as a baseline model), but possibly at the cost of being less realistic and of lower quality.  Thus, it would be good to discuss the apparent tradeoff in visual sample quality.", "theoretical_claims": "I did not carefully check the proofs in the appendix.", "experimental_designs_or_analyses": "Please see \"Methods And Evaluation Criteria\" question above.", "relation_to_broader_scientific_literature": "Empirically, the authors compare to the stable diffusion model as a baseline.  However, it may be good to include additional baselines to compare to, if this is feasible.  Also, it is a bit unclear from the captions in Figure 3 and Table 1 which models are the authors' and which are the baselines (I believe that $\\pi_pre$ is the authors and $\\pi_3$ is a baseline model, but this could be made a bit more clear)", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The main strengths of the paper are in proposing a method for entropy-maximizing exploration, and for providing an algorithm with one-step optimality and (asymptotic) convergence guarantees for this problem.  The main weaknesses are (i) that the convergence guarantees are asymptotic (there is no guarantee on how fast their method converges) and (ii) the empirical results could be made a bit more clear.  Specifically I found it a bit unclear what are the baseline model(s) being compared to empirically, and what the tradeoff is between image quality and \"creativity\" or \"entropy\" when the authors.  If I understand correctly, from Figure 3 there appears to be a tradeoff, so it would be good for the authors to discuss this in more detail.", "other_comments_or_suggestions": "Please see \"Other Strengths And Weaknesses\" question above.", "questions_for_authors": "Can the authors clarify in the empirical section which baseline model(s) they are comparing to?  I believe the baseline model is the stable diffusion model, but this is a bit unclear to me.    Also, if I understand correctly, from Figure 3 there appears to be a tradeoff, so it would be good for the authors to discuss this in more detail.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Given a pre-trained diffusion model, this paper proposes a method to sample from the maximum entropy distribution supported on the support of the diffusion model. Reviewers had several criticisms of the paper which were mostly addressed during the rebuttal, and I thus recommend acceptance.

To the authors: If the paper is accepted, I ask you to please follow the suggestion of reviewer WBXm and include the Vendi score as an evaluation metric. In your rebuttal you mentioned this was difficult because choosing the kernel is not trivial, and that it is unclear if this score correlates with entropy. As for the kernel choice, you can just use the default one in the public implementation. As for the potential lack of correlation with entropy, although you are technically correct, it would be quite surprising to observe it in practice and this would warrant additional discussion.
Decision: Accept (poster)