The Superposition of Diffusion Models Using the Itô Density Estimator
Authors: Marta Skreta
Lazar Atanackovic
Joey Bose
Alexander Tong
Kirill Neklyudov
Venue: ICLR 2025 Spotlight (2025)
Keywords: generative modelling
protein generation
image generation
diffusion models
Primary area: generative models
TL;DR: The principled way to combine the outputs of several diffusion models

Abstract: The Cambrian explosion of easily accessible pre-trained diffusion models suggests a demand for methods that combine multiple different pre-trained diffusion models without incurring the significant computational burden of re-training a larger combined model. In this paper, we cast the problem of combining multiple pre-trained diffusion models at the generation stage under a novel proposed framework termed superposition. Theoretically, we derive superposition from rigorous first principles stemming from the celebrated continuity equation and design two novel algorithms tailor-made for combining diffusion models in SuperDiff. SuperDiff leverages a new scalable Itô density estimator for the log likelihood of the diffusion SDE which incurs *no additional overhead* compared to the well-known Hutchinson's estimator needed for divergence calculations. We demonstrate that SuperDiff is scalable to large pre-trained diffusion models as superposition is performed *solely through composition during inference*, and also enjoys painless implementation as it combines different pre-trained vector fields through an automated re-weighting scheme. Notably, we show that SuperDiff is efficient during inference time, and mimics traditional composition operators such as the logical OR and the logical AND.  We empirically demonstrate the utility of using SuperDiff for generating more diverse images on CIFAR-10, more faithful prompt conditioned image editing using Stable Diffusion, as well as improved conditional molecule generation and unconditional *de novo* structure design of proteins. https://github.com/necludov/super-diffusion

Published as a conference paper at ICLR 2025
THE SUPERPOSITION OF DIFFUSION MODELS USING THE ITÔ DENSITY ESTIMATOR
Marta Skreta 1.22
Lazar Atanackovie*.12 1.22
Avishek Joey Bose
Alexander Tong*h 4,5
Kirill Neklyudov 4,5,f
Unniversity of Toronto
Vector Institute Uniiversity of Oxford
Mila- Quebec Al Institute
Université de Montréal
ABSTRACT
The Cambrian explosion of easily accessible pre-trained diffusion models suggests demand for methods that combine multiple different pre-trained diffusion models without incurring the significant computational burden of re-training a larger combined model. In this paper, we cast the problem of combining multiple pre-trained diffusion models at the generation stage under a novel proposed framework termed superposition. Theoretically, we derive superposition from rigorous first principles stemming from the celebrated continuity equation and design two novel algorithms tailor-made for combining diffusion models in DIFER SUPER leverages a new scalable Itô density estimator for the log likelihood of the diffusion SDE which incurs no additional overhead compared to the well-known Hutchinson's 's estimator needed for divergence calculations. We demonstrate that SUFERDIFF is scalable to large pre-trained diffusion models as superposition is performed solely through composition during inference, and also enjoys painless implementation through an automated re-weighting scheme. Notably, we show that SUPER DIFF is efficient during inference time, and mimics traditional composition operators such as the logical OR and the logical AND. We empirically demonstrate the utility of using SUFERDIFF for generating more diverse images on CIFAR-10, more faithful prompt conditioned image editing using Stable Diffusion, as well as improved conditional molecule generation and unconditional de novo structure design of proteins. ht tps:: :ps::/a /ggttttthb...g com/ mecludov/suped di siffusion
combines different pre-trained vector fields
INTRODUCTION
The design and application of generative models at scale are arguably one of the fastest-growing cases of machine learning, with generational leaps in performance that often exceed expert expectations (Steinhardt, 2022). A few of the central facilitators of this rapid progress are the availability of high-quality training data and large computing hardware (Kaplan et al., 2020); which in tandem provide a tried and trusted recipe to scale generative models in a variety of data modalities such as video generation (Brooks et al., 2024), natural language understanding (OpenAI, 2023; Achiam et al., 2023; Dong et al., 2022), and other challenging domains like mathematical reasoning (Trinh al., 2024), or code assistance (Bubeck et al., 2023). As result, it is not surprising that a driving force behind current generative modeling research is centered around developing open-source tooling (Dao al., 2022; Kwon et al., 2023) enable further scaling and understanding emergent behavior of such models (Schaeffer et al., 2023), including probing current limitations (Dziri et al., 2024).
Indeed, the rapid escalation of generative model development has also induced a democratizing effect, given the easy access to large pre-trained in the current AI climate (Stability AI, 2023; Midjourney, 2023; Ramesh et al., 2021). Furthermore, with the rise of open-source models, is now casier than ever to host and deploy fine-tuned models. However, the current pace of progress also makes it infeasible to easily scale further models without confronting practical challenges. For instance, for continuous domains such as natural images current pre-trained diffusion models already exhaust all public data, with a growing proportion of the web already populated with synthetic data (Schuhmann et al., 2022). Compounding these challenges is the tremendous cost of pre-training these large
Authors contributed equally. 1 Cooresponneece neal udov@gmai coom

---

Published as a conference paper at ICLR 2025
DOG and GAT
F FLAMINGO and CANDY CANF
4 SECYCLE MHEEL and SPIDER WEB
MAPFLE CONE VOLCAND
A DONUT and
FIREMORKS and DANDELION
SUPERDIFF (AND)
Avg. ofuutuus
Joint prompting
Figure I: Concept interpolations via different methods: SUPER (top row), the averaging of outputs with different prompts (middle row), and joint prompting with standard Stable Diffusion (SD) (bottom row) for six different prompt combinations. Here we use SUPERDIFF with the AND operation (sampling equal densities).
diffusion models, which at present makes computationally unattractive for individuals to build large pre-trained models on different datasets without re- re-training larger combined model.
compelling alternative to training ever larger models is to consider the efficacy of maximizing the utility of existing pre-trained models. In particular, it is interesting to consider the compositional benefits of combining pre-trained models at the generation stage in place of training single mono- lithic model (Du & Kaelbling, 2024). For diffusion models, which are the current de facto modeling paradigm over continuous domains, compositional generation can be framed as modifying the in- ference mechanism through either guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022) or applying complex MCMC correction schemes (Du et al., 2023). Despite advancements, methods that utilize guidance lack firm theoretical underpinning (Bradley & Nakkiran, 2024) and MCMC tech- niques are prone to scaling issues, and more importantly, remain unproven at combining existing large pre-trained diffusion models. This motivates the following timely research question: Can we combine pre-trained diffusion models solely at inference in a theoretically sound and efficient manner?
Present work. In this paper, we cast the problem of combining and composing multiple pre-trained diffusion models under a novel joint inference framework we term superposition. Intuitively we develop our framework by starting from the well-known principle of superposition in physical systems which summarizes the net response multiple inputs in linear system as the sum of individual inputs. Applying the superposition principle, even in the simplest case of a mixture model, requires constructing the resultant superimposed vector field; which can be built analytically by reweighting using the marginal density of generated samples along each diffusion model's inference trajectory.
We introduce SUPERDIFF for scalable superposition by proposing novel estimator of the density of generated samples along an SDE trajectory, which we expect to be of independent interest beyond superposition. particular, our Itô density estimator in Thm. is general as it is not restricted to the vector field under which the trajectories are generated, it requires no additional computation during inference, and, when the ground true scores of the marginals are available, it is exact. This is in stark contrast to all prior density estimation approaches that involve computing expensive and high variance divergence estimates of a drift vector field associated with the probability flow ODE, Armed with our new Itô density estimator we propose to combine pre-trained diffusion models by guiding the joint inference process through fine-graie fine-grained control of the relative superposition weights of the model outputs. More precisely. in Sec. 3.2, we propose algorithms for two specific instantiations, illustrated in (Fig. 1) of the superposition principle: (i) generating samples from a mixture of densities (an OR operation over models) or (ii) generating samples that are equally likely under all densities (an AND operation over models).
We test the applicability of our approach SUPERDIFF using the two proposed superimposition strategies for image and protein generation tasks. For images, we first demonstrate the ability to combine the outputs of two models trained on disjoint datasets such that they yield better performance
2

---

Published as a conference paper at ICLR 2025
than the model trained on both of the datasets (see Sec. 4.1). In addition, demonstrate the ability to interpolate between densities which correspond to concept interpolation in the image space (see Sec. 4.2). For proteins, demonstrate that combining two different generative models leads to improvements in designability and novelty generation (see Sec. 4.3). Across all our experimental settings, find that combining pre-trained diffusion models using SUPERDIFF leads to higher fidelity generated samples that better match task specification in text-conditioned image generation and also produce more diverse generated protein structures than comparable composition strategies.
2 PRELIMARIES
Generative models learn to approximate a target data distribution Pdata E P(Rd)) defined over Rd using a parametric model qe with learnable parameters 0. In the conventional problem definition, the data distribution is realized as an empirical distribution that is provided as training set of samples D = {er y Whilst there are multiple generative model families to choose from, we restrict our attention diffusion models (Song et al., 2020; Ho al., 2020) which are arguably the most popular modeling family driving current application domains. We next review the basics of the continuous time formulation of diffusion models before casting them within our superposition framework.
2.1 CONTINUOUS TIME DIFFUSION MODELS
diffusion model can be cast as the solution to the Stochastic Differential Equation (Øksendal, 2003),
dr.- = ft (x)) + g,dW,, To (x()))
(1)
In the Itô SDE literature, the function ft TRd + Rd is known as the drift coefficient while gr : T - TR is real-valued function called the diffusion coefficient and We is the standard Wiener process. The subscript index t E [0, 1] indicates the time-valued nature of the stochastic process. Specifically, we fix the starting time minal time & = 1 to an easy to sample prior such as standard Normal distribution Pnoise := q (z)) = N( J0, /). As such the diffusion SDE, also called the forward process, can be seen as progressively corrupting the data distribution and ultimately hitting a terminal distribution devoid of any structure.
0 to correspond to the data distribution Pdata := (xro) and set 1 as the ter-
To generate samples from the marginal density qe (2)) induced by the diffusion SDE in equation I we leverage the reverse-time SDE with the same marginal density as demonstrated below.
Proposition 1. [Reverse-time SDES/ODE) Marginal densities qe(x) induced by Eq. (1) corre- spond to the densities induced by the following SDE that goes back in time 1 t) with the corresponding initial condition
dx, = (-Si(or))+
gi (
+
+x)+)loog((z))) dr + v WW, I==0 ~ (zo),
(2)
where W, is the standard Wiener process in time T, and & is any positive schedule.
See proof in App. A.1. The reverse SDE flows backward in time T = - 1 and is linked to the diffusion SDE through the score V, log (z), and dW, is another Weiner process. As a result, parametric model V log (2; 0) may directly learn to approximate this score function for every point in time and then draw samples by simulating the reverse SDE in equation 2 by plugging back in the learned score. Notably, for & = O. the SDE becomes ODE which defines smooth change of measure corresponding to Paoise into the measure corresponding to Pdata-
In practice, for generative modelling, the forward SDE from Eq. (1) is chosen to be so simple that can be integrated in time analytically without simulating the SDE itself. This is equivalent to choosing the noising schedule first, and then deriving the SDE that corresponds to this schedule. Namely, for every training sample jt', we can define the density of the corrupted j as normal density with the mean scaled according to aa and the variance of, then the density of the entire corrupted dataset simply a mixture over all training samples μ¹, i.e.
N
(<)=== = -Nxx anpi of)) (x())=
N Σ dd(x).
(3)
Clearly, choosing at, o. such that ag = 1,00 = and ai = 0, 01 = 1, we guarantee Pdata :- ()()
and Pnoise g (x1) = Mxx 10, /). This perturbation of the data distribution using Gaussian
kernel offers specific forms for the drift ft, and diffusion coefficient 9: as described next.
3

---

Published as a conference paper at ICLR 2025
Proposition 2. [Orntein-Uhlenbeed SDEJ The time-dependent densities in Eq. (3) correspond the marginal densities of the following SDE, with the corresponding initial condition
dx, i log a4
log 0t al at
Tt dt +
?
dW r 2o (20
(4)
0L
f((2))
See proof in App. A.2. We highlight the simplicity of the drift term, a linear scaling, that allows us to simulate efficiently the reverse SDE and is crucial for the proposed density estimators in Sec. 3.1. Altogether, the derivations of this section allow us to go from the noise schedules of samples in Eq. (3) used during the training of given diffusion model to the corresponding forward SDE in Prop. 2, and finally to the reverse SDEs or ODE used in Prop. I.
2.2 SUPERPOSITION OF ODES AND SDES
In this section, we introduce the superposition of multiple time-dependent densities that correspond to different stochastic processes. A suggestive view of these densities as processes corresponding to different training data (e.g. different datasets), different conditions (e.g. different text prompts), or simply differently trained diffusion models. Namely, we consider N forward noising process that possibly start from different initial distributions {z)=o(>){{² (e.g. different datasets).
{ N
Assume that we know the individual vector fields vx(x) that define the change of corresponding densities (2) via the state-space ODEs and continuity equations,
dxp = -
=
ōt <i()) = = (x))) Vi [M].
(5)
dt The superposition of the noising processes (x)) is the mixture of corresponding densities:
N
mix (I) Σ of (z),
Ew'=1 wn 200
where w/ is mixing coefficient. Note that superimposed of (x:) also satisfies the continuity equation for the superposition of the vector fields (1r) demonstrated in the following proposition.
qiii
Proposition 3. [Superposition of ODEs (Liu, 2022)] The mixture density in Eq. follows the continuity equation with the superposed vector fields from Eq. (5), i.e.
2 Ot 4t (x) (=))=(VV>)
(V., mix (=)ua())))
vx(x)
Σ ())))))))))) (7)
Σ/ w
We reproduce the proof for Prop. in the context of superposition in App. A.3. The superposition principle is the core principle that allows for efficient simulation-free learning of the flow-based models (Liu al., 2022b; Lipman al., 2022; Albergo et al., 2023) and diffusion models (Song et al., 2020). We discuss how these frameworks are derived from the superposition principle in App. B.
The superposition principle straightforwardly extends to the marginal densities of SDEs. That is, consider marginals densities of (I) generated by the following SDEs
dx- n² (x-))dr + grdW,, r: Zr=0 = (==)
where one has note the same diffusion coefficient for all the SDEs. Then the mixture of densities from Eq. (6) can be simulated by the SDE from the following proposition.
Proposition 4. (Superposition of SDEs) The mixture (x) = , marginals { induced by SDEs fromfrom Eq. (8) corresponds to the following SDE
(x))())) of density
dr, = ur (x+) + g-dW,, ux(x) = S Σ (x)(()))(())...
(9)
See App. A.4 for the proof. Both Prop. and Prop. 4 can be easily extended to the families of
densities parameterized with continuous variable (see Theorem in Peluchetti (2023)).
4

---

Published as a conference paper at ICLR 2025
Train Data
Train Data
Generated samples
(a) Training datasets
(b) Averaging the vector fields
(c) Sampling the mixture of densities (ours)
(d) Sampling equal densities (ours)
Figure 2: An intuitive illustration of using model superposition for improving inference performance. We show an example of two disjoint datasets and train a model for each set. Each individual model learns generate samples only from their respective datasets. Using model superposition enables sampling from both densities.
3 SUPERPOSITION OF DIFFERENT MODELS
We now introduce our method for combining pre-trained diffusion models using the principle of superposition. The result of this is novel inference time algorithm SUPERDIFF which can be easily applied without further fine- tuning or post-pre pooteesssssinng of any of the pre-trained diffusion weights. Our proposed approach SUPERDIFF can be instantiated in two distinct ways that allow for the composition diffusion models that can be informally interpreted as logical composition operators the logical AND and the logical OR. More precisely, given two pre-trained diffusion models that trained datasets A and B inference using SUPER DIFF can be done by either sampling from the mixture of the two learned densities, i.e. logical AND Fig. 2c, or sampling from the equal density locus (logical OR Fig. 2d). In such a manner, superposition using AND leads to generated samples that are equally likely under both pre-trained diffusion models while superposition using OR creates samples that are pref- erentially generated by either pre-trained model-ane and thus mimics the empirical distributions A or B.
Method overview. SUPERDIFF is applicable in settings where the modeler has access to M pre-trained diffusion models, along with its learned score function V, log (z)) Each of the pre-trained models follows a marginal density (x) and as a result must admit a corresponding vector field (a:) that satisfies the continuity equation in Equation 5. The key idea of our approach is an adaptive re-weighting scheme of the pre-trained model's vector fields that relies on the likelihood of a sample under different models. A naive approach to estimating each marginal density during generation immediately presents several technical challenges as requires the estimation of the divergence of superimposed vector fields. In particular, these challenges can be stated as follows:
(C1) The marginal superpositioned vector field differs from the vector fields of either of the models. (C2) The divergence operation requires backpropagation through the network and is computationally expensive even with Hutchinson's 's trace estimator (Hutchinson, 1989).
Our proposed approach SUPERDIFF overcomes these computational challenges by introducing novel density estimator in Sec. 3.1. Crucially, this new estimator does not require divergence estimation and enjoys having the same variance as the computationally expensive Hutchinson's trace estimator, making it favorable choice when generating using large pre-trained diffusion models. In section Sec. 3.2 we exploit this new density estimator to formally present our algorithm SUPERDIFE and derive connections to the composition operators that intuitively resemble logical AND and OR.
3.1 EVALUATING THE DENSITIES ON THE FLY
In this section, we introduce a novel method for evaluating the marginal density of a diffusion model during the inference process. The conventional way to evaluate the density uses the continuity equation and solves the same ODE that is used for generating samples. This, however, is not easily possible in the case of our superposition of vector fields framework as outlined in Prop. 3. To solve this, we present the following proposition that disentangles the vector field generating the sample (u+(x) in the proposition) and the vector fields corresponding to different generative models (xr).
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: SIF pseudocode (for OR and AND operations)
Input: :M pre-trained score models v log (1),,, the parameters of the schedule at dt, stepsize dr 0, temperature parameter T, bias parameter and initial noise NO, I).
for == 0 do
f ==1 T, I),
xt softmax(T log (z. + // according ng Prop,
solve Linear Equations // AND according Prop-
ue(xx)) + Eisl n. Vog
de, + (-f1-r (x.) + gi (e(,)) 1) dw, // using Prop.
Artdr Ir du,
dlog - (x+) (drr, V Vloggi- (V, (1-((
+ + (1-T((e)) - ot Vogg V loggg--((z) )dr 11 using Thm.
return 27
Proposition 5. [Smooth density estimator] For the integral curve s=() solving dx:/dt = (1xs), à
and the density (((x(t) satisfying the continuity equation ät ()(x)) - (zz))(()(((x)) the log- log-density along the curve changes according the following ODE
d
log (x(()) - u/())))))))>->>>. dt
(V> log (x), ef(() ux)))
(10)
See proof in App. C. We use this proposition for our experiments conducted in Sec. 4.1. However, outlined previously, evaluating the marginal density via the continuity equation is restricted to small-scale models due to the computational challenges associated with efficiently estimating the divergence of the associated vector field. As result, common line of attack assumes constructing a stochastic unbiased estimator that trades for increased speed by introducing a bit variance into the divergence estimate. This approach is known Hutchinson's estimator and requires computing a Jacobian- vector product at every step of the inference.
Instead, we propose a new way to estimate density that allows for efficient computation while inte- grating the backward SDE from Prop. 1 with a specific choice of the diffusion coefficient.
[ltô density estimator] Consider time-dependent density q: (x) induced by the marginals of the following SDE
Theorem 1.
dr, = fe(x,)dt + g,dW,, It ~ [)(x), E [0, 1],
(11)
where dW, is the Wiener process. For the reverse-time (T = t) SDEs with any vector field u, and the same diffusion coefficient 9t, i.e.
dx, = u, (x., )dT + dW r: T E [0, 1], the change of the log- log-density log 9r (Ix) follows the following SDE
(12)
dloog (x+) = (dx,, V (log))+((-))> (VV, ((. (1-
(13)
+ ()))
2 -i
V log ạ1 -1- Vloglog----
(x+,, V Vlog ạ1 - )0a)))
2
We provide the proof in App. Notably, the SDE Eq. (13) that keeps track of the change log-density includes only the divergence of the forward SDE drift (V, However, in practice, when using the
SDE, this divergence simply a constant due to a linear drift scaling. In App. D, we derive the same estimator but in discrete time using the detailed balance condition.
3.2 SUFERDIFF: SUPERPOSING PRE-TRAINED DIFFUSION MODELS
Mixture of densities (logical OR). For mixture of the densities, superposition of the models follows directly from the propositions in Sec. 2.2. That is, for every (r), we assume that it can be generated from SDES or ODE from Prop. 1 and we assume the knowledge of scores V log gf()..

---

Published as a conference paper at ICLR 2025
Then, for the ODE simulation, according Prop. 3, can sample from the mixture of densities
quix (a) 1/ME Di=i (x) using the following vector field:
(x) Ej (x) Vlog (z))
uy 0x(x) = -
(14)
starting from samples 2,=0 91 (Iu). The densities are estimated along the trajectory using Prop. 5. We highlight this analogously applies for simulation using the SDE dz, = ur (z, (x,)dT + 91--dW, Namely, according to Prop. 4, we use the following vector field:
M
le (x))
(x) = f1--(()) + 9
Σ 1=1
Vlog (o)))i-(())
(15)
starting from the samples 2,=0 ~ 9i (IO). The densities are estimated along the trajectory using Thm. 1. We provide the pseudocode in Algorithm 1.
Sampling equal densities (logical AND). To produce the sample that have equal densities under different models rely
formula for the density update (see Thm. 1) to find the optimal weights for the vector fields. Indeed, for M diffusion models, we have system of M equations: the equal change of density for every model and the normalization constraint for model weights, which is a linear system w.r.t. vector field weights as we show in the following proposition.
Proposition 6. [Density control] For the SDE
M
dx, Σ
Kyju (x+ )dT + #1 -1--dW
(16)
j=1
where i. are the weights of different models and Ej Kig = 1, one can find x that satisfies
d doo (x-) d -/_ (x+), i,j E [M], by solving a system of M + linear equations w.r.l. K.
(17)
We provide the proof and the formulas for the system of linear equations in App. C.I. This proposition allows us to find k that controls the densities to stay the same for all the models as described in Algorithm 1. This approach can also be straightforwardly extended to the case of diffusion models for satisfying different prescribed density ratios, i.e.
d (#t) d'log (x+) +e.
(18)
EXPERIMENTS
4.1 JOINING MODELS WITH DISJOINT TRAINING DATA
We validate the proposed Algorithm for the generation of the mixture of the distributions (OR setting). We split CIFAR- 10 into two disjoint training sets of equal size (first labels and last 5 labels), train diffusion models each part, and generate the samples jointly using both models. Namely, the stochastic inference is the OR implementation is the integration of the ODEs (see Prop. 1) and the estimation of the log-density according to Prop. 5. For the choice of hyperparameters, architecture, and data preprocessing, we follow (Song et al., 2020).
Algorithm 1, whereas the deterministic setting
In Table 1, we demonstrate that the performance performance of the individual models and performs even better than the model trained on the union of both parts of the dataset. For comparison, we evaluate conventional image quality metries: Fréchet Inception Distance (FID), Inception Score (IS), and Feature Likelihood Divergence (FLD) (Jiralerspong et al., 2023), which takes into account the generalization abilities of the model.
SUPERDIFF drastically outperforms the
4.2 CONCEPT INTERPOLATION WITH SUPERDIFF (AND) AND STABLE DIFFUSION
Next, evaluate the ability of SUPERDIFF to interpolate different concepts using Stable Diffusion (SD) v1-4 (logical AND). In this setup, we generate an image from SD by conditioning it on prompt using classifier-free guidance. We define concepts - e.g. "a sunflower" and "a lemen". We can thus consider each prompt-conditioned model separate diffusion model and apply Algorithm to generate images sampled with
models using separate prompts that represent
7

---

Published as a conference paper at ICLR 2025
Table I: Unconditional image generation performance for CIFAR-10 with models trained disjoint
partitions of the training data (labeled and B). We compare SUFERDIFF (OR) with the respective models with
the model that is trained on the full dataset (model and random choice between two models (model oR B):
ODE inference
SDE inference
FID (J)
IS (1)
FLD (4)
FID (L)
1S (1)
FLD (J)
modelA modelg
14.00
8.73
15.50 = 0.17
15.33
7.98
15.47 + 0.18
13.09
7.89
18.90 + 0.18
13.50
7.98
18.54 + 0.23
model.Au
6.00
8.95
+
3.50
9.14
7.51
+ 0.11
model oz #
4.28
9.14
5.96 5.161
3.99
9.36
5.29 0.144
SUPERDIFF (OR)
4.41
9.12
6.10
4.00
9.36
5.33 0.05
SUPERDIFF 1-100 (OR)
4.11
9.21
5.89 0.11
4.00
9.48
5.20
+ 0.11
Table Quantativate evaluation of SD-generated images. We compare SUFERDIFF (AND), joint prompting, and averaging of outputs two concept prompts. We report the average minimum CLIP, ImageReward, and TIFA scores of each prompt pair, which gives a measure of how well both concepts are represented.
Min. CLIF (f)
Min. ImageReward (*)
Min. TIFA (f)
Joint prompting
23.87
-1.62
27.58
Average scores (Liu al., 2022a)
24.23
-1.57
32.48
SUPERDIFE (AND)
24.79
-1.39
39.92
proportionally equal likelihood with respect to both prompt-conditioned models. We generate 20 images for 20 different concept pairs (tasks) for our model and baselines (c.f. App. J).
Baselines for concept interpolation. We consider two approaches for composing images as baselines. The first is simple averaging of SD outputs based on the approach in Liu et al. (2022a); we set K 0.5. The second guides SD generation with a single joint prompt. The prompts are constructed by joining two input concepts with the linking term "that looks like" - c.g. "a sunflower that looks like lemon". For fairness, we also flip the order of the prompt and keep the image with the higher score for all metrics listed below (Luo et al., 2024).
SUPERDIFF qualitatively generates images with better concept interpolations. We plot sample generated images for SUPERDIFF (AND) in Fig. 1. For the complete set of generated images, see App. J (Figs. A5-A24). We observe that SUFERDIFF (AND) can interpolate concepts while also maintaining high perceptual quality. In contrast, the averaging baseline either fails to interpolate concepts from both prompts fully or yields images with lower perceptual quality. We observe that SD using a single prompt struggles to interpolate both concepts.
SUPERDIFF outperforms baselines on three image evaluation metrics. To quantitatively evaluate the generated images, we consider three metrics: CLIP Score (Radford al., 2021), ImageReward (Xu et al., 2024), and TIFA (Hu et al., 2023). CLIP Score measures the cosine similarity between image embedding and text prompt embedding. ImageReward evaluates generated images by assigning a score that reflects how closely they align with human preferences. TIFA generates several question- answer pairs using Large Language Model for given prompt and assigns a score by answering the questions based on the image with a visual DIFF and all baselines Table 2. For the logical AND setting, we evaluate the image against each concept prompt separately (i.e., "a sunflower" as one prompt and "a lemon" as the other) and take the minimum score for each metric, which measures how well both concepts are represented. We find that SUPER (AND) obtains the highest scores across all metrics, indicating that our method can better represent both concepts in the images, while the baseline methods typically only represent one concept or (especially in the case of averaging outputs), generate compositions with lower fidelity.
model. We report these metrics for Su-
4.3 PROTEIN GENERATION
Next, we apply our method in the setting of unconditional de novo protein generation. Protein generation has critical implications in drug discovery (Abramson al., 2024). A good understanding of the protein landscape is important to rationally find novel proteins. We evaluate proteins generated by the superposition of two existing protein diffusion models, Proteus (Wang et al., 2024a) and Fifm (Yim et al., 2023b). We report the results of our best model in Table 3, as well as results for each model individually and simply averaging them.
In Table 2, we report results for inference only using SDFs. For the ODE setting, evaluating densities takes about 1082 + 7 seconds per image with SD, whereas it only takes 209 = with our density estimator in the SDE setting (over 5-fold decrease!), while also requiring almost 30% less memory. DIFF (AND) SD v1-4: https: //huggi face . co/ superdi ff/ Ff/superdi ff-sd-vi
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation of SUPERDIFF and baseline models, Proteus & FrameDiff, for unconditional protein gen- eration. We show results for three categories of metrics: designability,, novelty, and diversity. We also include a baseline simple averaging scores (x = 0.5). We use the parameter (see Eq. (18)) to control and bias model superposition towards Proteus, i.e. (E > 0) or FrameDiff (t < 0). We use temperature values T = for all vari- ants of SIPERDIFF. For each type of model composition (averaging SIFERDIFF) we mark each metric with (f) if better than both Proteus and FrameDiff their own, and with a (+) if better than either one of them.
Disigubiity
Nordly
Diveralt
SCRMSD
[1)
(2)
ff)
Frnndii
0.392 0.3
1.315 a 6.25
6.152
0:216 = 0.61
0.570 + 0.02
0.125 - 0.01
6:317 4 0.02
0.528 0.05
Proteus Avange of somes (Lis at, 2022g)
(1928 IU22
100l4 0.07
0,000 0.61
0.530
000
0
0.3.2 0.00
0.377 0002
0.740=0..66
1.500 1 2...4*
0:30 0.05
0026 0.0.
0.511 0.0.1
0.130 0.0
0.310==0.01
6:253 e 0.01
0.752 0.68"
1.500 1-0.4
0.27 0.5"
0:288 0.61
0.517 0.01"
0.14 0.0."
0.500 = 0.02"
6.288 : 0.12"
S
0.976 2: 0.011
0.929
: 0.055
0.396
:: 0.08"
002: a: 0.0.
0.528 à 0.01"
0.127 0...0
0.507 = 0.002
0,246 a 0.00
-000"
2,079
0.366
0.040 0.01!
0.511 a 0.01"
01111 0.0
0.305 00.00
0.656 6
Metries for evaluations. We consider des- ignability, novelty, and diversity metrics for evaluation of unconditional generation of protein structures. Protein designability refers to the in-silico agreement between generated structures and refolded structures as computed using a purpose-buie built folding model e.g. ESMFold (Lin et al.. 2022), which is positively correlated with the synthesizability of the protein in a wet-lab setting. Generally, if the root-mean-square distance between the generated and refolded proteins (scRMSD) is less than 2A, is considered to be designable. We compute several novelty metrics based on the similarity of generated proteins those from the set of known proteins (the training set), which are called SCTM scores; the lower the score, the less similar the generated protein is to the training data. Lastly, we use diversity to assess the degree of heterogeneity present in the set of generated proteins. This is done by clustering the generated proteins in terms of sequence overlap, and measuring the fraction of proteins with challenging challenggin-----geeraaee secondary structures such as sheets (Bose et al., 2024). We provide details of all metries in this section in App. H.2. All results are averages over 50 generated proteins at lengths {100, 150, 200, 250, 300} for 500 timesteps.
SUPERDaFF (OR)
FrameDill
Proteus
Figure 3: UMAP visualization of protein structures showing cluster archetypes where structure diversity maintained with SUPERDIFF ==0 (OR).
SUPERDIFF improves structure generation. By combining two protein diffusion models, Su- PERDIFF is able outperform both of them. This is somewhat surprising as FrameDiff is substantially less designable than Proteus (0.392 vs. 0.928 SCRMSD) (see Table 3). Nevertheless, by using Su- increase designability, novelty, and maintain diversity, outperforming simple
PERDIFF (OR). averaging. We further investigate the composition made by SUPERDIFF (OR) in Fig. 3. Here, we see a few modes (particularly on the plot's left-hand side) that Proteus does not generate; SUPER DIFF (OR) can maintain knowledge of these clusters (although to a lesser extent) while maintaining designability.
We also find that SUPER DIFF (AND) outperforms averaging in designability and diversity. Perhaps more impressively, SUPER DIFF (AND) can generate the most proteins that are furthest away from the set of known proteins (sc score < 0.3) by almost two times more than the next best method. We visualize these proteins in Fig. A2 and explore composition in Fig. A1. This motivates the utility of applying our method in novel discovery settings.
4.4 SMALL MOLECULE GENERATION
We evaluate SUPERDIFF (AND) for multi-property molecule generation, where candidates must satisfy multiple constraints (e.g.. binding to a target protein, non-toxicity, and bioavailabilityy. Conventional approaches filter molecules post-hoc or optimize properties separately (Schneuing et al., 2024), but SIFERDIFF (AND) enables direct control. We investigate molecule generation with the following pairs of property pairs: (1) high binding score to the protein GSK3,3 and high drug-likeness (QED) (Bickerton et al., 2012) and (2) high binding scores to the proteins GSK3,3 and JNK3 (an example of dual-target drug design (Jin et al., 2020; Chen et al., 2024)). We generate molecules using LDMol (Chang & Ye, 2024), a latent diffusion model that generates a SMILES string (Weininger, 1988) conditioned diffusion model. Following Wang et al. (2024b), we use the prompts "This mo molecul inhi Enhibits [GSK3B/JN33) and "This molecule looks like drug".
text prompt. Again, we take each prompt-conditioned model as separate

---

Published as a conference paper at ICLR 2025
Table 4: Multi-property molecule generation results. For a set two target properties (P₁ and P2), we report the average top- 10 and top-1 best performing molecules (for the joint methods, "best" is taken as the largest Pi PPa scores). We also report the average minimum scores of each property pair, as well as the diversity, validity & uniqueness (V&U), and quality of all molecules. Metrics are taken from runs of batch-size 512.
Top-10
Top-1
Pr/Py
minE1. Fa)
Py
(1)
Quai (1)
0,107 uas
0.360
2.680
1.2
0.967
0.712
Joint prompeing
Aug spores (Lin 2022e)
G5K3d
QED
0.15
0.200
0.088
0:287a0000
0 0:367amm4
01631 1cus
03338 -133
0.307
0.919
0.743.0
0.308.0.311
0.921
0.001
0.010
0.630
0.731
0.775
0.191
0.114
0.157
SupraDere (A3I)
0:17740-
0.277
0.088
0.410gn122
0,680
0.014
0.785
0.141
ouly
0.007 6077
0.4 0.3
0.136
0.550
0,208 0.208.a.ve7
0.907
0.741
0.070
Pa ouly
Joint pummpinn
Avg. of scceox (1 iu all, 30226)
GSK3
JNK33
0.073
0.056 L.
0.07
0.186,3005
0.
0.351 0.351 15707
0.270
0.367
0.156
2.200
0.187 Lnaus
0.340
0.551 0.395
0.466_0334
0.325
0.296
0.515
0.910
0.911
0.500
0.500
0.718
0.702
0.098
0.082
0.077
SUPERDerT (ANI)
0. 40.80
0 20.00..000
0.361aeux
0.214
0 11:06
0. 0.560
0.908
0.726
0.088
Evaluation metries. We evaluate properties using oracles from Therpeutic Data Commons (Huang et al., 2021);; we also compute diversity (Tanimoto distance on Morgan fingerprints (Rogers & Hahn, 2010)) and quality (Lee et al., 2025), which is the fraction of molecules that are valid, unique, have QED 2 0.6 and synthetic accessibility (SA) S 4.0 (Erll & Schuffenhauer, 2009).
SUPERDIFF improves multi-property molecule generation. In Table 4, we evaluate molecules generated by prompting for each for property separately (P 222/22 only), taking the average of scores, and with SUPERDIFT (AND). We also compare to joint prompting, where we generate using a prompt of both properties: "This mo molecule (prop_l _1 and (prop_2) ". We create a prompt for both property orderings and randomly sample 50% of the final molecules from each ordering to evaluate the same number of molecules. We find that SUPER DIFF (AND) is able to generate molecules that have higher products of predicted properties compared with averaging scores and joint prompting. SUPERDIFF (AND) also has a higher average minimum score for each property pair, indicating that the molecules better satisfy both properties simultaneously. We visualize the best molecules in App. I.
RELATED WORK
Compositional diffusion models. Combining multiple density models into one model with better properties is a classical subject in machine learning (Hinton, 2002). For diffusion models, the most straightforward combination is via averaging their respective prpeetivelearned learned score functions (Liu et al., 2022a; Kong et al.. 2024), which can also be viewed as form of guidance (Ho et al., 2020; Dhariwal & Nichol, 2021; Bansal et al., 2023). Another way to combine diffusion models comes from their connec- tions energy-based models (Du al., 2023; 2020; Nie et al., 2021; Ajay et al., 2024). This, however, comes under a strong assumption that the marginal densities of the noising process are given, which not the case in most of the modern applications (e.g. (Rombach et al., 2022)); our method resolves this. Density estimation, also, is the main bottleneck for the continual learning of diffusion models, e.g. Golatkar al. (2023) proposes learn a separate model for the densities in order to simulate the vector fields from Prop. 3. Finally, Zhong et al. (2024); Biggs et al. (2024) propose to combine models by averaging their weights which, however, does not allow for merging of models with different archi- tectures or different conditions both resolved by SUPERDIFF in Sec. 4.3 and Sec. 4.2 respectively.
The concurrent work (Karezewski et al., 2024) independently proposed the smooth density estimator (Prop. 5) and an estimator similar to the Itô density estimator (Thm. 1). However, for the latter, the authors do not consider other types of dynamics besides the reverse-time SDEs from Prop. 1. For additional related works regarding protein generation, see App. E.
6 CONCLUSION
Despite the ubiquity of diffusion models, many possible ways of performing generation remain unexplored, with classifier-free guidance being the only practical option (Ho & Salimans, 2022). In this paper, we address this shortcoming by proposing two novel methods for combining different models (or the same model with different condition variables) using SUPERDIFF for joint generation.
Limitations. While computationally efficient SUPERDIF is still limited by the computational budget required to produce the outputs of each model. In particular, the combination at the level of model outputs cannot be simply done via cheap combinations of pre-trained model weights. This, however, invites us to develop more principled grincipled-archieetture architecture and training-agnoster method method that does not require any assumptions prior assumptions which makes SUPERDIFF a general purpose method.
Future Work. Our method unlocks novel research directions by allowing for the principled generation of novel samples that were not previously possible to generate easily Sec. 4.2. Furthermore, we argue that the proposed way to estimate the density during the generation enables numerous new potential ways to control the generation process by providing information about the likelihood.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To facilitate reproducibility of our empirical results and findings, we intend to make our code publicly available in the final version. We describe all mathematical and algorithmic details necessary to reproduce our results throughout this paper. In Sec. 3 we outline the theoretical basis and mathematical framework for our method. Furthermore, we provide pseudocode for our method in Algorithm For our theoretical contributions, we provide detailed proofs for all theorems and propositions in App. A, App. B and App. C. We provide experimental details for the CIFAR-10 image experiment results in Sec. 4.1. Details regarding experiments for concept interpolation via Stable Diffusion are discussed in Sec. 4.2 and App. J. Experimental details for unconditional protein generation are described in Sec. 4.3 and App. H.l.
ACKNOWLEDGMENTS
The authors thank Rob Brekelmans, Francisco Vargas, and the anonymous reviewers for providing the feedback significantly improving the paper. The authors thank Viktor Ohanesian for imple- menting the algorithm for Stable Diffusion XL https: / /wwughggnggg ingface. co/ superdi
fuperdiff-sdxl- -11xx--v1-0. 0. The research was enabled part by by the Province of Ontario and companies sponsoring the Vector Institute (http:// http://eetrrrsstttt.too
ai ai/partners/////eaeeaaaaaaaaaaat the computational resources provided by the Digital Research Alliance of Canada (https: / //alliancecar ca), and Mila (https:
KN is supported by IVADO. AJB is partially supported by an NSERC Post-doe fellowship. This research is partially supported by EPSRC Turing AI World-Leading Research Fellowship No. EP/X040062/1.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 200.042303.08774, 2023.
Anurag Ajay, Seungwook Han, Yilun Du Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslic Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36, 2024.
Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiv:2303.0879) 2023.
Arpit Bansal, Mong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
G Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar, Yusheng Xie, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Diffusion soup: Model merging for text-to- image diffusion models. arXiv preprint arXiv:2406.08431, 2024.
Avishek Joey Bose, Tara Akhound-Sadegee Guillaume Huguet, Killian Fatras, Jarrid Cheng-Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexander Tong. Se(3)-stochastic flow matching for protein backbone generation. In The International Conference on Learning Representations (ICLR), 2024.
Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is a predietor-coorectorr. arXiv preprint arXiv:2408.0990 2024.
11

---

Published as a conference paper at ICLR 2025
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https://openai. eeceenii.cmmmee htt.://o///ereaeeeee research/ asseeessssssw-rdee simllmmtutt.r.....
rideo-generation
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2712, 2023.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint wrXiv22402.04997 2024.
Jinho Chang and Jong Chul Ye. Ldmol: Text-conditioned molecule diffusion model leveraging chemically informative latent space. arXiv preprint arXiv:2405.1782) 2024.
Sheng Chen, Junjie Xie, Renlong Ye, David Daqiang Xu, and Yuedong Yang. Structure- dual- target drug design through collaborative learning of pharmacophore combination and molecular simulation. Chemical Science, 15(27))10366-10388 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35:16344-16359, 2022.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et Robust deep learning- based protein sequence design using proteinmpnn. Science, 378(6615):44) 5):49556, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Q Dong, Lei L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey for in-context learning. arXiv preprint arXiv:2301.000 00234, 2022.
Yilun Du and Leslie Kaelbling. Compositional generative modeling: A single model not all you need. arXiv preprint arXiv:2402.0f 103, 2024.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
Yilun Du Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Composi- tional generation with energy-base diffusion models and meme. In International conference on machine learning. pp. 8489-8510. PMLR, 2023.
Nouha Dziri, Ximing Lu, Melanie Selar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances Neural Information Processing Systems, 36, 2024.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1:1-11, 2009.
Aditya Golatkar, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Training data protection with compositional diffusion models. arXiv preprini arXiv:2308.01953 2023.
Herbert and M Sternberg. Maxcluster: tool for protein structure comparison and clustering, 2008.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
Jonathan Ho and Tim Salimans. Classifier--ree diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6853 2020.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang. Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20406-20417, October 2023.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Processing Systems, NeurlPS Datasets and Benchmarks, 2021.
Guillaume Huguet, James Vuckovie, Kilian Fatras, Eric Satbodeau-Laufees Pablo Lemos, Riashat Islam, Cheng-Hao Liu, Jarrid Rector- Brooks, Tara Akhound-Sadeen Sadegh, Michael Bronstein, al. Sequence-augmentee se (3)-flow matching for conditional protein backbone generation. arXiv preprint arXiv:2405.20:: 2024.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in StatistissSimullatio and Computation, 1853):105--1076 1989.
John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier. Dana M Lord, Christopher Ng- lg-Thow-Hing, Erik R Van Vlack, et al. Illuminating protein space with a programmable generative model. Nature, 62307989):100--1778 2023.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference on machine learning, pp. PMLR, 2020.
Marco Jiralerspong, Avishek Joey Bose, Ian Gemp, Chongli Qin, Yoram Bachrach, and Gauthier Gidel. Feature likelihood score: Evaluating generalization of generative models using samples, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 20V::2001.08661,, 2020.
Rafal Karezewski, Markus Heinonen, and Vikas Garg. Diffusion models as cartoonists! the curious case of high density regions. arXiv preprint arXiv:2411.0129% 2024.
Lingkai Kong, Yuanqi Du, Wenhao Mu, Kirill Neklyudov, Valentin De Bortol, Haorui Wang. Dongxia Wu, Aaron Ferber, Yi-An Ma, Carla Gomes, et al. Diffusion models as constrained samplers for optimization with unknown constraints. arXiv preprint arXiv:2402. 18012, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems
Principles, pp. 611-166666 626, 2023.
Seul Lee, Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Yuxing Peng, Saee Paliwal, Weili Nie, and Arash Vahdat. Genmol: A drug discovery generalist with discrete diffusion. arXiv preprint arXiv:2501.06155. 2025.
Yeqing Lin, Minji Lee, Zhao Zhang, and Mohammed AlQuraishi. Out of many, one: Designing and scaffolding proteins the scale of the structural universe with genie 2. arXiv preprint arXiv:2405.1 15489, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Faral-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2025
Sidney Lyayuga Lisanza, Jake Merle Gershon, Sam Tipps, Lucas Amnllt, Samuel Hendel, Jeremiah Nelson Sims, Xinting Li, and David Baker. Joint generation of protein sequence and structure with rosettafold sequence space diffusion. bioRxiv, pp. 2023-05, 2023.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423 439. Springer, 2022a.
Qiang Liu. Rectified flow: marginal preserving approach to optimal transport. arXiv preprint arXiv:2209. 14577, 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. International Conference on Learning Representations, 2022b.
Michael Luo. Justin Wong. Brandon Trabucco, Yanping Huang, Joseph E. Gonzalez, Zhifeng Chen, Ruslan Salakhutdinov, and Ion Stoica. Stylus: Automatic adapter selection for diffusion models. Advances in Neural Information Processing Systems, 2024.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2018.
Midjournyy. https: / wwww mi wwtp::iimdjoouero..oooo.o wttpeeey/cco///////.ee/////// 2023. Accessed: 2023-09-09.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34: 13497-13510, 2021.
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pp. 65-84. Springer, 2003.
OpenAI. Gpt-4 technical report. ArXiv, awt/aaa3.////////// 2023.
Stefano Peluchetti.
Diffusion bridge mixture transports, Schrödinger bridge problems and generative modeling. arXiv preprint arXiv:2304.00917 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. PMLR, 2021.
Milong Ren, Tian Zhu, and Haicang Zhang. Carbonnovo: Joint design of protein structure and sequence using unified energy- -ased model. In Forty-first International Conference on Machine Learning, 2024.
David Rogers and Mathew Hahn. Extended-cooneecivvty fingerprints. Journal of chemical information and modeling, 4055):742-554 2010.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? arxiv. arXiv preprint arXiv:2304. 2ee.22334.1100 15004, 2023.
Arne Schneuing, Charles Harris. Yuanqi Du, Kieran Didi, Arian Jamasb, Ilia Igashov. Weitao Du, Carla Gomes, Tom Blundell, Pietro Lio, et al. Structure-based drug design with equivariant diffusion models. Nature Computational Science, 4(12):899-909, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022.
14

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2020.
Stability AL. httppp:///taaaalltt...aaaaaaaaaa 20233. Accessed: 2023-09-09.
Jacob Steinhardt. Al Forecasting: One Year In.
regret to/ ai - forecastingggg-ee one year-errrinnn in/, 2022. Accessed: 2023-09-09.
Trieu H Trinh, Yuhuai Wu, Quoe V Le. He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482 2024.
Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023. URL https:ttp//....
/wwppreeevvieeeeeeeeeeeeeeeeeeee new. net/ / forum? i dd=6TXRxNNME1X.
Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature biotechnology, 42(2))243-246, 2024.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 16671:1661-11774 2011.
Chentong Wang, Yannan Qu, Zhangzhi Peng, Yukai Wang, Hongli Zhu, Dachuan Chen, and Longxing Cao. Proteus: exploring protein structure generation for enhanced designability and efficiency. bioRxiv, pp. 2024-02, 2024a.
Haorui Wang, Marta Skreta, Cher-Tian Ser, Wenhao Gao, Lingkai Kong, Felix Chenru Duan, Yuchen Zhuang. Yue Yu, Yanqiao Zhu, et al. Efficient evolutionary search over chemical space with large language models. arXiv preprint arXiv:2406. 16976, 2024b.
Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew Borst, Robert J. Ragotte, Lukas Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock. Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Nature, 12097976):1089-11000 2023. ISSN 1476-4687.
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang. and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Jason Yim, Andrew Campbell, Andrew Y. K. Foong, Michael Gastegger, José Jiménez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S. Veeling, Regina Barzilay. Tommi Jaakkola, and Frank Noé. Fast protein backbone generation with se(3) flow matching, 2023a.
Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprini ar arXiv:2302.022: 02277, 2023b.
Jason Yim, Andrew Campbell, Emile Mathieu, Andrew YK Foong, Michael Gastegger, José Jiménez- Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Frank Noé, et al. Improved motif-scaffolding with se (3) flow matching. arXiv preprint arXiv:2401.00882, 2024.
Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint 102.:24022.16443, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROOFS FOR SEC. 2
A.1 PROOF OF PROP. 1
Proposition 1. [Reverse-time SDES/ODE) Marginal densities qe (x) induced by Eq. (1) corre- spond to the densities induced by the following SDE that goes back in time (T = 1 t) with the corresponding initial condition
dx- = -(fRlor)) + ( g6
+5+ vogg))o )ar + 22,, 27=0 (xo):
(2)
where Wa 7 is the standard Wiener process in time T, and & is any positive schedule.
For the forward SDE we can write the corresponding Bokker-PPlance equation
of 2 (q((x),
Öt (()))) - (V, (v.((=)(((=))
(19)
then for the inverse time 7 = 1 - L, we have
Ør 1-91- (z) (V. 91 (z)f (x)) gi- (q1-+())
(20)
2
= =(V, v 11 (x)) 11 (1-)(x)=
gi-t
logg (0g)1-+(1)
(21)
2
g + &r (log )) )) E, (q))(()),
(o) (x) -fi -f)-r(()+ +
(22)
which corresponds the SDE
gi-s + V (loggi-((0))) + 22r
dx, dr (fi-r((r)
(23)
2
A.2 PROOF OF PROP. 2
Proposition [Ornstein (Ornsten-UUlleneer) Uhlenbeck SDE) The time-dependent densities in Eq. (3) correspond to the marginal densities of the following SDE, with the corresponding initial condition
log Ot dl +
V2 20 θ Tt
da.
log
dW., 2o ~ (26))
(4)
Ot f((x))
at
Proof. For individual components of from Eq. (3), we have to find the vector field satisfying the continuity equation, which we can rewrite as
① log (z)) = (V (x)) (V..og
(24)
Using the formula for the density of the normal distribution (((x) = Nax amp of), we have
1
V log (z) = o7 (x asp²),
(25)
0
8
appi,pt) 1r) Oo, || - |ap-"1111 2 og (=--app'
da: 0t
log (x) = d ddd logo, + o₂
(26)
Ot at
/
2 algg oi
Olog 0t
dos
V
log (x))
Ol (x aspp)) + at
n
(27)
v((x))
(V., vf())) (V, log (x),,, u((=))
(28)
16

---

Published as a conference paper at ICLR 2025
For the mixture of densities qi from Eq. (3), use Prop. 3 and write
log at
da, (xc aspt)) +
(xxx) (
(a(a)
Ot n²)
(29)
ôt
==1
2 log at
Oc:
a olog
0t
(x(x)) Σ db(x))n'
(30)
Ot
At the same time
N
V (o))
= (x(x) Σ (((a)
op ( ampi))
(31)
N
Σ
1
[ož V (x) +2]
(32)
(c(x)
at
i=1
Using this formula in Eq. (30), we have
i Olg at
1 Oo,
log 0z
alog
oz
1 of Vz log (e()
vx(x))
+
at a
(33)
Öt
at öt
Øt
Öt
Ot
2 log oe Ot
2 0
log
V log (xz))
(34)
Öt
an
Hence, we have
0t (9(2)= = - (VV.((((V))))) -
-v()) (
Olog ar
a)) + of 0
log Aq(()), at
at
(35)
Ot
Ot
which corresponds to the SDE in the statement.
A.3 PROOF OF PROP. 3
Proposition 3. [Superposition of ODEs (Liu, 2022)] The mixture density in Eq. (6) follows the continuity equation with the superposed vector fields from Eq. (5), i.e.
i Øt 41 (x) - <V., mix (x)o((1)) vx(x) = Ej w (l)
N
Σ i=1
(x)
(7)
Proof. By the straightforward substitution, have
i
m
4t
(r)
at (x) - Σ of (Vnn(()))))))))))
(36)
Ht
m
of mix (x) m
(
(37)
gi (x)
V St mix (x)
(((())(((z)
(38)
Σjn wit oo (x)
((()))
A.4
PROOF OF PROP. 4
Proposition [Superposition of SDEs) The mixture (x)) ΣNn w'(((a) of density
marginals {g. ( induced by SDEs from Eq. (8) corresponds to the following SDE
dx, u- (x g.dW,, ux(x) Σnn wil
Σ (d
(9)
17

---

Published as a conference paper at ICLR 2025
Proof. By the straightforward substitution, have m
① mix
Σ ( = Σ w (-VVang((o)a((e)) o sai()))
1/2
0t
9
(z)
(39)
The first term is analogous to Prop. i.e.
m
( Vx, Vz. mix (x)
m
(-(Ve.)(()))(()))
Σ w (x)))((())))))))))))(())){)))(
(40)
1== تن gl (x)
(c(
The second term is
= g
Ag) = 22 (
Σ
(x)
>
w'
(41)
2
2
2
i=1
This results in the following PDE
9
0L 4 (x)
mix
Vx
Li
mix (x)
w ())(()((())
+ 2
Ag. Am)i(()) (x).
(42)
プラ=1 wi oe (x) ==1
ux(w))
B FLOWS AND DIFFUSION AS A SUPERPOSITION OF ELEMENTARY VECTOR FIELDS
From Prop. 3, immediately get the main principle for the simulation free training of generative flow models, as demonstrated in the following proposition.
Proposition 7. [Superposition of La-losses] For the parametric vector field model Velar; 0) with parameters the Lz-loss for the vector field from Prop. can be decomposed into the losses for the vector fields from Eq. (5), i.e.
S. dt -
Σ wi S dd - 0x1 + constant,
N
where the constant does not depend on the parameters 0.
Proof. By the straightforward calculation, we have
S/ dt (l)))))) v. (x; = S/¹ dt (xg [Iv((=)]) 2(0((=). 0)]
where the first term is constant w.r.t. 0 and the last term is amenable for a straightforward estimation. The middle term, according to Prop. 3, is
f dr a. (=(1)))))) ff
f dx Σ (dde)
(43)
Thus, we have
S/d dt Eapr(a)) [|r( -1(x)))1-((v((x)) vola; 0)) + (1)((2;))11
(44)
S.' dt [ - Σ w { + Σ w [
(45)
2
w E (i))) (0(()),))
+
w ا l1
(46)
18

---

Published as a conference paper at ICLR 2025
L di Σ w B
m
(to))ll v)(1;0)][ + constant.
(47)
The proof for proposition follows a simple extension of Liu (2022). Consequently, the generative modeling problem explicitly boils down to learning time-dependent vector field (2; 0). Moreover, for the generative modeling task, the noising process is required to satisfy the following boundary conditions & (z) ö(x u²) and qf (z) Phoise (a:) Vi D. Generating samples that resemble ()ata(μ) during inference then simply amounts to solving the reverse-time ODE with the learned model (2;; 0) starting from noisy sample 2 Pnoise
Diffusion models can be incorporated into the flow-based framework by selecting Gaussian forward process which gives the marginal density at a timestep as the mixture corresponding Gaussians,
of xz) Nx app' ot), (e(z)
Σ (z)
(48)
Under this Gaussian diffusion framework, we can express the vector fields analytically using the score, which formalized by the following proposition.
Proposition 8. [Diffusion processes] The time deeeddennndent densities in Eq. (48) satisfy the following continuity equations
Ot (x)) = - (V.. (x))) (f()) = log a: ①
0
fae 0t
(x - axp²)) +
(49)
Ot
Ot (((x) = - -VV2.
((xx) =
i log 04
ôt x
log
at
v log (x)).
Öt
aq
(50)
The proof repeats the proof for Prop. 2 in App. A.2 since Eq. (50) corresponds to the Omstein- Uhlenbeck process:
d log an Øt
rill + V2(02 log - dW..
dz.
(51)
Diffusion models using Gaussian perturbation kernel enjoy the benefit of giving an exact expression for the Stein score of the perturbed data which leads to the celebrated denoising tive (Vincent, 2011; Ho et al., 2020). Analogously to Prop. we can write the denoising score match- ing objective the superposition of the scores as demonstrated in the following proposition.
matching objec-
Proposition 9. [Denoising Score Matching (Vincent, 2011)] For the parametric score model V log q (z; 0) with parameters 6. the score matching objective can be decomposed into the corresponding objectives for the individual scores, i.e.
f dt (x) log (z) V log qi (z; ()1²
1 di Be((()) V log (x) - V log qe 0)|| + constant,
N
where the constant does not depend on the parameters 0 and V log (f(x) = Jo (x asspe))
Proof. From Prop. 8, have
Ölog a:
da,
(x) =
(x - anpp)) +
(52)
It
Ot
Olog Ti (x - (x + o² log (x)))
da: (r + of (x)
(53)
01
Øt a
19

---

Published as a conference paper at ICLR 2025
i loog 01
a
2
log
- V log (x)
(54)
di
at
(x) =
log at
x + oi 0t
log at V. log qt (z)
(55)
Öt
Applying the same change of variables to the parametric model
b Olog + oi log V (z;
(2 0)
(56)
Ot
0t
and using Prop. have
S( di (x Σ ! di
m
1
Bat( [o((()) - oe(1; 0)12 constant, (57)
N
S( dt (x) Va (x) - v 2o q: 0)]
(58)
= m
Σ / dt E (G)
constant
V. (x) V. log (z;
(59)
i 0t
log
which concludes the proof.
C DENSITY ESTIMATORS
Proposition 5. [Smooth density estimator] For the integral curve xtt) solving dr/dt u(xs)) @
and the density g (xx()) satisfying the continuity equation on (x)=) log- log-density along the curve changes according to the following ODE
(=<)))((((()) the
d dt
log (o(g)(x(t)) = -V,,,, - (v> log (r)g (f()) (((() u,())
(10)
Proof. By straightforward computation, have
log (x())) = log (1()) + (Vo logg(())) di di
(60)
at
and using the continuity equation de ( - (V=)(()))((())) we have
0
(og = - (V., 2x()) - log of()) Øt d
(V
(61)
हi [ogg(((()) = - (V., ux(x)) V.
ue(x))
(62)
Theorem 1. [Itó density estimator] Consider time time-dependent density (z()) induced by the marginals of the following SDE
dr, = fe(xz)dt + gidW,, Iz=o ~ go(x), E [0,1],
(11)
where dW, is the Wiener process. For the reverse-time (T = SDEs with any vector field u, and the same diffusion coefficient 9t. i.e.
dx, ur (2+)dT + -1--dW r, T [0, 1], the change of the log- log-density log qr (Ix) follows the following SDE
(12)
dlog 91-7 (xx) (dr,,, V Vlog Vlogg1-((5))) (1r
(13)
9 gi-
+ /(-((x=)
V log Q - (x+), V logg) dr.
2
20

---

Published as a conference paper at ICLR 2025
Proof. The Fokker-Planck equation describing the evolution of the marginals for the forward process
@q((x) = (V. + ge
Aq.(x))
(63)
Öt
2
hence, for the inverse time 7 = 1, have
R-- (q1-r(1)
θT (z) = VV. f1-1f11)1)
(64)
2
log Q (1- = (V, f1, + (V log Q1 ( (z),f11 [1-r(z))
θT
(65)
gi-r A log 91 (x)
-1-r
V log 91)
2
For the following reverse-time SDE
dx, = ur (z,) + -1-
(66)
using Itô's lemma for log (1-r(x7) (2r), we have
log (17) =
log 91 1111-1110 (x7) + (V loggl--(((r)) ur (I+))
(67)
g r A log (x.) dr + 91- (1r (I,)
+
(68)
2
Using Eq. (65), we have
((V 91 ((v1gg)--(())))
- log 91 9 (0)))+
Φi V
dlog (x+) =
(2-))) f1-
(69)
2
+ (V. (V. dr+
Vlog (logg=>(<rr) ur (I+ dT + 91-11-10 91-,dW,
(70)
dz,
C.1
PROOF OF PROP. 6
Proposition 6. [Density control] For the SDE
M
dx.
Σ (xx) + -1--dW
(16)
where are the weights of different models and E, Kij = 1, one can find that satisfies
d dloog ( = dllog (i,((177) (x7), V i,j E (M) by solving system of M linear equations w.r.t.
(17)
Proof. Using the result of Thm. 1, for the density change of every model, we have
doo (x,) = (dx,, v log ( + ( (z)))
(71)
f1 (x+) 0 v ( V log qी qi ((st)) dr
(72)
Σ
(jdT(uuk (I+) (u. v Voog (2+)) + (V+))
(73)
jml
{ 01 (w - (fi
g g v log vi dr, Vlog (i
(74)
2
21

---

Published as a conference paper at ICLR 2025
Let us denote
au dr (xy), log loggi-r(<r))
(75)
b, - (V. + (o- + - V log qi -
R-
(()))
)dr dr, v log (xT)
2
(76)
Then the change of densities can be written the following linear transformation
dl (x,)
d log - (x7)
= Ak + b.
(77)
d log q1" (3i+)
Note that we want to find x such that d log - = dlog o equivalent to the following system of M linear equations
Vi.j E [M] and E, Kg
This
@1
aIM
b1
(78)
@.1
OM M
RM dlog
bm
1
where we have introduced new variable d log = dloog _i V E [M]
D DISCRETE- TIME PERSPECTIVE ON THM. 1
In this section we derive Thm. 1 from another perspective by operating with discrete time transition kernels and the detailed balance. Namely, we aim compute the marginal density by starting from the detailed balance equation, which states the equivalence of the following joint densities:
(x (x) (v) (t+ xr).
(79)
The detailed balance condition simply states the pair (xr, y) can be sampled two equivalent ways: 1.) sampling y from the marginal gi (y) and then sampling via the noising kernel ka:(r) y or 2.) sampling x from (++A+ and then denoising it via ( (x). Indeed, there are infinitely many valid kernels that may satisfy the detailed balance; we make specific choice informed by the following principle. We aim to construct a universal noising kernel that is independent of the data distribution or other densities, i.e. Paata (-), (-()), (+) Remarkably, this principle results in unique kernel choice, which we formalize in the following proposition.
Proposition 10. [Universal noising kernel] For any data density Paota(p) for the continuous noising process (x) = du N(x aefs, Pa)taa((()) there exists unique noising kernel
ka 11) = N astai y, S.tAs) (9t-2e(x) = =
=f
ff dy (y x))((1),
(80)
where SHA = 0f+ of aisse and is independent of the densities Paata (-), gc((), (4)
The proof for this proposition is presented in App. D. Note that the proposition holds exactly for any At-ic. there are no any assumptions on the scale of At any approximations.
Once the noising kernel is fixed, the detailed balance Eq. (79) uniquely defines the denoising kernel Tae(y z) that propagates samples back in time. However, the analytic form of this kernel depends on the densities (x)) and qe (v), which unavailable for the modern large-scale diffusion models that are served using limited API endpoints. Therefore, we consider the Gaussian approximation of the reverse kernel and justify its applicability in the following theorem.
Theorem 2. [Denoising kernel] For the universal noising kernel kse(æ y), the Gaussian approximation of the corresponding reverse kernel (yr x)) ksr( Vi) (t)/4/+))///)))) is
Farlyl x) =N -N(1)
at
S
Vog log
ar S S ) (81)
01+ar
04-At
01+
22

---

Published as a conference paper at ICLR 2025
which corresponds single step of the Euler discretization of the following SDE
dx, - ② Ot
ô
dt v loga(z)) dr + v 2 0 or dW,, (82)
log au 207 log
log
Ot
at
Ot at
and
DeL (Fa x ( x)) o(A))
(83)
See proof in App. D.3. Given the noising kernel from Prop. 10 and the approximation of the reverse kernel from Thm. 2. we propose to estimate the log marginal density of the current sample using the detailed balance as follows,
Par ar)
log qe (v) log (() log (1 (v) log Par(y ())
log
(84)
Par x))
where the last term involves the unknown reversed kernel. We argue that this term can disregarded due to Lemma 1. Namely, in the following theorem, we study the distribution of the last term for different samples y generated from x and argue that this term can be ignored.
Theorem 3. [Detailed balance density estimator error] For y = x + Ât. u I Ate,
where 9t+At = 20, at log at and E is a standard normal random variable, the following estimator is unbiased w.r.t. the generated samples y. i.e.
Tar x) = o(At),
Ee
log
(85)
FAI x)
and the variance is
De log rod (y at)
A 2i at D.
( Prg 8) + o(A())
(86)
Fai (1 x)
Ox
where 8 low 01 (a()) is the Hessian matrix of the log- densiiyy.t.
We postpone the proof of this theorem until App. D.4. Note that the next variable y is completely defined by the random variable E. Thus. the mean and the variance are evaluated taking into account different outcomes of y. However, the variance of this distribution is related to the variance of Hutchinson's 's estimator. Indeed, for the Euler integration scheme of the log-density log (I+A:()++ar) log g(x+) At (Vo.)( (zx)), Hutchinson's 's estimator is
r (vr( E ~ T (rv((()))
V us (xx)) = E
E, E~ NN(,1),
(87)
ax
Ozi
where
Ovalze) is the Jacobian matrix. The variance of this estimator is Dxc
D& De(log (+<ae(F(4a)) Ar e dv((2))
(88)
Dxs
Note that here the variable e is newly introduced and is not related anyhow to F++ Thus, we arrive at the following estimator of the change of density
log (1) log qt+ (x))
log ka. (x v) log Far(y) x)).
(89)
We further highlight that since the generation via the inference process happens in many steps with small At, we approximate the relation Eq. (89) up to the second order terms in At.
Proposition 11. [Recurrence relation for the log- log-density] For y === x + At u + gI Ate. where Ii+At 20 0 log a and is standard normal random variable, the estimator from Eq. (89) can be expanded as follows
at
llog kar v) + logiselyl 1)) = dAi ②
log 01+4: Al Fi+ log logg)+
H.
+ At At I AtE + At
log ALT V Vlog + +((At)
Ft
(90)
23

---

Published as a conference paper at ICLR 2025
The proof for this proposition is included in App. D.5. For practical use cases note that Eq. (90) the final form of our density estimator.
D.1 PROOF OF PROP. 10
Proposition 10. [Universal noising kernel] For any data density Pdata(p)), for the continuous noising process qe (x())) = dp (xx) aitt, of Paata (11), there exists unique noising kernel
(a 1) -
a+tat y, S.tA)) ,4- . (o())(()= f dy kady x)qe(y),
(80)
at
where SHA = o oi
: and is independent of the densities Paata (-), gc(-), (+t2r()) of
Proof.
(x) = / du N( oept, Pamaa (14)
(91)
Let's derive the incremental kernel kad (2 y) from this formula, i.c. (++r(x))
f dy karlæ (2) (y)
(92)
I
ʃ dy kar(=ll)) f du Noy aip, of (pmma())
du du Ni(x N(xl(x ((lxxla4ssPpp of a: (1)
(93)
S du M(ElaetAIAe (e.H,. ott
/ dy karl (B) aeps 02) Pasultt) 0
(94)
N(x attat di+se) = fdy dy kar v) atp of)
/
(95)
Obviously, one can perform the following change of variables
= -0+A/ y/ 1 + Re, where ~ Ny a444, o?), and & ~ 0,,
(96)
where we have to take 5 R² 2
(97)
To make the kernel independent of μ we have to choose ot
Ott
then the kernel is
aq
kacl v) = -N(x)
a++Ai
y,
ot
of
(98)
a₁
of
S7+
D.2 LEMMA
Lemma [Reverse kernel lemma] For the universal noising kernel ka((1) y). the correspond- ing reverse kernel (1 x)) kz(( (1)((()//+++(()) and its Gaussian approximation
( (at stss)) (99)
)))(ylz)= =
at x +
at S V log (+)r(())
ac+st
we have
r + A U Ate (x)
log
(100)
Fa + U- Ate (
92 At Alog -T -
02 log ((()) 8) + o(At),
(101)
dx
where Gt+A1
/ 220 0 log Bttar ++ an aniar
24

---

Published as a conference paper at ICLR 2025
Proof. From the detailed balance equation, we have
log Acl x) = log kas (x: y) + log ge (V)
log ((+)((()))
(102)
1
O'++ At at
d log 2 ++ + log q: (v) - log q:+ (z)), (103)
25 2
2
where we use the definition of the forward kernel. Thus, the difference between kernels is
(y x)
-1
d log 2T S + log (v) log (++ (104)
log
(A
xx)
2. 25t+
2
1
at
2 + S Sit V Vlog (log-+ai(t))
V
(105)
2(
Ar astat
)2
Or+At
ar+At
d log 2T S
+ d
at
log
(106)
2
2
Opening the brackets, we have
log (V x))
-1
11
+ log qc(()) - 1ogg++a(())+
(107)
(1 x)
25
+ 2.5
11
I, V log flog)+aa(x))
(108)
At
04
SP+
+ d
+
log
(109)
ac++tt)
The constants can be estimated as follows
0f ait
doL+s!
Litse
S oo+AA
= o74 -07+ss 2dtoc+
+ oo()t)
(110)
at
2
ot
dot+A! + o(Ar)) (1 1 11+26 2dt 2 I+ dar+st
(0 2dto
+ o(Att
Ōt
aitsi
(111)
2dto i+Si Zl
i log
Ot+At + t((t))
(112)
and
a++11
astAt dou+ (- + o(Att = At
Olog 0++ o(At)
1
(113)
a1
Ot
Øt
d
04
log
dtd log aatt + o(At).
(114)
2
0++A1
Ot
Thus, we have
log
Pyr [x)
log q: (v) - log (x)) + Ato? 0t
log Ø+ IV log -+)a())))=- (115)
( x²)
a+t1i
dtd log acts: - (1+ alog ) - V.-,,, Vlogg+41(01) ) + to(tt). (116)
Ot
Øt
From Prop. 8, the time derivaiiv of the density
log (x) = - (x)) (o)
(117)
Ot
a log 0t
o log
A (z)
(118)
dt
Ot
Ölgg an (Vloga(a), +
0 ② log σt
(z))
(119)
Øt
ot
a,
25

---

Published as a conference paper at ICLR 2025
Hence, we have
log (x() log ((+)(()) -t d
log (1(tta(()) + ((At)
(120)
Ot
Atd alog Plogai+rr aI+Ar
At
o++i 01
8
log @+A1
A Algg++a(z))+
(121)
a1+at
log i log a++ V log (+)) At (+ a1+Ar
0 a+4
+
[Vog
(122)
Using it in Eq. (116), we have
log (y [)) FAl xx)
log (v) log (x) At oi log a++
A Alogg(a/(()
(123)
a1+11
θ Glog a1+ (y 2, log log + o(At).
-(1 + At
(124)
Ot
Finally, we consider y = x + At v + I1+A1 V Ate.
Then we can estimate
1 (y x) 0 log q (x)
log (v())) log 9: (x) V log q: (x), -2) +
(y x)
(125)
Ox
+ +ollyy - x||))
(126)
= (o) + v log q: (x)), At w 9 A Ale
(127)
At Rise 82 log q: (x)
E + d(At),
(128)
Ox²
8² qe(2) denotes the Hessian of the log-density. Thus, for y = + At + I4t.I Me. Ox2
where
we have
Ty x.)
gi-se 82 log (x) At
2 (otts a log a++ ot a4+1/
log
At
A Alg (129)
PAr x))
2
Or
v log (xx) V Vlog (2r) Al 94AIV + +o(A)) (130)
siss
The last term o(At) by expanding the scores time, hence, for 2 have
log ge+ss we गi axtat*
(x + At IIAAI Ate x)
log
(131)
FA + At S4
- 8 (og 2) + o(tt).
AAt o2t log
A (log(()
(132)
ot autsi/
dx
D.3 PROOF OF THM.
Theorem 2. [Denoising kernel] For the universal noising kernel ksc(1) 1)) the Gaussian approximation of the corresponding reverse kernel T (z) = kas( (1) (1)(//((()))))) is
V log ( 1_SS41
Faclyl x)) -N -
at
Si a
Ot
x
(81)
autar
at-at
which corresponds to a single step of the Euler discretization of the following SDE dx, = - 0t
log axx - 20 ②
log a: Vv log (og)(())) dr + 202 θ
log Tt dW,,
(82)
at a
at a
and
DRL(FAely)
x)) = o(At).
(83)
26

---

Published as a conference paper at ICLR 2025
Proof. To find the reverse the kernel, we first Tweedie's formula to find the expectation, i..
(4sa(x)) / dy V log (+>a(())=
I ar+
(v)()))) (133)
SH+
at
82 Ar
V log ((+ @++r((x) / dy
at
karl=llla (134)
Ot
ar+tr
at
St
V log dy kar( uv)c(())
/ dy vr (135)
0t++ At
(++Ar(x)
Thus, we are going to approximate
Setae) ))
FArly x)) = N
Ot
Or S V log Vlogn-ar()),
(136)
a++Ar
a++Ar
a++1
2
where (ai4 S1+11) estimated as follows
is chosen to match the leading term of k2r(2 The constants can be
dot+s: Öt
o7ta0
$2 S44 o4
o
0x
1+ - o4tr
241o+ta!
+ o(At))
(137)
at
at
2 - -(oitAr
2
dot+Al
+ +0(A1)) (1+2A/ 2A/
2HT dos+st
+ o(Ai)
Øt.
01.
(138)
i
TetAt
2A1o A+ Hl
log + +o(A))
(139)
O1+
and
0
dai+sl (-At) + +(A)) = 1 Al
e log acti + +(())..
(140)
Ot-At
Ot
Øt.
Ø1
Thus, y can be generated
y y=n At
0 log a++
2 Dt 0t
log U++! V log (t))
(141)
Øt
a1tAr
2 24te At Øt
log JetA
(142)
a+tar
where E is a standard normal random variable. This corresponds to the single step of the Euler discretization of the following SDE
dx,
a Öt
a log O.r 20,
log ot v. log de (2)) at
2a log
à Tt
LdW,
(143)
Øt
ot
a₁
For the KL-divergence between the reverse kernel and its Gaussian approximation, we have
FAo( A / AtE x))
( =)) Eg log
+((t)) (144)
rar At tr Als
(r)
where
log σt v loggo logg(()): and I+- = /202 20? Øt
log at
log ax - 20t St
(145)
Ot
a
at
From Lemma 1. we have
DKL (Farly) x)]lrar(1| |x) =
(146)
log ot-i
( log
E - A Alog -logg+ + o(Ar)
= At oft St actss
Eg
(147)
dx
o(At).
(148)
D.4 PROOF OF THM.
27

---

Published as a conference paper at ICLR 2025
Theorem 3. [Detailed balance density estimator error] For y = x + AL. u + g++ Ale, 20 De log as and & is a standard normal random variable, the following oe
where Gt+ - 208 estimator is unbiased w.r.t. the generated samples y, i.e.
(y x)
EEg
log
o(St),
(85)
and the variance is
rod 22) Pac x)
At Pi- D.
eT 8 log 1og x) + t((f))
Dg log
(86)
dx²
0 log (w) Ox
where
the Hessian matrix of the log-density.
Proof. From Lemma 1, we have
(a x))
litAc
A Alogg+4c(z)) - - log (())
Eg log
Eg At
o(Dt)
(149)
Tho (x)
Ox
At Pitse
2
Alog Eg Tr eer 0² log 9e (2)
1) + +o(At)
(150)
2
Ox²
Al Ai+
02 log qo (I))
( - Tr
+ o(Art)
(151)
2
Or
=((i))
(152)
For the variance, we have
De log "Ar(Ul α))
At Iit E.
A (log - log -
)) 2
+(At))
(153)
FA x)
Ox
A A
Ditsi
Er
Byy 11
02 log (x()
91 1
=T T
log ())((z) ) + +(Att
8)
(154)
4
Ox²
dx
A Ie+At
0d log qe (x)
D
Or
8
+ o(At?),
(155)
where we used a standard normal random variable 1/-
D.5 PROOF OF PROP. 11
Proposition 11. [Recurrence relation for the log-density] For y x At + g44aV Ate. where It+At = /20? 0 log a and is standard normal random variable, the estimator at from Eq. (89) can be expanded as follows
llog kar (x) v) + Fog x) = dAi ④
log 01+4: Al Fi+
logg)+a(())))+
At At. I44I AtE + At Ft
log a+ At. V Vlog Vlogg++a(()) + +((At)
(90)
28

---

Published as a conference paper at ICLR 2025
Proof. Indeed,
log Pai(y|2)
ai+1t
1/
2
d
log
2T. (+
(156)
2B
at
2
1
y
ag
a
St Sttr V log (ogg+ss(x))
(157)
At 2(St+
a++
log 2T
-dlog
at.
(158)
2
a++
a1
/attsty yー V log (+>+(())))))
= dlog
+
(159)
a++At
a4
S
V llg (og))+s((>))
(160)
Expanding the time around t + At, we have
log Farly [x)
dAl 0 log oggatsr++ (1 Al. log ac+ a:)y x, Vlog
2
(161)
Øt
0 At A+ti log o+tDr
|| log (og)))))))))1 o(tt).
(162)
act11
For y = x At V Aigi+aIe I++A= / (207+2100
log aetat ai-ar"
we have
Farly x) log ( ly)
d&t 2
log 0+ At RitAr V log (og
(163)
+ A AHAIVAAE Ate + At i a4 V Vlog Vloga+s((()) + ((A)).
(164)
Ot
E ADDITIONAL RELATED WORK
Protein generation. Structure-based de novo protein design using deep generative models has recently seen a surge in interest, with a particular emphasis on diffusion-based approaches (Watson et al., 2023; Yim et al., 2023b), and also flow matching methods (Bose et al., 2024; Yim al., 2023a; 2024; Huguet et al., 2024). Moreover, building on the initial SE(3) equivariant diffusion paradigm multiple recent approaches have sought to increase the performance of the methods through architectural innovations (Wang et al., 2024a), conditioning on auxiliary modalities such as sequence sidechains (Ingraham al.. 2023; Lin et al., 2024). Finally, recent approaches tackle the problem co- co-generation which seeks to define joint inference procedure over both structure and sequences (Campbell et al., 2024; Ren al., 2024; Lisanza et al., 2023), but remains distinct from the setting of this work which attempts to combine different pre-trained models using superposition.
F BROADER IMPACTS
In this paper, we present theoretical results and demonstrate use cases in generation tasks such as image generation and unconditional protein generation. Because of the theoretical nature of our contributions, this work carries little societal impact. SUPERDIFF can be used generated protein structure from a composition of existing protein diffusion models. Better protein generation methods can potentially lead to negative in generating bio-hazardous molecules and proteins. We do not perceive this as a great risk at the current stage of these models.
G ADDITIONAL RESULTS FOR CIFAR-10 10
H PROTEIN GENERATION
H.1 EXPERIMENTAL DETAILS
In our setting, we consider two state-of-the-art diffusion models for protein generation: Proteus (Wang al., 2024a) and FifmeDiff (Yim al., 2023b), which were trained on protein structures from
29

---

Published as a conference paper at ICLR 2025
Table Al: Image generation performance for CIFAR-10 with conditional models trained random partitions of the training data (labeled and B). We compare SUPERDIFF (OR) with respective models (modelA and modely) with the model that is trained on the full dataset (modelAuD) and random choice between two models (model B).
ODE inference
SDE inference
FID (L)
IS (1)
FLD
(4)
IS (t)
model
4.75
8.98
6.95 + 0.12
4.66
9.35
6.39 : 0.13
model
4.78
8.97
4.36
9.45
model
5.30
9.04
6.82 = 0.00
2.83
9:11
6.26 = 0.11
model oa
4.75
8.94
0.86
0.15
4.41
9:40
6.3
SUPPERDIFF (OR)
4.74
9.00
6.98
0.10
4.46
9.40
6.22 + 0.15
SUPERDIT T-000 (OR)
4.63
8.96
4.23
9.42
6.27
Protein Data Bank (PDB, (Berman al., 2000)) to estimate the special Euclidean group (SE(3)) cquivariant score over multiple diffusion steps. The models' outputs predict the coordinates of monomeric protein backbone.
We use pre-trained checkpoints from Proteus and FrameDiff", During protein generation at inference, we separately combine scores for translations and rotations from both models using Algorithm 1.
We investigate the of different temperature (T) settings scale for controlling the densities. We also found that adding small bias (w) towards Proteus log densities improved designability.
H.2 METRICS FOR EVALUATING GENERATED PROTEINS
Designability. We assess designability using the self-consistency evaluation from Trippe et al. (2023), where given a generated backbone, we predict its scaffold using sequence prediction model (we use ProteinMPNN (Dauparas et al.. 2022)) and re-fold the sequence using structure prediction model (we use ESMFold (Lin et al., 2022)). We then compare the re-folded protein to the original generated backbone by computing their template modeling score (scTM) and root-mean-square square-distance (SCRMSD). A protein is considered to be designable if its SCRMSD is < 2 to the refolded structure. For each protein, we repeat this process 8 times and keep the sequence with the lowest scRMSD. We report the fraction of designable proteins for each method in Table 3 as well the scRMSD mean of the resulting dexignable proteins.
Novelty. A significant impetus for generative modelling in biology and chemistry is to propose compounds that have not been previously identified (i.e., different from the training data), but are also possible make. We compute three novelty metrics: the fraction of designable proteins with a scTM score < 0.5 (higher is better), the fraction of designable proteins with a scTM score < 0.3, and the mean ScTM score between generated proteins and the proteins from PDB that the original diffusion models were trained on, which represents the collective human knowledge of protein structures; lower score indicates higher distance from the training set and is desirable since shows generalization ability.
Diversity. To measure how diverse the generated proteins are, we compute their mean pairwise SCTM score (lower is better), and also report the fraction of unique clusters formed after clustering them with MaxCluster (Herbert & Sternberg, 2008) (higher is better). Finally, we report the fraction of proteins that contain 3-sheet secondary structures, as it has been found that these structures typically more rare to generate (Bose et al., 2024).
H.3 PROTEIN DIVERSITY EXPLORATION
To embed and cluster the generated protein backbones we use the Foldseek (Van Kempen et al., 2024) package to compute pairwise aligned TM-scores. We then use the UMAP (MeInnes et al., 2018) package to compute 2D embedding. Where proteins are represented as points that are close to each other if they are structurally similar (by aligned TM-Score).
We then clustered the proteins again with the Foldseek tool find representative structures. Finally we used KMeans to explore the space and narrow down which protein structures belong where on the protein structure manifold.
'https:/ /ttpss///ii www..ccoo.cccccccccccc......cooc httpbb.ccommassookyyynn////eeee ffusion
30

---

Published as a conference paper at ICLR 2025
FrameDiff Proceus
Avg. of outputs
FrrmeDeff
Figure A1: UMAP visualizations protein structures with (a) SIFER (AND) and (b) averaging of outputs.
FADD KEADOO AAAARO ianrannss
TAT AAAAAA
Figure A2: Protcins generated by SUPERDIFF (AND) with SCTM score < 0.3.
In Fig. AI, we show UMAP visualizations of proteins generated with SUPERDIFF (AND) and averaging of outputs (Liu et al., 2022a).
H.4 PROTEIN NOVELTY EXPLORATION
In Fig. A2, we visualize the proteins generated by SUPERDIFF (AND) that are furthest away from the set of known proteins (sCTM score 0.3).
I SMALL MOLECULE GENERATION
I.1 VISUALIZING TOP-PERFORMING MOLECULES
In Fig. A3, we display the best molecules generated with the objective of GSX3,8 inhibition and QED, We evaluate the performance of molecules according to both objectives by taking the product of the individual objectives. In Fig. A4, we display the best molecules generated with the objective of high GSK3,8 inhibition and JNK3 inhibition.
J GENERATING IMAGE COMPOSITIONS WITH STABLE DIFFUSION
For our concept interpolation experiments, we use aublicly-available pre-trained weights, mod- els, and schedulers from Stable Diffusion v1-4 https: http///ggggngacceeee/ ctt.ac//icc//oppcccscc//c stable-di ffusi cn-v1-4.
In Figs. A5-A24, we show examples of image compositions generated by SUPER (AND), averaging of outputs, and joint prompting. Prompts are shown in each image caption. We show
31

---

Published as a conference paper at ICLR 2025
H2S1
QED:
сaкзр: 0.440
NHz
0.51
HgN
NHH
свкзр:
QED: 0.62
QED:
(a) Molecules generated by SUPERDIFF (AND).
eskI:: 0.32
COKa::
0.78
gEa: 0.58
N NH
05K39: 0.40
8.58
H_N
Nill
HN
NH
HI
HI
05кap: 0.27
QED: 0.80
оokaр: 0.45 0.53
HgN
NH2
(h) Molecules generated by averaging scores.
Figure A3: Best molecules generated by (a) SUPERDIFF (AND) and (b) averaging scores (Liu et al., 2022a) where target properties are high GSK3,8 inhibition and high OED.
o5K3:: 0.69
0вкзр:
3MC: 0.52
ЗNК3: 0.
HO
HyN
H₂N
NO
HN
C1
BSK30:
GSK3р: 0.49
JAK3: 0.54
(a) Molecules generated by SUPERDIFF (AND).
(b) Molecules generated by averaging scores.
Figure A4: Best molecules generated by (a) SIPERDIFF (AND) and (b) averaging scores (Liu et al., 2022a)
where where target properties high G3K33 inhibition and high JNK3 inhibition.
images generated by the first 6 seeds (uniform sampling), well our favourite images generated from 20 seeds. For joint prompting, we generate prompts from two concepts using the linking term "that locks like" For example, given the concepts "a Teman" and "a sun f1 ower". the resulting prompt would be "a sunflower that looks like lemon". We also generate images with the reversed prompt ("a lemon that looks like sunflower"), and keep the images generated by the prompt resulting the higher mean score for each metric. The order the concepts in each image caption reflects the ordering that obtained the higher TIFA score.
J.1 STABLE DIFFUSION WITH SUPERDIFF (OR)
Baselines for concept selection (OR). For the OR setting, our baseline prompting SD that prompts the model select between two concepts: flip the prompt and keep the better image for each metric. As an upper bound, we generate images from SD by prompting it with random choice between the concepts (Prompt oR B).
sunflower or lemon", As with AND, also
32

---

Published as a conference paper at ICLR 2025
Table A2: Quantativate evaluation of 3D-generated images for logical OR. We compare SUFERDIFF (OR), joint prompting. and upper bound randomly selecting a single concept prompt (Prompta on B). report maximum scores from each prompt pair, as well as the absolute difference between the maximum and minimum scores (AA These metries reflect how well a method can select one concept to generate.
M|MM CLIP (1)
[ /Max. ImageReward (f)
|A 712A (1)
Prompi. (uncorvelated random choice)
9.13/29.72
2.87/0.70
86.21/97.58
Joint prompting
7.20/20 80
2.47/0.59
79.46/97.92
SUPER (OR)
8.58 /29.87
2.76/0.64
84.10/95.87
SUPERDIFF qualitatively generates images with better concept interpolations and selections. In Fig. A25, we show examples of image compositions generated by SUPERDIFF (OR), joint prompting, and an upper bound of randomly selecting a prompt of one concept (coin flip). As with SUPER DIFF (AND), keep the prompt order that resulted in higher scores for the baseline. We find that SUPERDIFF (OR) can faithfully generate images with a single concept. The joint prompting baseline can sometimes generate images that combine fragments of both concepts, but other times it also generates images of single concept, typically the first concept in the joint prompt (this also underscores why this method struggles with concept interpolation).
We evaluate SUPERDIFT (OR) using the same three metrics SUPERDIFF (AND) (CLIP Score, ImageReward, and TIFA) and display the results Table A2. again evaluate the image against each concept prompt separately and take both the maximum score and the absolute difference between both scores for each metric. This is so that we can measure how well one concept is represented. The upper bound for this setting is randomly prompting SD with either of the prompts; are almost able to match this setting across all scores, indicating that our method is able to faithfully select single concept. SD with joint prompting does not perform well, as nothing prevents it from combining components from both concepts.
find that
33

---

Published as a conference paper at ICLR 2025
AN AIRPLANE and AN EAGLE first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
AIRPLANE and EAGLE best seeds
SUFERDIFF
Avg. ourputs
Joint prompting
Figure A5; Image compositions generated using SUPERDIFF (AND), averaging of outputs, or joint prompting for the concepts "an airplane" and "an eagle".
34

---

Published as a conference paper at ICLR 2025
BOTTLE CAP and CHESS PAKN
first 6 seeds
SUPERDIFF (AND)
Aug, of outputs
IIHH
Joint prompting
BOTTLE and CHESS PAWN best out of 26 seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A6: Image compositions generated using SUPERDIEF (AND), averaging of outputs, or joint prompting for the concepts "bottle cap" and "chess pawn".
35

---

Published as a conference paper at ICLR 2025
BICYCLE WHEEL and SPIDER WEB
first 6 seeds
SUPERDIFF (AND)
Aug, of outputs
Joint prompting
BICYCLE WHEEL SPIDER of 20 seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A7: Image compositions generated using SUPERDIFF (AND), averaging of outputs, or joint prompting for the concepts "a bicycle wheel* and "a spider web".
36

---

Published as a conference paper at ICLR 2025
CHAIR and AN AVOCADO first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
CHAIR and AN AVOCADO
of 29 seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A8: Image compositions generated using SUPERDIEF (AND), averaging of outputs, or joint prompting for the concepts "a chair" and "an avocado".
37

---

Published as a conference paper at ICLR 2025
A CINNAMON ROLL and A SNAIL first seeds
SUPERDIFF (AKD)
Aug, of outputs
Joint prompting
CINNAMON ROLL SNAIL
best
seeds
SUFERDIFF
Avg. ourputs
Joint prompting
Figure A9: Image compositions generated using SUPERDIFF (AND), averaging of outputs, or joint prompting for the concepts "a cinnamon roll" and "a snail".
38

---

Published as a conference paper at ICLR 2025
COOKIE and MOON
first 6 seeds
SUPERDIFF (AKD)
Aug, of outputs
Joint prompting
COOKIE MOON of 20 seeds
SUFERDUFF (AND)
Avg. ourputs
Joint prompting
Figure A10: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "cookie" and "moon".
39

---

Published as a conference paper at ICLR 2025
A DOG and A CAT first 6 seeds
SUPERDIFF (AKD)
Aug, outputs
Joint prompting
and of 20 seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure All: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a dog" and "a cat".
40

---

Published as a conference paper at ICLR 2025
A DONUT and A MAP
first seeds
SUPERDIFF (AND)
Aug, of outputs
Joint prompting
DONUT and
MAP best
seeds
SUFERDUFF (AND)
Avg. ourputs
Joint prompting
Figure A12: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a donut and "a map".
41

---

Published as a conference paper at ICLR 2025
DUCK and OTTER
first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
DUCK OTTER
best
seeds
SUFERDIFF
Avg. ourputs
Joint prompting
Figure A13: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "duck" " and "otter".
42

---

Published as a conference paper at ICLR 2025
FIREWORKS and DANDELION
first seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
FIREWORKS and DANDELION
best seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A14; Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "fireworks" and "dandelion".
43

---

Published as a conference paper at ICLR 2025
A FLAMINGO and A CANDY CANE
first seeds
SUPERDIFF (AND)
Aug, of outputs
Joint prompting
FLAMINGO and CANDY CANE best seeds
SUFERDUFF (AKD)
Avg. ourputs
Joint prompting
Figure A15: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a flamingo" and "a candy cane".
44

---

Published as a conference paper at ICLR 2025
HELICOPTER and A DRAGONFLY first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
HELICOPTER and DRAGONFLY best of 20 seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A16: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a helicopter" and "a dragonfly".
45

---

Published as a conference paper at ICLR 2025
PEBBLES ON A BEACH and A TURTLE first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
PEBBLES ON BEACH and TURTLE best of seeds
SUFERDUFF
Avg. ourputs
Joint prompting
Figure A17: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "pebbles on beach" and "a turtle".
46

---

Published as a conference paper at ICLR 2025
A PINEAPPLE and A BEEHIVE first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
PINEAPPLE BEEHIVE of 26 seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A18: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a pineapple" and "a beehive".
47

---

Published as a conference paper at ICLR 2025
A ROCKET and A CACTUS first 6 seeds
SUPERDIFF (AKD)
Aug, outputs
Joint prompting
ROCKET and CACTUS best of seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A19; Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a rocket" and "a cactus".
48

---

Published as a conference paper at ICLR 2025
SILHOUETTE OF A DOG and MOUNTAIN LANDSCAPE first 6 seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
SILHOUETTE OF DOG MOUNTAIN LANDSCAPE best of 28 seeds
SUFERDUFF
Avg. ourputs
Joint prompting
Figure A20: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a silhouette of a dog" and "a mountain landscape".
49

---

Published as a conference paper at ICLR 2025
A SUNFLOWER and A LEMON first seeds
SUPERDIFF (AKD)
Aug, outputs
Joint prompting
SUNFLOWER and LEMON best seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A21: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a sunflowers and "a lemon".
50

---

Published as a conference paper at ICLR 2025
TEDDY BEAR and PANDA
first seeds
SUPERDIFF (AKD)
Aug, outputs
Joint prompting
TEDDY BEAR
PANDA
best
seeds
SUFERDIFF
Avg. ourputs
Joint prompting
Figure A22: Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "teddy bear" and "panda".
51

---

Published as a conference paper at ICLR 2025
A WAFFLE CONE and A VOLCANO first seeds
SUPERDIFF (AND)
Aug, outputs
Joint prompting
WAFFLE CONE and VOLCANO best seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A23; Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "a waffle cone" and "a volcano".
52

---

Published as a conference paper at ICLR 2025
ZEBRA and BARCODE
first 6 seeds
SUPERDIFF (AKD)
Aug, of outputs
Joint prompting
ZEBRA BARCODE
best seeds
SUFERDIFF (AND)
Avg. ourputs
Joint prompting
Figure A24; Image compositions generated using SUPERDIFF (AND), averaging outputs, or joint prompting for the concepts "zebra" " and "barcode".
53

---

Published as a conference paper at ICLR 2025
CANDY CANE FLAMINGO first seeds
Promplaue (coin flip)
SuruDire (OR)
Joint prompting
A PINEAPPLE or A BEEHIVE
first 6 seeds
Prompla (coin flip!
SupERDiFF (OR)
Joint prompting
Figure A25: Concept selections using SUPERDIFF joi),joint prompting, and randomly selecting prompt of one concept. The top subfigure shows generated images for the concepts "a candy cane" or "a flamingo". The bottom subtigure shows generated images for the concepts "a pineapple" "a beehive".
54


Reviews:
[{"review_id": "TctjB43Or5", "reviewer": "Reviewer_h88w", "summary": "The authors propose the method to combine the multiple pre-trained diffusion models at the inference time without retraining the models. They come up with the theoretical principles using the continuity equation to show how diffusion models can be viewed in the superposition of elementary vector fields. Here, they implement two algorithms to combine pre-trained diffusion models. One is a mixture of densities (sampling from one model OR another), and the other is equal densities(samples that are likely to belong to one model AND another). They also overcome the challenges with existing diffusion models, such as (1. Differences of Marginal super-positional vector field between different models) and (2. Divergence operation’s time complexity) by introducing their density estimator. They apply their approach in various ways, such as combining models trained on disjoint datasets, concept interpolation in image generation, and improving the structure of protein design.", "questions": "**Major Questions**\n1. I am curious why text-based evaluation metrics such as Clip Score were not used. It seems like an obvious choice to do.\n2. In section 2.1, how were the mixing coefficients $wj$ actually set? Is the model capable of adjusting the weights for mixing? I am also curious about how $N$ for the individual forward process was actually set.\n3. The method overview on page 5 mentions that pre-trained diffusion models can be used, but I am curious if the only one actually used is CIFAR-10, as shown in Table 1. (The experiment by providing the models with CIFAR-10 with two sets of labels divided into five and five) I think if the authors provide the results using the output of various datasets, the paper will be stronger.\n\n**Minor Questions**\n1. I think there should be punctuation after *\"...a superposition of elementary vector fields\"* on page 3, lines 140 and 141.\n2. I think the introduction of the abstract is too long. This could be reduced since the intro occupies 1/3 of the entire amount.\n3. It would have been interesting if there was a comparison according to the distance of the disjoint set.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand. There are almost no grammatical errors. By developing the idea of superposition and using theoretical principles, the authors prove the idea's potential and present a reasonable result.\n2. They apply their work to two individual tasks, which could be divisive among readers, but I found it interesting.\n3. Also, it is interesting that the authors discover their model follows traditional operators such as logical OR and logical AND, making it intuitive. Similarly, the background of explaining how the superposition emerged from the diffusion models by using the vector fields and propositions is interesting.\n4. They use nine propositions, two theorems, and one lemma to support their idea, which helps readers understand why their algorithms work.", "weaknesses": "**(Main) Qualitative Results and the Quantitative Results with figures**\n1. Figure 1 is weak to verify the novelty of the model. I also think the generated images in the appendix, as well as the qualitative results, are mediocre.\n2. The author only uses the AND operation (sampling equal densities) for qualitative results, and OR operation for the quantitative results. I believe that including the results for the OR operation in qualitative results and the AND operation in quantitative results would strengthen the paper. This would provide a more comprehensive view of the statement on line 104 on page 2: \"improvements in designability and novelty generation\".\n3. Figure 2 does not show how the generated images are actually arranged. It is necessary to verify if the same results occur when directly arranging the generated images with the trained datasets.\n\n**Evaluation metrics and ablation study**\n1. The comparative group for the paper's qualitative results is insufficient. Comparisons with other recent models that produce dual images, such as factorization diffusion or visual anagram (Geng et al.,2024), could be added. Since it is clear that the latent diffusion result for just adding the prompt 'that looks like' would indeed be worse than the proposed method.\n2. Similarly, in the process of making baselines for concept interpolation, I wonder if the value of the ablation study would have increased if the direction of A->B and B->A was changed and the comparison group was chosen by using the better result.\n3. The execution times for the experiments were not provided. The authors claim to have solved the computational expense issue, but no results support this claim.\n\n**Clarity of the paper**\n1. Proposition 8 appears to be quite important but confusing because it was cut off the page. Listing the individual terms of $Aκ = b + o(Δt)$ on the same page would improve comprehension.\n2. The related work section comes out almost at the end of page 10, and I think this part should come out more front. It comes out so out of the blue that it somewhat interferes with understanding.\n3. The protein generation part is not clearly introduced. The authors compare Designability, Novelty, and Diversity, and there is no separate explanation of how this part is meaningful in protein generation. I didn't feel that logic was connected smoothly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pB62FvoBfS", "reviewer": "Reviewer_DUd3", "summary": "This paper proposes a novel algorithm for combining multiple pre-trained diffusion models at inference time, by the principle of superposition of vector fields. The method demonstrates more diverse generation results, better prompt following on image data, and improved structure design of proteins as well.", "questions": "* Why are there no quantitative results on SD, and detailed discussion of other very relevant methods as referenced earlier?\n* FID statistics on CIFAR-10 are computed on the whole dataset. Is it fair to evaluate models trained on a partial dataset using such statistics, especially when the two partitions are generated by splitting the classes?\n* What are the practical implications of the OR operator, especially in the field of image generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The theoretical framework is solid.\n* The method is well-motivated and supported by the theory.\n* The method is training-free, and could be applied to diffusion models with different architectures.\n* The results of protein generation outperform other baselines.", "weaknesses": "* The practical implications of AND, and OR operators are not explained clearly in both image and protein generation settings. What effect will the OR operator create on images, compared to the AND operator?\n* Lacks quantitative results on SD. Could have used metrics such as TIFA Score [1]  and Image Reward [2]. I wonder if there is any reason that no such metric was used. \n* Lacks comparison against other relevant methods [3-6]. In particular, [3,4,6] are all inference-time methods that sample from some sort of mixture of scores and demonstrate multiple practical uses, such as composing objects, styles, scenes, or improving text-image alignment. Need more discussions on the capabilities of the proposed method versus others: besides the different theoretical perspectives, how SUPERDIFF performs differently, the strengths and weaknesses of SUPERDIFF than the other methods. If experiments are not possible, please include a more detailed discussion. The comparison could help readers understand the proposed method in a broader context. \n\n[1] Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 (2023)\n\n[2] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36 (2024)\n\n[3] Du, Y., Durkan, C., Strudel, R., Tenenbaum, J.B., Dieleman, S., Fergus, R., SohlDickstein, J., Doucet, A., Grathwohl, W.S.: Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In: International conference on machine learning. pp. 8489–8510. PMLR (2023)\n\n[4] Golatkar, A., Achille, A., Swaminathan, A., Soatto, S.: Training data protection with compositional diffusion models. arXiv preprint arXiv:2308.01937 (2023)\n\n[5] Biggs, Benjamin, et al. \"Diffusion Soup: Model Merging for Text-to-Image Diffusion Models.\" arXiv preprint arXiv:2406.08431 (2024).\n\n[6] Liu, Nan, et al. \"Compositional visual generation with composable diffusion models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTR2gGin8P", "reviewer": "Reviewer_1WGb", "summary": "This paper introduces a novel, principled, and efficient way to combine diffusion models trained on different datasets (or conditioned on different prompts) to generate images from the mixture and the \"intersection\" of the corresponding distributions. It is based on a clever way to evaluate the densities $\\log p^i_t(x_t)$ of the current iterate $x_t$ under each (noisy) distribution $q^i_t$ during synthesis.", "questions": "- How do the authors explain the source of their numerical improvements using SuperDiff OR?\n- What density is being sampled from when using SuperDiff AND?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The main strength of the paper is the important observation that the probability density function of generated images can be efficiently evaluated without the need for computing the divergence of the score. It is leveraged to sample from mixtures of densities, where the weights can be defined implicitly and adaptively (in the case of the logical AND operator as defined here). The experimental results convincingly demonstrate the effectiveness of the resulting approach.", "weaknesses": "In my opinion, the main weakness of the paper is in the clarity of the presentation of the central theoretical result (culminating in Proposition 7) and the motivation for the approach. I believe it can be significantly improved, which could enhance the impact of the paper. \n- I found section 2.1 to be unnecessary complicated and rather irrelevant for the rest of the exposition. To my understanding, the main ideas are (1) that SDEs define linear equations on the densities, so that a mixture of clean distributions $\\sum w_i p^i$ leads to a mixture of noisy distributions $\\sum w_i p_t^i$ and (2) the relationship $\\nabla \\log (\\sum w_i p^i_t) = \\sum w_i p^i_t \\nabla \\log p^i_t / \\sum w_i p^i_t$. These motivate the need for evaluating $p^i_t$ to combine scores in the correct way to sample from mixtures.\n- The equations are obscured by the use of general schedules with arbitrary $\\alpha_t$ and $\\sigma^2_t$. I encourage the authors to state the results in the main text with e.g. $\\alpha_t = 1$ and $\\sigma_2^t$ (known as the variance exploding SDE) to simplify the exposition and relegate the general case to the appendix. \n- Some results are also less intuitive (in my opinion) due to the choice to work in discrete time. For example, Proposition 6 and Theorem 1 are nothing but approximating the kernels $k_{\\Delta t}$ and $r_{\\Delta t}$ with Euler-Maruyama discretizations of the corresponding forward or backward SDEs (and analyzing the discretization error in Theorem 2). Similarly, Proposition 7 can be obtained in continuous time first (and then discretized) by applying Itô's formula to $\\log q_t(x_t)$ where $x_t$ is a solution of the backward SDE (and using the fact that $q_t$ solves a Fokker-Planck equation). As an example, in the variance-exploding case, one obtains that $\\mathrm{d} \\log q_t(x_t) = \\frac{\\mathrm{d}t}2 ||\\nabla \\log q_t(x_t)||^2 + \\langle \\mathrm{d}x_t, \\nabla \\log q_t(x_t)\\rangle$, which is the $\\Delta t \\to 0$ limit of Proposition 7 with $\\alpha_t = 1$ and $\\sigma^2_t = t$. I believe this result to be of independent interest, and would thus benefit from being highlighted and stated as simply as possible.\n\nAnother issue I have is regarding the logical OR and AND operators as defined in this paper.\n- The logical OR operator corresponds to a fixed-weight mixture of distributions, and it is thus trivial to sample from. One can simply select one diffusion model with probability corresponding to the mixture weight, and then use exclusively the score of the chosen diffusion model during generation. Using SuperDiff should be equivalent to this algorithm. So either the improved results in section 4 can also be achieved with this simple baseline, in which case the theoretical results are not needed, or the baseline underperforms, in which case the improvements come from unknown implementation choices which are completely orthogonal from the theoretical analysis. In both cases, this raises questions.\n- The real strength of the approach, I think, is when the mixture weights are adaptive (i.e., they are allowed to depend on the current iterate $x_t$). In that case, however, it is not clear what density we are ultimately sampling from. If I understand correctly, here the logical AND operator is defined implicitly, and produces samples $x$ such that $q^1(x) = q^2(x)$. A perhaps more usual definition is that one would aim to sample from the normalized product $q^1(x)q^2(x)/Z$ (or geometric mean $\\sqrt{q^1(x)q^2(x)}/Z$), but this seems difficult to achieve with the formalism of this paper. It could be beneficial to include a short discussion of this matter in the paper.\n\nFinally, I could not see where the parameters $\\omega$ and $T$ in Table 2 were explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)