Compositional Sculpting of Iterative Generative Processes
Authors: Timur Garipov
Sebastiaan De Peuter
Ge Yang
Vikas Garg
Samuel Kaski
Tommi S. Jaakkola
Venue: NeurIPS 2023 poster (2023)
Keywords: generative model composition
GFlowNets
diffusion models
classifier guidance
probabilistic methods
TL;DR: We propose a framework to compose iterative generative processes: GFlowNets and diffusion models.

Abstract: High training costs of generative models and the need to fine-tune them for specific tasks have created a strong interest in model reuse and composition.
A key challenge in composing iterative generative processes, such as GFlowNets and diffusion models, is that to realize the desired target distribution, all steps of the generative process need to be coordinated, and satisfy delicate balance conditions.
In this work, we propose Compositional Sculpting: a general approach for defining compositions of iterative generative processes. We then introduce a method for sampling from these compositions built on classifier guidance.
We showcase ways to accomplish compositional sculpting in both GFlowNets and diffusion models. We highlight two binary operations $\\unicode{x2014}$ the $\\textit{harmonic mean}\\unicode{x00A0}(p_1 \\otimes p_2$) and the $\\textit{contrast}\\unicode{x00A0}(p_1 \\,\\unicode{x25D1}\\,\\, p_2$) between pairs, and the generalization of these operations to multiple component distributions.
We offer empirical results on image and molecular generation tasks. Project codebase: https://github.com/timgaripov/compositional-sculpting.

Compositional Sculpting of Iterative Generative Processes
Timur Garipov I+
Sebastiaan De Peuter?
Ge Yang!. Tommi Jaakkola'
Vikas Garg25
Samuel Kaski2.3
'MIT CSAIL
Aalto University University of Manchester
Inntittte for Artificial Intelligence and Fundamental Interactions YaiiYai Ltd
Abstract
High training costs of generative models and the need to fine-tune them for specific tasks have created a strong interest in model reuse and composition. A key challenge in composing iterative generative processes, such as GFlow Nets and diffusion mod- els, is that realize the desired target distribution, all steps of the generative process need to be coordinated, and satisfy delicate balance conditions, In this work, we propose Compositional Sculpting: a general approach for defining compositions of iterative generative processes. We then introduce a method for sampling from these compositions built on classifier guidance. We showcase ways to accomplish compo- sitional sculpting in both GelowNets and diffusson models. We highlight two binary operations the harmonic mean (p) & P2) and the contrast (p1 O p2) between pairs, and the generalization of these operations to multiple component distributions. We offer empirical results on image and molecular generation tasks. Project codebase: https: //g thub. com/ Lingaripov. composi iatestiiona teioallssculptti.......ee. sculpting.
Introduction
Large-scale general-purpose pre-training of machine learning models has produced impressive results in computer vision [1-3]. image generation [4-6], natural language processing [7-11], robotics [12- 14], and basic sciences [15]. By distilling vast amounts of data, such models produce powerful inferences that lead to emergent capabilities beyond the specified training objective [16]. However, generic pre-trained models
often insufficient for specialized tasks in engineering and basic sciences. Field-ace field-adaapattin via techniques such as explicit fine-tuning on bespoke datasets [17], human feedback [18], or cleverly designed prompts [19, 20] is therefore often required. Alternatively, capabilities of pre-trained models can be utilized and extended via model composition.
Compositional generation [21-28] views complex target distribution in terms of simpler pre-trained building blocks which can be mixed and matched into a tailored solution a specialized task. Given set of base models capturing different properties of the data, composition provides way to fuse these models into a single composite model with capacity greater than any individual base model. In this way it allows one to specify distributions over examples that exhibit multiple desired properties simultaneously [22]. The need to construct complex distributions adhering to multiple constraints arises in numerous practical multi-objective design problems such molecule generation [29-31]. In this context, compositional modeling provides mechanisms for control of the resulting distribution and exploration of different trade-off's between the objectives and constraints.
Prior work on generative model composition [21, 22, 24] has developed operations for piecing together Energy-Based Models (EBMs) via algebraic manipulations of their energy functions. For example, consider two distributions p1 (x)x and (xtx))xxx(pEE(5. induced by energy
"Correspondence Timur Garipov (timur@csail.. edu.
37th Conference on Neural Information Processing Systems (NeurTPS 2023).

---

(a) P1
(b)
(c) P; @
(d) P1 1
(c) Pj ! Pz
Figure I: Composition operators. (a.b) base distributions P1 and p2. (c) harmonic mean of P and P2- (d) contrast of #1 with P2 (e) reverse contrast P1 P2- Lines show the contours of the PDF level sets.
functions E and Ez. Their product (x) a P1 (x) exp(-(E,(x)) E2(x)) and negation Pneg (x)) a (x)) energy functions. The product assigns high likelihood to points x that have high likelihood under both base distributions but assigns low likelihood points that have close-to- one (or both). The negation distribution assigns high likelihood to points that are likely under p1 but unlikely under P2 and assigns low likelihood to points that are likely under P2 but unlikely under P1-
0 correspond to operations on the underlying
likelihood under
Iterative generative processes including diffusion models [5, 32-34] and GFlowNets [35, 36] progres- sively refine coarse objects into cleaner ones over multiple steps. Realizing effective compositions of these models is complicated by the fact that simple alterations in their generation processes result in non-trivial changes in the distributions of the final objects. Unlike for EBMs, products and negations of diffusion models cannot be realized through simple algebraic operations on their score functions.
Du al. [28] show that the result of the addition of score functions is not equal to the score of the diffused product distribution and develop a method that corrects the sum-of-scores sampling via additional MCMC steps nested under each step of the diffusion time loop.
Jain et al. [31] develop Mujei-O GelowNets (MOGFNs), an extension of GFlowNets for multi-objective optimization tasks. While vanilla GelowNet captures distribution induced by single reward (objective) function p(x)x R(x) (see Section for details), an MOGFN aims to learn single conditional model that realize distributions corresponding to various combinations (e.g. convex combination) of multiple reward functions. Though a single MOGFN realizes a spectrum of compositions of base reward functions, the approach assumes access to the base rewards at training time. Moreover, MOFGNs require the set of possible composition operations to be specified at training time. In this work, address post hoc composition of pre-trained GelowNets (or diffusion models) and provide a way to create compositions that need not be specified in advance.
In this work, introduce Compositional Sculpting, general approach for the composition of pre- trained models. We highlight two special examples binary operations - harmonic mean: (p) @ p2) and contrast: (P) O P2). More general compositions are obtained as conditional distributions in probabilistic model constructed on top of pre-trained base models. We show that these operations can be realized via classifier guidance. We provide results of empirical verification of our method on molccular generation (with GFlowNets) and image generation (with diffusion models).
2 Background
Generative flow networks (GFlowNets). GFlowNets [35, 36] are an approach for generating structured objects (c.g. graphs) from a discrete space X. Given a "reward function" R(x) 2 O, GFlowNet seeks to sample from p(x) Rxx/z, where Z Σ, R(x), i.c. the model assigns larger probabilities to high-reward objects.
Starting at fixed initial state Sp, objects are generated through sequence of changes corresponding S, x). The structure of possible
trajectory of incomplete states t (sp - 5 + trajectories corresponds to by DAG (S, A) where S is a set of states (both complete and incomplete) and A the set of directed edges (actions) s s'. The set of complete objects (terminal states) X subset of S. The generation process starts at sg and follows a parameterized stochastic "forward policy" Pp |s; 0) which for each state E \ X specifies probability distribution over all possible successor states s' (s s') E A. The process terminates once terminal state is reached.

---

Diffusion models. Diffusion models 32-34, 37] are family of generative models developed for continuous domains. Given an empirical (data) distribution 1(x)=1/n Σ, 8z, (x) in X' Rd, diffusion models seek to approximate M(x) via a generative process p(x).
diffusion process is a noising process that gradually destroys the original "clean" data x. Viewed as a stochastic differential equation (SDE) [34], it is time-indexed collection of random variables {x in X' = R which interpolates between the data distribution Po(x) = p(x) at = 0 and a prior distribution px(x) at I = T. The evolution of is described by the "forward SDE" dx, = f((x)) + & du, with drift coefficient f, : Rd - Rd and diffusion coefficient & E R. Here, wy is the standard Wiener process. Crucially, the coefficietss fi, & are generally chosen such that the prior Pr and the transition probabilities (x, 0 S s < f I T have a closed form (see [34]).
Song et al. [34] invoke a result from the theory of stochastic processes [38] which gives the expression for the reverse-time process or "backward SDE" dx, = [f,(x,) - z²V,
x log p,(xx)) di + 8, wheee we is the standard Wiener process in reversed time. This SDE involves the known coefficients f. & and the unknown score function V₂ log p,()) of the marginal distribution p-(-) time t. A "score network" sy(x; 0) (a deep neural network with parameters 0) is trained to approximate V. log (x). Once s,(:0) is trained, sampling reduces to numerical integration of the backward SDE.
Classifier guidance in diffusion models. Classifier guidance [32, 39] is technique for con- trollable generation in diffusion models. Suppose that each example Xp is accompanied by a discrete class label The goal is to sample from the conditional distribution (xpX)) The Bayes rule py( & (x(xx) implies the score-function decomposition V., log (,(x,ly) = Vx, log (x,) + x,, log (r(xx, where the first term is already approximated by a pre-trained uncon- ditional diffusion model and the second term can be derived from a denendepnnd classifier p,())x,) Therefore, the stated goal can be achieved by first training the classifier p, p,(Y|x,, using noisy samples x, from the intermediate steps of the process, and then plugging in the expression for the conditional score into the backward SDE sampling process [34].
3 Related Work
Generative model composition. In Section we reviewed prior work on energy-basee composition operations and Aulti--Ojecctive GFlowNett. Learning mixtures of Generative Adversarial Networks has been addressed in [40]. where the mixture components are learned simultaneously, and in [41], where the components are learned one by one in an adaptive boosting fashion. Algorithms for additive and multiplicative boosting of generative models have been developed in [42].
This work focuses on the composition of pre-trained models. Assuming that each pre-trained model represents the distribution of examples demonstrating certain concepts (e.g. molecular properties). the composition of models is equivalent to concept composition (e.g. property "A" AND property "B"). The inverse problem is known as "unsupervised concept discovery". where the goal is to automatically discover composable concepts from data, Unsupervised concept discovery and concept composition methods have been proposed for energy-based models [25] and text-to-image diffusion models [43].
Controllable generation. Generative model composition is a form of post-training control of the generation process an established area of research. A simple approach to control is training a conditional generative model p(x[c) on pairs (x, c) of objects x and conditioning information c. Annotations can be class labels [39], text prompts [4, 6, 44], semantic maps, and images [4]. Different from out work, this assumes that generation control operations are specified at training time. Dhariwal and Nichol [39] apply classifier guidance [32] on top of (conditional) diffusson models to improve their fidelity. Ho and Salimans [45] develop classificr free guidance by combining conditional and unconditional score functions. In ControlNet [17], an additional network trained to enable a -reitraeeed diffusion model to incorporate previously unavailable conditioning information. Meng et al. [46] and Couairon et al. [47] develop semantic image editing methods which first partially noise and then denoise an image to generate an edited version, possibly conditioned on a segmentation mask [47]. Similar to conditional diffusion models, conditional GFlowNets have been used to condition generation reward exponents [36] or combinations of multiple predefined reward functions [3 I]. Note that the methods developed in this work can be combined with conditional diffusion models and GFlowNets: p(xlc;).... .... p(x(cmp) can act base generative models to be composed.
3

---

Compasitional generalization. The notion of compositionality has broad spectrum of inter- pretations across variety of disciplines including linguistics, cognitive science, and philosophy. Hupkes et al. [48] collect list of aspects of compositionality from linguistic and philosophical the- ories and design practical tests for neural language models. Conwell and Ullman [49] empirically examine the relational understanding of DALL-E 2 [50], text-guided image generation model, and point out limitations in the model's ability to capture relations such as "in", "on", "hanging over", etc. In this work, we focus on a narrow but well-defined type of composition where we seek to algebraically compose probability densities in a controllable fashion, such that we can emphasize or de-emphasize regions in the data space where specific base distributions have high density.
Connections between GFlowNets and diffusion models. Our method is applicable to compositions of both GFlowNets and diffusion models. This is due to deep connections between these two model families. GelowNets were initially developed for generating discrete (structured) data [36] and diffusion models were initially developed for continuous data [5, 32]. Lahlou et al. [51] develop an extension of GFlowNets for DAGs with continuous state-action spaces. Zhang et al. [52] point out unifying connections between GelowNets and other generative model families, including diffusion models. In this work, we articulate another aspect of the relation between GFlowNett and diffusion models: in Section 5.2 derive the expressions for mixture GFlowNet policies and classifier-guii -uuided GFlowNet policies analogous to those derived for diffusion models in [32, 39, 53, 54].
4 Compositional Sculpting of Generative Models
Suppose can access number of pre- trained generative models (p,(x)),,, over a common domain X.. We may wish to compose these distributions such that we can, say, draw samples that are likely to arise from pl (x) and (x),), or that are likely to arise from P (x) but not from (x(x). In other words, we wish to specify a distribution that we can shape to emphasize and de-emphasize specific base models.
4.1 Binary Composition Operations
Let us first focus on composing two base models. We could specify the composition as a weighted sum p(x)= Ei= (x), with weights Wn Φ, 2 0 summing to one. The weights determine the prevalence of each base model in the composition, but beyond that our control is limited. We cannot emphasize regions where P1 and P2 both have high density, or de-emphasize regions where P3 has high density.
An alternative is to use conditioning to shape a prior p(x) based on the base models. When we condition x on some observation y1, the resulting posterior takes the form p(xly)) x | Points x that match y1 according to p(y, will have increased density, and the density of points that do not match it decreases. Intuitively, by defining F1 E {1,2) as the event that x was generated by a specific base model, we shape prior p(x) based on the densities of the base models. To this end we define a uniform prior over 84 and define the conditional density p(xly) = i) to represent the fact that x was generated from (x))) This gives the following model:
xxxly; =)) = p,(x), pl1 -) = 1/2, p(x) = p(xly; 1) =) + p(xly 2) 2)..
(1)
Under this model, the prior p(x) is uniform mixture of the base models. The likelihood of y1 アor 1[x)
- 21 (px) + P2(x)),
implied by this model tells us how likely it is that x was generated by p (x) rather than (x)... In fact, it corresponds to the output of optimal classifier trained to tell p and p2(x) apart.
Our goal to realize compositions which generate samples likely to arise from both p (x) and p2(x) or from P₁ (x) but not (x). Thus we introduce a second observation y E [1,2] such that y, and y/2 are independent and identically distributed given x. The resulting model and inferred posterior are:
I x)), p(x)= = _ [x((x) + _22(x) pox =|x) p,(x) P2(x)) (3) k=1
1
p(x. F2, p(x)
(xx)) The above posterior shows clearly how conditioning on observations y1 أ // has shaped the prior mixture to accentuate regions in the posterior where the observed base models i. have high density.
7xl,y
p = = =
(4)

---

Conditioning on observations = and y22 =2, or equivalenty y1 y2=1 == results in the posterior (p p2)(x) p(xly; = y2 =2) a Pl (x(x)px(x)/((pxx) + P2(x)),
(5) We refer to this posterior as the "harmonic mean of p: and p2"; and denote it as binary operation Pi Pz. Its value is high only points that have high likelihood under both (x(x) and P2(x) the same time (Figure (c)). Thus, the harmonic mean is an alternative to the product operation for EBMs. The harmonic mean is commutative (p) @ Pz = P2 8 p1) and is undefined when P1 and Pr have disjoint supports, since then the RHS of (5) is zero everywhere.
Conditioning on observations y1 and y/2 results in the posterior
(6) We refer this binary operation, providing an alternative to the negation operation in EBMs, as the "contrast of Pj and P2". and denote it as P1 1 P2- The ratio (6) is high when P1 (x) is high and p2(x) is low (Figure (d)). The contrast not commutative (pt O P2 # P2 P1. unless P1 = P2). We denote the reverse contrast as Pj O P2 Pz O P1- Appendix C provides a detailed comparison between the contrast and negation operations, and between the harmonic mean and product operations.
(P) p2)(x) := p(x y2 = 1) (x)( (x) + P2(x)).
Controlling the individual contributions of Pj and P2 to the composition. In order to provide more control over the extent of individual contributions of P1 and p2 the composition,
modify model (3). Specifically, we introduce an interpolation parameter a and change the likelihood of (x))(()))) / (ap, (x) + (1 ax)x(x)), observation y/2 in (3): ply2 = l|x; a) (x)) ((1
where a E (0, 1) and [-] denotes the indicator function. Conditional distributions in this model give harmonic interpolation and parameterized contrast:
(x) (pi (11 P2)xx up (x) + - -x) (P) O (1-a) (x) ap)(x) + (1 pxp2(x)
(x)(x))
Operation chaining. As the operations we have introduced result in proper distributions, we can create new N-ary operations by chaining binary (and N-ary) operations together. For instance, chaining binary harmonic means gives the harmonic mean of three distributions
(xx)))((x));(x1
(e @ P2) pz)(x) (p) @ (p2 & px))(x)
(8)
P1(0)))((80++ + p2(x)p3(x)
4.2 Compositional Sculpting: General Approach
The above approach for realizing compositions of two base models can be generalized compositions of m base models Pı (x)..... (x) controlled by n observations. Though operator chaining can also realize compositions of m base models, our generalized method allows us specify compositions more flexibly and results in different compositions. We introduce an augmented probabilistic model p(x, y1. ................................ , Fn)) as a joint distribution over the original objects x X' and observation variables E y , Fn, E Y where y [L.... m) By defining appropriate conditionals pfy.1x) we can controllably shape prior p(x) into a posterior p(xly)....... y,))
As in the binary case, we propose to use a mmiformly-weighter mixture of the base models p(x) (/m)- Σ (x)). The support of this mixture is the union of the supports of the base models: U (x)) supp( This is essential the prior can only be shaped in places where has non-zero density. As before we define the conditionals p(y; x])x to correspond to the observation that x was generated by base model i. The resulting full model is
I Plrlx), = 1 Σ (x)x,, Fly =))= Pl,x =||x= A=1
m
p,(x)
Mx, .1 ((x)
(9)
Σ DJ (x)
Note that under this model the mixture can be represented as p(x) Σ
for any
k.
The inferred posterior over x for this model is
p(xly) = Vn -, p(x) = ... yn ix(x)
(10)
//(// P,(x)
n-1
p Il ply = xx)x) &
i Pig (x)
(11)
"the harmonic interpolation approaches P, when O and P3 when
5

---

The posterior p -I..... J,, -, (2) is composition of distributions [p,(x)) that can be adjusted by choosing values for y1..... yn- By adding or omitting an observation y;: = i we can sculpt the posterior to our liking. emphasizing or de-emphasizing regions X' where P; has high density. The observations can be introduced with multiplicities (e.g., = 1, = y, = 2) to further strengthen the effect. Moreover, one can choose to introduce all observations simultancously as in (10) or sequentially as in (11). As we show below (Section 5.1 for GFlowNets; Appendix A.2 for diffusion models), the composition (10) can be realized by a sampling policy that can be expressed as function of the pre- rreitrai (base) sampling policies.
Special instances and general formulation. The general approach outlined in this section is not lim- ited to choices we made to construct the model in equation (9), i.e. p(x) does not have to be uniformly weighted mixture of the base distributions, }1, .1.....3. y), do not have to be independent and identically distributed given x, and different choices of the likelihood p(y=ilx) are possible. For instance, in the model for parameterized operations (7) the likelihoods of observations p(y]lx), p(y2/x) differ.
5 Compositional Sculpting of Iterative Generative Processes
In this Section, we show how to apply the model above to compose GFlowNets, and how one can use classifier guidance to sample from the composition. The similar method for diffusion model composition is described in Appendix A.2.
5.1 Composition of GFlowNets
Besides a sample x from (x),), a GFlowNet also generates trajectory t which ends in the state x. Thus, we extend the model p(x, V1, Fn), described above, and introduce as a variable with conditional distribution p(rly; =) - 18,,), where Pt,F the forward policy of the GFlowNet that samples from P:-
Our approach for sampling from the composition is conceptually simple. Given m base GelowNets that sample from PI..... Pm respectively, start by defining the prior p(x) the uniform mixture of these GelowNets. Proposition 5.1 shows that this mixture can be realized by policy constructed from the forward policies of the base GFlowNets. We then apply classifier guidance to this mixture to sample from the composition. Proposition 5.2 shows that classifier guidance results in a new policy which can be constructed directly from the GFlowNet being guided.
Proposition 5.1 (GFlowNet mixture policy). Suppose distributions P (x)......PPP (m)(x) are realized by GFlowNets with forward policies Pı,r(-1)),.. ......... Pm..(.l.). Then, the mixture distribution Pw(x) =
Em (X) with .I.... 0m 2 0 and Σm 002 = is realized by the GFlowNet forward policy
PM. (s) = Σ ply ((s)p,
(12)
where y is a random variable such that the joint distribution of a GFlowNet trajectory t and y is given by p(t, y= = (,P,(t) for m))
Proposition 5.2 (GFlowNet classificr guidance). Consider a joint distribution p(x. y) over a discrete space X such that the marginal p(x) is realized by a GFlowNet with forward policy Pr Assume that the joint distribution of x, y, and GFlowNet trajectories t = (Sg... =xx) decomposes as p(I, x, y) p(, x))() i.e. y is independent of the intermediate states { s-11 in v given x. Then,
1. For all non-terminal nodes s E ESS S\2, the probabilities plyls) satisfy
p(rls)
Σ
P=(s'ss)p(sss').
(13)
d'(s-SsJEA
2. The conditional distribution p(xly) is realized by the classifier-guided policy Pr(s'1s, (s.slss,))) y) Pr(s'(s) polss)/pssss/))))))))))))))))p plyss).
(14)
Note that (13) ensures that Ee ps( 1.

---

Proposition 5.1 is analogous to results on mixtures of diffusion models (Theorem 1 of Peluchetti [53], Theorem 1 of Lipman et al. [54]). Proposition 5.2 analogous to classifier guidance for diffusion models 39). the best of our knowledge, our work is the first to derive both results for GetowWets.
Theorem 5.3 summarizes our approach. The propositions and the theorem are proved in Appendix D. Theorem 5.3. Suppose distributions p (x.,.... Pm) are realized by GFlowNets with forward policies PL,(C)), ... Pm,r((l-) respectively. Let 'n be random variables defined by (9). Then, the conditional p(x|y1,... ........ .Ya) is realized by the forward policy
ply ... -ynls')
m
Pr(d'ss,y yn) ply Fn1s)
Σ
Pr. (1. i=1
(15)
Note that the result of conditioning on observations yI.....,,, Yn is just another GFlowNet policy. Therefore, to condition more observations, can apply classifier guidance repeatedly.
5.2 Classifier Training (GFlowNets)
The evaluation of policy (15) requires knowledge of the probabilities ply..... y,ls), p(r(s), These probabilities can be estimated by a classifier fitted to trajectories sampled from the base GFlowNets.
Let (......... sy be a classifier with parameters & that we wish to train to approximate the ground- truth conditional p......, y,ls). Under model (9), ,', are dependent given a state E S \ a' but, are independent given terminal state x E X.. This motivates separate treatment of terminal and non-terminal states.
Learning the terminal state classifier. For a terminal state x, the variables .y..... are inde- pendent, hence
the factorization 2₂00 ),Ix) I (x(x)) Moreover, all distributions on the r.h.s. must be the same, i.e. is enough to learn just 2 (v(vixx). This marginal classifier can be learned by minimizing the cross-entropy loss
LT(() (e)))
E
log -
(16)
Here 9i is sampled from ply)). which is uniform under choice of p(x). Then, 8101 P1) is generated from the base GelowNet pg,, since (9) implies that pxxyy= = Pÿj (x).
Learning the non-terminal state classifier. Given non-terminal state E S \x, we need to model JI..... y, jointly, and training requires sampling tuples (3, 8 .... P,). Non-terminal states
can be generated intermediate states trajectories r = (5g 51 + x). Given a sampled
trajectory and a set of labels Di..... Pn we denote the trajectory cross-entropy by
|r|
e(t, DT. Φ) = Σ
y, = log T0('1 -............................... [-
,s
(17)
Pairs (f, (F. can be generated in the same way as in the terminal classifier training above: 1) f ~ ply)): 2) f Pý, (r). Sampling ...... Fn given & (the terminal state of ?) requires access values Plr- = ), which are not directly available. However, the terminal classifier learned as described above, the estimates w, wx () C = -|x= = X) can be used instead. In this case, the loss and the sampling procedure for the non-terminal classifier rely on the outputs of the terminal classifier. In order to train two classifiers simultaneously. and avoid the instability due to feedback loop, we employ the "target network" technique developed in the context of deep Qelarnning [55]. We introduce a "target network" parameter vector & which is used to produce the estimates w/ +)) for the non-terminal loss. We update o as the exponential moving average of the recent iterates of Φb
After putting all components together the training loss for the non-terminal state classifier is
E Σ ... Σ F
Aw 1., (2: (2: C C(F. F. )
P,i
(18)
(.).)))()))))
We refer the reader to Appendix D.4 for a more detailed derivation of the loss (18). Algorithm A.1 shows the complete classifier training procedure.
7

---

LoNe
Pi
P2
Pi Pz
Pt P2
Pt
0.95 Pz
i 8
Pi
P2
P3
Figure 2: Composed GFlowNets on 32 32 grid domain. (Top) operations two distributions. (Bottom) operations on three distributions. Cell probabilities are shown with color, darker is higher. Red circles indicate the high probability regions of P1. P2. P3-
04
0.1
01
01
35
065
2.6
155
0.5
022
05
10
54
044
DA 10
14
03
1.
02
04 81
10
(a) Base at / = 32 (b) Harmonic mean
(c) Contrasts
(d) Base at 96
(c) Harmonic mean
Figure 3: Reward distributions in the molecular generation domain. (a) Base GelowNets at
=232: PSEH and PSA are trained with (x) and (x) (b) harmonic mean of PSER and PSA
at 32. (c) contrasts at = (d) base GFlowNets at 9= (e) harmonic mean at 86 Lines
show the contours of the level sets of the kernel density estimates the (RSEH. RSA) plane.
6 Experiments
2D distributions via GFlowNet. We tested our GFlowNet composition method on 2D grid [36], controlled domain, where the ground-truth composite distributions can be evaluated directly.
In the 2D grid domain, the states are the cells of an H X H grid. The starting state is the upper-left cell so (0, 0). At each state, the allowed actions are: 1) move right; 2) move down; 3) terminate the trajectory at the current position. We first trained GelowNets p,(x) o R,(x) with reward functions (x) > 0, and then trained classifiers and constructed compositions following Theorem 5.3.
Figure (top row) shows the distributions obtained by composing two pre-trained GelowNets (top row; left). The harmonic mean P: @ P2. covers the regions that have high probability under both Pj and P2 and excludes locations where either of the probabilities is low. P1 1 P2 resembles PI but the relative masses of the modes of Pj are modulated by P2: regions with high have lower probability under contrast. The parameterized contrast P1 0.95 P2 with 0.05 magnifies the contrasting effect: high (x) implies very low (p) O 0.95 (x).). The bottom row of Figure 2 shows operations on distributions. The conditional p(xly; = )2=2) likelihood under both pl and P2. Similarly, the value =(x 22 y3=3) is high if x is likely to be observed under all three distributions the same time. The conditionals =(x y2=2) and = 22 2) highlight the points with high (x) but low P; (x) and (x). Conditioning on three labels results a sharper distribution compared to couble-conditioning We provide quantitative results and further details in Appendix F.I. The classifier learning curves are provided in Appendix G.4.
concentrated on the points that have high
Next, we evaluated our method for GFlowNet composition on a large and highly structured data space, and assessed the effect that composition operations have
Molecule generation via GelowNet.
8

---

Table 1: Reward distributions of composite GetowNets
18
(f)
SHH
high
SA
low
high
low
high
PSA
QED
low
high
low
high
low
high
low
high
PSEn
0
0
0
0
62
9
24
PSA
0
73
0
0
18
PQED
0
40
26
0
21
13
(e) PSFH
(0)
-0.
(a) y= [SBH,
1
0
16
3
54
18
-03 3.1
0.5
1D
(h) y= [SMM, QED]
0
11
0
4
1
48
4
32
(6) y SA QUD]
0
15
8
Figure 4: 2D t-SNE embed-
(d) y=
QED
0
7
19
10
49
dings of three base GFlowNets trained with Rx) (x) (x) at P = 32 and their compositions, The t-SNE embed- dings are computed based on pair- wise earth mover's 's distances be- tween the distributions. Labels (a)-(g) match rows in Table 1.
(e) [SE,, SEH, SEH
0
0
0
63
9
24
(1) y= SA, SA
0
0
74
0
01
17
(g) y= [QED, QED, QED)
23
In each row, the numbers show the percentage of the samples from the respective model that fall into one of 8 bins according to rewards. The "low" and "high" categories are decided by thresholding SEH: 0.5, SA: 0.6, QED: 0.25.
on resulting data distributions in a practical setting. To that end, we conducted experiments with GFlowNets trained for the molecular generation task proposed by Bengio et al. [36]. We train base GFlowNets with 3 reward functions (x), (x),x), and (XED(X) measuring distinct molecular properties (details in Appendix F.2). Following Bengio et al. [36], we introduced the parameter B which controls the sharpness (temperature) of the target distribution: p(x) a R(x),, increasing B results in a distribution skewed towards high-reward objects. We experimented with B = 32 and / 96 (Figures (d)). After training base GFlow Nets with the respective reward functions, we trained classifiers with Algorithm A. In order to evaluate the base models and the composed GelowNet policies, we generated 000 samples with each policy and anaylzed the samples. Further details are in Appendix F.2. Classifier learning curves are provided in Appendix G.4. Sample diversity statistics of base GelowNets at different values of B are provided in Appendix G.5.
Figure 3 shows reward distributions of base GFlowNets (trained with rewards (x) RSA (x)),, (32, 96]) and their compositions. Base GFlowNet distributions are concentrated on examples that score high in their respective rewards. For each model, there is considerable variation in the reward that was not used for training. The harmonic mean operation (Figures 3(b), 3(e)) results in distributions that are concentrated on the samples scoring high in both rewards. The contrast operation (Figure 3(c)) has the opposite effect: the distributions are skewed towards the examples scoring high in only one of the original rewards. Note that the tails of the contrast distributions are retreating from the area covered by the harmonic mean.
We show reward distribution statistics of three GelowNets (trained with SEH, SA, and QED at P = 32) and their compositions in Table 1. Each row of the table gives a percentage breakdown of the samples from a given model into one of 2 = 8 bins according to rewards. For all three base models. the majority of the samples fall into the "high" category according to the respective reward, while the rewards that were not used for training show variation. Conditioning on two different labels (e.g. y= (SEH, QED)) results concentration on examples that score high in two selected rewards, but not necessarily scoring high in the reward that was not selected. The conditional y= (SEH, QED, SA shifts the focus to examples that have all three properties.
Figure 4 shows 2D embeddings of the distributions appearing in Table 1. The embeddings were computed with tNE based on the pairwise earth mover's 's distances (details Appendix F.2, complete summary of distribution distances in Table G.5). The configuration of the embeddings provides insight into the relative positions of the base models and conditionals in the distribution space. The points corresponding to the pairwise conditionals lie in between the two base models selected for conditioning. The conditional y= (SEH, SA, QED) } appears to near the centroid of the triangle (PSEH. PSA. PQED) and lies close the the pairwise conditionals. The distributions obtained by repeated conditioning on the same label (e.g. y ( (SEH, SEH, SEH)) are spread out to the boundary, lying closer to the respective base distributions and relatively far from pairwise conditionals.
Colored MNIST generation via diffusion models. Finally, we empirically tested our method for the composition of diffusion models image generation task. In this experiment, composed

---

8
03
/
0
2
2
2
2
2
2
2
2
3
0
2
2
2 2 2
/
2
0
1
6
0
0
O 2 0 0 2
2
2
2
3
0
0
2
2
1
0
3
1
5
0
Q
3 0
2 2
2
2
2
3
3
3
3
0
O
2
2
0
2
2
2
2
2
2 2
P1
P2
Pb
P
2
2
2
2
2
R
0
0
2 2
A
2
2
0
2
2
2 2 2
2 2
2
2
2
2
2
2
2 2
2
7
p
Figure 5: Composed diffusion models on colored MNIST. Samples from 3 pre-trained diffussion models and their various compositions.
three diffusion models trained to generate MNIST [56] digits (0, 1, } in two colors: cyan and beige. Each model was trained to generate digits with a specific property: P1 generated cyan digits, P2 generated digits less than 2, and P3 generated even digits.
We built the composition iteratively via the factorization p(x|y1.32. vx) P(x) 1x) To this end, we first trained classifier ((1x,)21x,) on trajectories sampled from the base models. This allowed us generate samples from p(x|y (xx())1)22) We then trained additional classifier (0y31x,,)1.9) on trajectories from compositions defined by (2,,22) to allow us to sample from ((x1)1,)2.13)) Further details can be found in Appendix F.3.
Figure shows samples from the pre-trained models and from selected compositions. The negating effect of not conditioning observations is clearly visible in the compositions using two variables. For example, p(x|y; = 1. y/2 1) only generates cyan 3's. Because there we do not condition on Pa or P3, the composition excludes digits that have high probability under P2 or P3. i.e. those that are less than 2 or even. In pxx = 1, y2 33,, cyan even digits have high density under both and py. but because P2 is not conditioned on, the composition excludes digits less than two (i.e. cyan 0's). Finally, p(xly) = 12 = 2, 3) generates only cyan O's, on which all base models have high density.
7 Conclusion
We introduced Compositional Sculpting, general approach for composing iterative generative models. Compositions are defined through "observations", which enable us to emphasize or de-emphasize the density of the composition regions where specific base models have high density. We highlighted two binary compositions, harmonic mean and contrast, which are analogous to the product and negation operations defined on EBMs. A crucial feature of the compositions we have introduced is that we can sample from them directly. By extending classifier guidance we are able to leverage the generative capabilities of the base models to produce samples from the composition. Through empirical experiments, we validated our approach for composing diffusion models and GFlowNets on toy domains, molecular generation, and image generation.
Broader impact. We proposed a mathematical framework and methods for the composition of pre-trained generative models. While the primary emphasis of our work is on advancing foundational research on generative modeling methodology and principled sampling techniques, our work inherits ethical concerns associated with generative models such as creation of deepfake content and mis- information dissemination, as well as reproduction of biases present in the datasets used for model training. If not carefully managed, these models can perpetuate societal biases, exacerbating issues of fairness and equity. Our work further contributes to research on the reuse of pre-trained models. This research direction promotes friieedelll AI development, with the long-term goal of reducing energy consumption and carbon emissions associated with large-scale generative model training.
10

---

Acknowledgements
TG and TJ acknowledge support from the Machine Learning for Pharmaceutical Discovery and Synthesis (MLPDS) consortium, DARPA Accelerated Molecular Discovery program, the NSF Expe- ditions grant (award 1918839) "Understanding the World Through Code", and from the MIT-DSTA collaboration.
SK and SDP were supported by the Technology Industries of Finland Centennial Foundation and the Jane and Aatos Erkko Foundation under project Interactive Artificial Intelligence for Driving R&D. the Academy Finland (flagship programme: Finnish Center for Artificial Intelligence. FCAI; grants 328400, 345604 and 341763), and the UKRI Turing Al World-Leading Researcher Fellowship. EP/W002973/1.
VG acknowledges support from the Academy of Finland (grant decision 342077) for "Human-steere" next-generation machine learning for reviving drug design", the Saab-WASP initiative (grant 41 1025), and the Jane and Aatos Erkko Foundation (grant 7001703) for "Biodesign: Use of artificial intelligence in enzyme design for synthetic biology".
GY acknowledges support from the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org).
We thank Sammie Katt and Pavel Izmailov for the helpful discussions and assistance in making the figures.
We thank NeurIPS 2023 anonymous reviewers for the helpful feedback on our work.
References
[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ArXiv, abs/2304.00643, 2023.
[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning. pages 8748-878 8763. PMLR, 2021.
[3] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and J Jenia Jitsev. Reproducible scaling laws for contrastive language-imape image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2822 2023.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[6] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray. Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1 010118800,4 2018.
[8] Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] OpenAI, Chatgpt (mar 14, 2023 version). Large language model, 2023. URL https: // /ccat.. openai com/chate

---

[10] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang. Brian Lester, Colin Gaffney Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alexandru Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier García, Jianmo Ni, Andrew Chen, Kathleen Kenealy, J. Clark, Stephan Lee, Daniel H Garrette, James Lee-Thorp. Colin Raffel, Noam M. Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy B. Maitin-Shepard,, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. ArXiv, 115/22033.11899 2022.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.02311, 2022.
[12] Anthony Brohan, Noah Brown, Justice Carbajal. Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch. Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Anand Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Ho Vuong, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. ArXiv, 0aa//2112.08811 2022.
[13] Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-e: An embodied multimodal language model. ArXiv, Abss2303.03378. 2023.
[14] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz. Brian lehter, Alex Irpan. Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth. Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Huen Lee, Sergey Levine. Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not i say: Grounding language robotic affordances. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Leurning Reseurch, pages 287-318. PMLR, 2023.
[15] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn funyasuvunakoo! Russ Bates, Augustin Židek. Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-59) 2021.
[16] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh. Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie. Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky. Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan,
12

---

Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christo- pher R'e, Dorsa Sadigh. Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng. Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, ArXiv, 0os/2108.002555 2021.
[17] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ArXiv, abs/2302.05543, 2023.
[18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744, 2022.
[19] D'idac Sur'is, is, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. ArXiv, abs/2303.08 08118888 2023.
[20] Jason Wei. Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Advances Neural Information Processing Systems, 35:24824-24837, 2022.
[21] Geoffrey E Hinton. Products of experts. In Ninth International Conference on Artificial Neural Networks, volume 1, 1999.
[22] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[23] Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy, Generative models of visually grounded imagination. In International Conference on Learning Representations, 2018. URL https: /ww.revieeeeeeeeeeoe/ee net. /tt///rrrmmdddiammml......
[24] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6666 2020.
[25] Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34: 1 15608-15620, 2021.
[26] Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose visual relations. Advances in Neural Information Processing Systems, 16117718 2021.
[27] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 423-439. Springer, 2022.
[28) Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International Conference Machine Learning, pages 8489-8510. PMLR, 2023.
[29] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objectin molecule generation using interpretable substructures. In International conference on machine learning, pages 4849-4859. PMLR, 2020.
[30] Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS: Markov Molecular Sampling for Multi-objective Drug Discovery. In International Conference on Learning Representations, 2021. URL htwph:tt//////t //operrviee//eeieeeee wttpeeiwwweeeeeeueneeeeeee net/forum? id-kHSuu4exFXX.
13

---

[31] Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez Garrnndez-Garria, Jarrid Rector- Brotor-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-ojjective GFlownets. In International Conference on Machine Learning, pages 14631-14653. PMLR, 2023.
[32] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamicn In International Conference on Machine Learning, pages 2256-22252 2265. PMLR, 2015.
[33] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33: 08:1238-11245 2020.
[34] Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: httt:::///penreviers net/forumiid-ee Px..ouuni2ddPIII111RER..1
[35] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55 2023, URL http: /ttp://mmml http/appers////// 0364. html.
[36] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394, 2021.
[37] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[38] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[39] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794, 2021.
[40] Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Mgan: Training generative adver- sarial nets with multiple generators. International conference on learning representations. 2018.
[41] Ilya O Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel and Bernhard Schölkopf. Adagan: Boosting generative models. Advances in neural information processing systems, 30, 2017.
[42] Aditya Grover and Stefano Ermon. Boosted generative models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[43] Nan Liu, Yilun Du, Shuang Li, Joshua Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-image generative models. ArXiv, abs/2306.05357,, 2023.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photoreal- istic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494,20 2022.
[45] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL httpps://opeereriiee net/forum? (d=ooummmduuuuAx
[46] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. URL. ptttps://operreviii htt///orumiidddBBeeeee cPu_tE.
14

---

[47] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In The Eleventh International Confer- ence on Learning Representations, 2023. URL. https: / http://eeeeeeeeeeee/oooronnn id= 31ge0p50-M-.
[48] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compasitionallty decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
[49] Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. ArXiv, abs/2208.0000 00005,000550000 2022.
[50] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. ArXiv, abs/2204.0612 25,, 2022.
[51] Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández-Garee Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pages 18269-18300. PMLR, 2023.
[52] Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. ArXiv, abs/2209. 02600, 2022.
[53] Stefano Peluchetti. Non-denoising forward-time diffusions, 2022. URL https:// net//forumndd=~V/IIKunggg
[54] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Representations, 2023. URL https: :ttps:://ppoo /ttp///ppeeeeeeeeee/eeeeeeeeeeee net/forummid
wttpeenrrrie..netotuuumuaaaaaaaa htt./oorrmmddPPPPMMMDCTTTT. PquMRDCJT9t.
[55] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):52-5333 2015.
[56] Yann LeCun. The mnist database of handwritten digits. http://sam.lecum.com/ee// 1998.
[57] Damiano Brigo. The general mixture-diffusiig sde and its relationship with uncertain- volatility option model with volatility- asset decorrelation. ArXiv, 10.0528224402,2022 2008
[58] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory bal- ance: Improved credit assignment in GFlownets. In Advances in Neural Information Processing Systems, volume 35, pages 5955-5967, 2022.
[59] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ArXiv, abs/1412.6980, 2014.
[60] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR. 2017.
[61] Peter Ertl and Ansgar Schuffenhauer. Estimation synthetic accessibility score of drug drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1:1-11, 2009.
[62] Greg Landrum. Rdkit: Open-source cheminformatics, 2010. http::tss://www rdkit. www.iittttt.....ooooooooooo
[63] Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
[64] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
15

---

[65] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer- Interven- lion - MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[66] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-75477 2020.
[67] Matthew D Zeiler. Adadelta: adaptive learning rate method. ArXiv, abs/ 221..5711,
[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, 32, 2019.
16

---

A Compositional Sculpting of Iterative Generative Processes (Section 5, continued)
A.1 Classifier Training (GFlowNets): Algorithm
Algorithm A.1 Compositional Sculpting: classifier training
1: Initialize & and & &b
2: for step = 1, ...., num_steps do
3: for = 1, ...., do
4: Sample ? ~ (r)
5: end for
6: Cr(()
Σ
log (z =ix==xx))))
(Terminal state loss (16))
i=1 7: W x) (== i.j [l....m) m}
(Terminal probability estimates) )
(Non-terminal state loss (17)-(18)
8: LN(Ø Φ) Σ Σ
π
(3g
018g.11.....
31-1
-
2=2
9: = (T) + 7(step) L (Φ. Φ))
10: Update & using V vg((. Φ); update Φ = Pbp + (1 - f)Φ
end for
A.2
Composition of Diffusion Models
In this section, show how the method introduced above be applied to diffusion models. First, we adapt the model we introduced in (9)-(11) to diffusion models. diffusion model trained to sample from p,(x) generates a trajectory = | over range of time steps which starts with a randomly sampled state Xy and ends in xo- where Xg has distribution (x)==(() (x). Thus, must adapt our model to reflect this. We introduce a set of mutually dependent variables for E (0, 7'] with as conditional distribution the transition kernel of the diffusion model p,(x,1xp).
Given m base diffuusio models that sample from PI.... .m respectively, we define the prior p(x) as mixture of these diffusion models. Proposition A.l shows that this mixture is a diffusion model that can be constructed directly from the base diffusion models. We then apply classifier guidance to this mixture sample from the composition. We present informal version of the proposition below. The required assumptions and the proof are provided Appendix D.5.
Proposition A.1 (Diffusion mixture SDE). Suppose distributions (,(x), ..., .......P(333 (x) are realized by diffusion models with forward SDEs dx,,, = (x,( dt + 8i, dwi,, and score functions Ci,(),, respec-
tively. Then, the mixture distribution PM(()) = >m= (x),(x) with
m 2 0 and - 02 = 1 is
realized by a diffusion model with forward SDE
dx,- Σ
pl==ilx)f..xx))
dr+
Σ 1=1
p(y=ilx)e, dw,
(19)
fas
BMr((())
and backward SDE
Σ ply= p(y= (1xx(() -eip))())))))) di Σ ply= dw,
dx, =
(20)
with
py=illx)) = (1p),((())
(21)
Σn (x,)
17

---

If the base diffusion models have a common forward SDE dx,, = f((x,,) di + dwi, equations (19)-(20) simplify
( [===())D)))))
m
)
dx;- = f((x)dd + g,dw, dx, = ((xx) ((x)) - of
Σ
di g,dw,
(22)
Theorem A.2 summarizes the overall approach.
Theorem A.2. Suppose distributions pi p((x).... ......... (.m))....PPP(.. realized by diffusion models with forward SDEs dx = fi,((;,))) dt &ia dwi, and score functions S; respectively. Let y1.... Yn be random variables defined by (9). Then, the conditional p(xly!..... is realized by a ceassifier-guidee diffusion with buckward SDE
dx, uc, F1....... .1...... (t)d..... + (x,(x,
(23)
with
(x.
Σ p(== 1 (|x;) fix(x,) - Big (s)r(xx)) x,, log poy..........
......))))))
(24)
m11
(x, =
Σ p
(25)
The proof of Theorem A.2 is provided in Appendix D.6.
A.3 Classifier Training (Diffusion Models)
We approximate the inferential distributions in equations (22) and (23) with time-conditioned classifier ",,,,,,,,,,,,,,, with parameters vb. Contrary to GFlowNets, which employed a terminal and non-terminal state classifier, here we only need single time-dependent classifier. The classifier trained with different objectives on terminal and non-terminal states. The variables yj....> Yn are dependent given state x, for i E [0, T), but are independent given the terminal state Xy. Thus, when training on terminal states we can exploit this independence. Furthermore, we generally found beneficial to initially train only on terminal states, The loss for the non-terminal states depends on classifications of the terminal state of the associated trajectories, thus by minimizing the classification error of terminal states first, we reduce noise in the loss calculated for the non non-terminal states later.
For terminal state xg. the classifier 2g01 ,/lxx, ,,1xx)) can factorized as II (0()))x0)) Hence
can train @ by minimizing the cross cross-entropy loss
(T( (1)x..))))))))
[- log
=
(26)
Samples p(Xp- y1) can be generated according to the factorization ply, (0)))(x)l))) )) First, F1 is sampled from p(y,), which is uniform under our choice of p(x)). Then, 2, D1) is generated from the reverse SDE of base difffusion model Pś; (x). Note that equation (9) implies that all observations
have the same conditional distribution given Thus, 26(y (x)) is also a classifier for observations Y/2 ......>> Fn
For a non-terminal state x, with / E (0, T}, we must train Q to predict yj...., jointly. For non-terminal state X, and observations D Fnr the cross-entropy loss is
E2 P,: p) = log 0,(0- = S ",
(27)
Tuples ( ....... Pr,) are obtained as follows: 1) 9 ~ P)))) A trajectory = = [x1) is sampled
from the reverse SDE of diffusion model y1- At this point, we would ideally sample Pz.... Fr given
&o but this requires access to plyx Rg) Instead, we approximate this with u;(3; Φ) 2 ilx0= kn) and marginalize over Fz. ..... 8r to obtain the cross-entropy loss
m
(x( (f.,)))(
Σ
Σ
Σ
[
w'y (oi (R,, FaisD)
(28)
(
9==1
D,=1
18

---

B Classifier Guidance for Parameterized Operations
This section covers the details of classifier guidance and classifier training for the parameterized operations (Section 4.1).
The complete probabilistic model for the parameterized operations on two distributions given by
1 f(x, F1: bx) f(x) 2 (x)) 22((x,
(29a)
p2(x) Pi(x) D2()) ))===1(;;)) (x) [i=1] ((1-a)p2(x)) [i=2]
7 =1|x)
(29b)
ap + -1 (x)
While in the probabilistic model (3) all observations y; are exchangeable, in the parameterized model (29) y1 and F2 are not symmetric. This difference requires changes in the classifier training algorithm for the parameterized operations.
We develop the method for the parameterized operations based on two observations:
Y1 appears (29) in the same way as in (3);
the likelihood p a) of y2 given x can be expressed as the function of Ply (x) and a:
(y(s)
(x)
Pj (x)++22X0
b02=11(;;)-
ap,(x) xx)p(xx
P2(X)
P;(X)
p2(0)
())
p2(0)+22xx)
(30a)
x
aply () =|xx
(1 (x) Px(x)
(1 (x)x)
P1(0+p2(x)
-1;=)(x;;)
(x) + ax)2(x)
(1 a)) Pz(x)
P: (x)+2((
Pi(00+2(X))
(30b)
(1 (1 (1-a)p(1;=3 =||x)
a + (1 a) =
These two observations combined suggest the training procedure where 1) the terminal state classifier trained to approximate ply; =i[x) in the same way as in Section 5.2; 2) the probability estimates wx, ã; Φ) ply; =1: @) expressed through the learned terminal state classifier ply; = =/[x) via (30). Below we provide details of this procedure for the case GFlowNet composition.
Learning the terminal state classifier. The marginal yj classifier (v (x) is learned by minimiz- ing the cross-entropy loss
Cr(()= E
| -og log - =3)
(31)
(-P)))))))))
Then, the joint classifier 0r 121x; a) is constructed as
2s0, 121x; (
(32)
where (x(221x; can be expressed through the marginal 150,,xx) via (30).
Learning the non-terminal state classifier. The non-terminal state classifier 20 y21s a) ar models y1 and y/2 jointly. Note that is one of the inputs to the classifier model. Given sampled trajectory f, labels 91. Pz. and a, the total cross-entropy loss all non-terminal states in is
1E-11
C S. a +
Σ
| log = -
(33)
19

---

Pi S
0:001 0,500
066
0.10
01
0.1
0.1
(a) Base distributions P1. P2
(b) HM (ouss) Product
(c) Contrasts couns) Pt 0 P2 (d) Negations Pj neg, P2
Figure C.I: Compositional sculpting and energy operations applied to ID Gaussian distributions.
(a) Densities of base ID Gaussian distributions Pi (x) = Nx; -/44 1) and P2(x) N (x; 5/4, 1/2). (b) harmonic mean pj @ P2 and product P1 prod p2. (c) parameterized contrasts p1 O P2 at different values of a. (d) negations Pi neg, P2 at different values of y. Curves show the PDFs of distributions.
The pairs (?, F1) can be generated via a sampling scheme similar to the one used for the terminal state classifier loss above: 1) V1 p(y,) and 2) Pý, (r). An approximation of the distribution of F/ given & is constructed using (30):
a =
w 10,22, â:
1002=11>>> = ,;
(34a)
+
(1 (1=@)@,())=2)x== x)
W2( 8:
a0 = 8) (1- -1=)(((11)),)))) & p0 x;
(34b)
Since these expressions involve outputs of the terminal state classifier which is being trained simulta- neously, we again (see Section 5.2) introduce the target network parameters & that are used compute the probability estimates (34).
The training loss for the non-terminal state classifier
C (b. Φ) =
E
Σ
W3y (,,
(35)
(-
5--1
where p(a) is sampling distribution over a E (0, 1). In our experiments, we used the following sampling scheme for a:
2 U~UB B).
(36)
1 exp(-22)
C Analysis of Compositional Sculpting and Energy Operations
The harmonic mean and contrast operations have introduced are analogous to the product and negation operations for EBMs respectively. Although the harmonic mean and product operations are quite similar in practice, unlike the negation operation our proposed contrast operation always results a valid probability distribution. Figure C. shows the results of these operations applied to two Gaussian distributions. The harmonic mean and product, shown in panel (b), both concentrated on points that have high probability under both Gaussians. Figure C.1(c) shows parameterized contrasts Pi (L at different values of a, and panel (d) shows negations P: neg, P2 at different values of y. The effect of negation at y = 0.1 resembles the effect of the contrast operation: the density retreats from the high likelihood region of p2. However, as 7 increases 0.5 the distribution starts to concentrate excessively on the values < -3. This is due to the instability of division Pi (x)/(x2(())) in regions where (x(x) 0. Proposition C. shows that negation p1 neg, Pr in many cases results in an improper (non-normu) normalizaezl) distribution.
Mathematical analysis of operations. Harmonic mean and product are not defined for pairs of distributions P1 P2 which have disjoint supports. such cases, attempts at evaluation of the expressions for Pi ② P2 and pi prod P2 will lead to impossible probability distributions that have zero
20

---

probability mass (density) everywhere'. The result of both harmonic mean and product are correctly defined for any pair of distributions Pt. P2 that have non-empty support intersection.
Notably, contrast is well-defined for any input distributions while negation is ill-defined for some input distributions Pt. P2 formally stated below (see Figure C.I (d) for concrete example). Proposition C.I.
1. For any a E (0, 1) the parameterized contrast operation Pt O (I-a) P2 (7) is well-defined: gives proper distribution for any pair of distributions Pt. P2-
2. For any y E (0, 1) there are infinitely many pairs of distributions Pt P2 such that the negation
(p) negy (x) o exp (xp(-((1())) a P₂(x)
(37)
(D2(x))²
results in improper (non-normalizable) distribution.
Proof. Without loss of generality, we prove claims of the proposition assuming absolutely contin- hous distributions Pt P2 with probability density functions P1 (-), P2(-).
Claim For any distributions P1. P2 we have (x) 2 O. p2(x) 0, (x) fx P2(x) dx < co. Then, the RHS of the expression for the parameterized contrast operation
Pi (1-a) P2 (7) satisfies
Pi (x)2
Pi(x)
(x(
Pi(x) S
Vx E supp(p)) supp(p2). (38)
ap, (x) (1 a))p(x)
Pı(x) + (1-0)
(x
a
For points x $ supp(p;) U supp(p2), we set ite distributions do have probability mass outside of the union of the supports of the original distributions. The above implies that
ap (x+((-aaa)pp since by construction the compos-
xx(x)
dx dx - p / a
(39)
(x) + (1 axp((x)
a
Therefore, the RHS of the expression for the parameterized contrast operation P1 (1-a) P2 (7) can be normalized, and the distribution P1 (1-a) Pz well-defined.
Claim 2.
For any y E (0, 1) we provide an infinite collection of distribution pairs and P2 such that negation P neg, P2 results in non-normalizable distribution.
For the given y E (0, 1) we select four numbers #1 R. H2 R., > O. 02 > such that
1 of z -
(40)
Consider univariate normal distributions (x) Nx: M1, of), p2(x) = Nx: N of) with density functions
(x (x-11)² 21)
(x(x) Nx: u;: 0?)
exp
i (1.2).
(41)
20²
V 2no
For such P1 and Pz, the RHS of (37) is
(x)
of
exp
2
10, - of
H1
H2
2 + y
"
(42)
(pz(x)))
(V2x))
oi
20702
o
2
20
2
20
Condition (40) implics that the quadratic function under the exponent above has a non-negative coefficient for x Therefore this function either grows unbounded as (if the coefficients for the quadratic and linear terms are not zero), or constant (if the coefficients for quadratic and linear
terms zero). In either case, fe Pı(x)/122(0)) dx
Inforrmal interpretation: distributions with disjoint supports have empty "intersections" (think of the inter- section of sets analogy)
21

---

D Proofs and Derivations
D.1 Proof of Proposition 5.1
Our goal is to show that the policy (12) induces the mixture distribution Par(x) = E (,,,(x),
In our proof below we use the notion of "the probability of observing state E S on GelowNet trajectory". Following Bengio et al. [35], we abuse the notation and denote this probability by
Preliminaries.
|r|-1 II
p,(s) - p,((= sEt))
Σ rET
(s,,,))) 1=0
(43)
where T is the set of all (sub)trajcctories starting So and ending s. The probabilities induced by Sp.s the policy (12) are denoted by (s). Note that (s(s) and PMr(s) should not be interpreted as probability mass functions over the set of states S. In particular (,(5) = (5g) = and sums Eves (s), Eses Par (s) are not equal to I (unless S (so]). )). However, the functions p,(.), PM restricted the
set of terminal states X' give valid probability distributions over N: Exex (x = ExEX PM (x) 1.
By definition p,()) and PM (-) satisfy the recurrent relationship
(,(s)
Σ
(s(s))
(s)())
Σ
PM.(s))M..((slss))
(44)
5,11s, -S)A
The joint distribution of and r described in the statement of Proposition 5.1 is y()) This joint distribution over y and trajectories implies the following expressions for the distributions involving intermediate states s.
p(= 0
(45)
|-|-1
p(rly=1)=p,( = P.r 7=
(46)
m
m
p(r) Σ p(rly=i)p(==))= Σ
e (r),(r)
(47)
p(ly=1) = p,(s),
(48)
p(s) Σ p(sly=i)((==))= Σ (s).
(49)
Proof. Using the notation introduced above, can formally state goal. We need to show that (x) induced by PM. F gives the mixture distribution
PM (x) (x) Σ (x),
(50)
1=1
We prove a more general equation for all states s E S
(M) Σ (,,,(s)
(51)
T=1
by induction over the DAG (S.A).
Base case. Consider the initial state So S. By definition p,(5)) (M) 1 which implies
m
(M)) = Σ (s))(sp))
(52)
22

---

Inductive step. Consider a state s such that (51) holds for all predecessor states s, (s, 5) A. For such a state we have
(s) = Σ
((,
(used (44))
(53)
-
Σ
PM(s,)
Σ ply i|s, (s
{used definition of PM,F)
(54)
(M))
Σ
pls. DOsSDD=DDD=D (used Bayes' theorem)
(55)
5,66, p(s,)
PM)(s))
Σ
Σ 2o, (used (45), (48), (49))
(56)
- (sP)(s.)
s, -S)EA
Σ
Σ
osP.ss.)):.((ssss))
[used induction hypothesis)
(57)
5, -SEEA
Σ !
Σ
P((s,) p((s))).((((ss.))
(changed summation order)
(58)
i=1
I1
= Σ
(s(s(),
{used (44))
(59)
which proves (51) for
D.2 Proof of Proposition 5.2
Claim 1. Our goal to prove the relationship (13) for all non-terminal states E S \ &. To prove this relationship, we invoke several important properties of Markovian probability flows DAGs [35].
By Proposition 16 of Bengio et al. [35] for the given GFlowNet forward policy pr(+))) there exists unique backward policy Pa(-I-) such that the probability of any complete trajectory r (5o
S/z] = x) in DAG (S, can be expressed as
F
p(r) = p(x) Il
(l(s)
(60)
and the probability of observing state E S on trajectory can be expressed as
p(s) Σ p(x) XER
Σ TET, T=1
I Pn(s,-1ss))
(61)
where Txx is the set all (sub)trajectories starting at and ending at x. Moreover, Pr(-l-) and Pg(.1.) related through the "detailed balance condition" [35, Proposition 21]
p(s)pr(5115 = p(sp)(()))))))) (s(ss'',,
((s - s')
(62)
By the statement of Proposition 5.2, in the probabilistic model p(x, y). the marginal distribution p(x) realized by the GFlowNet forward policy (+(-).) and y is independent intermediate states s. The joint distribution p(s. is given by
le|
p(s,y) Σ y)x,)) X
Σ
II
Pn(s,_lss))
(63)
rET
[=1
|r| Σ rET,
Σ y(x,)) Σ
Pg(s|s')
I
pB(s,_1ss)
(64)
23

---

|=]
Σ
PB(s(s')
Σ p(x,
Σ
Il
p_(s))1ss)
(65)
XEN
rET
Σ
Pg(s)s')p(ss.))
(66)
Expressing the conditional probability p(y]s) through the joint p(s, y) we obtain
plyls) p(s.y)
{used definition of conditional probability)
(67)
p(s)
Σ
Pa(s(s')p(''s [used (66))
(68)
pls)
110--1'AA
Pa(s|s') p(s')
Σ
p(s) D(rls') (decomposed p(s',y) p(s)))()ls))
(69)
Pr(d's))p(lls d'(0-=))
Σ
[used (62))
(70)
which proves (13).
Claim 2. Our goal is show that the classifier-guide policy (14) induces the conditional distribution p(r|x).
We know (Section D.1) that the state probabilities induced by the marginal GFlowNet policy p=(-1.) satisfy the recurrence
p(s)
Σ
(sls).
(71)
5,:(s_ -JEA
Let py(.) denote the state probabilities induced by the classifier-guided policy (14). These probabilities by definition (Section D.1) satisfy the recurrence
py(s) = Σ
(((s))pp(s(ss..))
(72)
We show that
pv(s) pssl))
(73)
by induction over DAG (S,A).
Base case. Consider the initial state so- By definition (s() 1. At the same time p(sply) p((r: : so E {)ll)) 1. Therefore (y(5)) = p(sgly)
Inductive step. Consider For such a state we have
such that (73) holds for all predecessor states s, (s. s) A.
P,((s) = Σ
(pp(ss) {used (72))
(74)
=
Σ
(s((ss plyls)
p(yls.)
[used (14))
(75)
-.:(s,-s)EA
p(rls)
Σ
((..l))p(((ss.)
(used induction hypothesis)
(76)
p(yls_)
5,66
Σ
p(rls))p(s.)
(r(s(s.)
p(yls) (used Bayes' theorem)
(77)
ply)
p(yls,)
plyls) Σ ((s,s)(((ss.)
[rearranged terms)
(78)
p(y)
5,;(,,5)e..
24

---

p(yls)
p(y) p(s) {used (71))
(79)
= =p(sly), (used Bayes' theorem)
(80)
which proves (73) for state
D.3 Proof of Theorem 5.3
By Proposition 5.1 we have that the policy
m
PM.r(s'ls)= Σ 1s
(81)
generates the mixture distribution PM (x) = Σ (x
In the probabilistic model f(x. Vn,) the marginal distribution p(x) = (x) is realized by the
mixture policy PM.F- Therefore, p(x, y..... ........,))) satisfies the conditions of Proposition 5.2 which states that the conditional distribution p(xly)..... .,, ) is realized by the classifier-guided policy
pp yi.....y, Vn,) PM,r(s' Pl......... ... Vil.''"...lll''........... pl F2 Fals') m
Σ
(82)
Pl...... PL y' 1nl
D.4 Detailed Derivation of Classifier Training Objective
This section provides more detailed step-by-step derivation of the non-terminal state classifier training objective (18).
Step 1. Our goal is train classifier ē6% ................................ Vrls). This classifier be obtained as the optimal solution of
min
(83)
where é(-) is defined in equation (17). An unbiased estimate of the loss (and its gradient) can be ob- tained by sampling (ê, f1.... Pry) and evaluating (17) directly. However sampling tuples (r, y1.-.. Fi)) not straightforward. The following steps describe our proposed approach to the estimation of ex- pectation in (83).
Step 2.
The expectation (83) can be expressed as
-
Σ
Σ
πI
pu; fu,=ū;xx=)) 1=2
(ai
(84)
where we re-wrote the expectation over
as the explicit sum of the form (()))(((2)]
Σzez (Z)) The expectation over (r, (1) can be estimated by sampling pairs (f,Y) as described in the paragraph after equation (17): 1) f poy,) and 2) è ~ Pý, (r). The only missing part is the probabilities ply; = D, x) which are not directly available.
Step 3. Our proposal is to approximate these probabilities as ply =]x==})
i|x = 2). The idea here is that the terminal state classifier 1 |x), when trained to optimality, produces outputs exactly equal to the probabilities Ix), and the more the classifier is trained the better is the approximation of the probabilities.
Step Steps 1-3, give procedure where the computation the non-terminal state classification loss requires access to the terminal state classifier. As we described in the paragraph preceding equation (18), we propose to train non-terminal and terminal classifiers simultaneously and introduce "target network" parameters. The weights w are computed by the target network Q
Combining all the steps above, we arrive at objective (18) which we use to estimate the expectation in (83).
25

---

Note that equation (18) involves summation over P2.... ...... Fn with m= terms in the sum. If values of and m are small, the sum can be evaluated directly. In general, one could trade off estimation accuracy for improved speed by replacing the summation with Monte Carlo est estimation. In this case, the values Fg are sampled from the categorical distributions (z(+)x). Note that labels can be sampled in parallel since y; are independent given x.
D.5 Assumptions and Proof of Proposition A.1
This subsection provides a formal statement of the assumptions and more detailed formulation of Proposition A.I.
The assumptions, the formulation of the result, and the proof below closely follow those of Theorem I of Peluchetti [53]. Theorem I in [53] generalizes the result of Brigo [57] (Corollary 1.3), which derives the SDE for mixtures of ID diffusion processes.
We found error in the statement and the proof of Theorem I (Appendix A.2 of [53]). The error makes the result of [53] for D-dimensional diffusion processes disagree with the result [57] for I-dimensional diffusion processes.
Here we provide a corrected version of Theorem I of [53] in modified notation and simplified setting (mixture of finite rather than infinite number of diffusion processes). Most of the content is directly adapted from [53].
Notation:
for vector-valued f R.D RRD, the divergence of f is denoted as V (f(x)
à Ed=1 dxg (x),
for a vclles-values a R R, the divergence of the gradient of (the Laplace operator)
is denoted by A(a(x)) V (Va(x)) Σ d=1 dx
a(x).
Assumption 1 (SDE solution). A given D-dimensional SDE(f.g)
dx, (
(85)
with associated initial distribution Px(x) and integration interval [0, 7'I admits a unique strong solution on [0, 7j.
Assumption 2 (SDE density). A given D-dimensional SDE(F.8) g) with associated initial distribution Po(x) and integration interval [0, T] admits marginal density on (0, with respect the D- dimensional Lebesgue measure that uniquely satisfies the Fokker-Plank Kkolmogorov-fowware partial differential equation (PDE):
(x,(x) = (f,x)(())))))))) ((,)
(86)
Assumption 3 (positivity). For a given stochastic process, all finite-dimensional densities, conditional or not, are strictly positive.
Theorem D.1 (Diffusion mixture representation). Consider the family of D-dimensional SDEs on I E [0, T] indexed by i E (l,...,
dx,, = (x,((x,)dt + Br,dwi, X1,0 Pi.--
(87)
where the initial distributions Pıo and the Wiener processes Wiy are all independent. Let Pip E (0, T) denote the marginal density of X For mixing weights (wo mixture marginal density for (O. and the mixture initial distribution PM. by
1 f-r 0; 2 0. Σm 0 = 1, define the
(x)= Σ (x),,()) PM,o(x)=
Σ p(PM)
(88)
1=1
Consider the D D-dimensional SDE on [0, defined by
(x) E (x),(()
E (x),,
(x)
(89)
(x)
(x)
26

---

dx, fm,x) + BM,(x,)dw,
XMO ~ PM.
(90)
It assumed that all diffusion processes X1 and the diffusson process x MMs satisfy the regularity Assumptions 1, 2, and 3. Then the marginal distribution of the diffusion X is PM.
Proof. For 0 < < T we have that
((), a
Σ
(91)
dr
1=1
m
(x)
@
(92)
dt
1=1
=
Σ
1
-V ((x(()))),(x)) + _A(S)
(93)
-v. (0
Pi,(x
=
Σ 1f i=1
PMJ(x)
PM..(O))
+
2
1
(x,(x)
Pmn,(0))
(94)
(1P
@xPr,
PN.,(X))
+ 2
Σ
(x) PM)
(95)
(x)(x)
= = - (f (x))
(96)
The second is an exchange of the order of summation and differentiation, the third line is the application of the Fokker-Planck PDEs for processes x,,, the fourth line is a rewriting in terms of PM the fifth line is another exchange of the order of summation and differentiation. The result follows by noticing that PM PM, (x) satisfies the Fokker-Planck equation of SDE(J M 8m).
Proof Proposition A.1. Below, we show that the result of Proposition A. follows from Theo- rem D.I.
First, we rewrite (x,) and (x,) in (89) in terms of the elassifier probabilities (21):
(x,(x,) Em (iPr,
pb= p()=
(97)
(x,)
Σm (x,
(x, =
Σ pby=i|x,)8;
(98)
(M,)
With these expressions, we apply the result of Theorem D. 1 to the base forward processes dx, = (i)(x,)) di + gi, dw,,, and obtain the mixture forward process in equation (19).
From the forward process, we derive the backward process following Song et al. [34]. Using the result of Anderson [38], the backward process for (19) is given by
dx, = fM, - Vx, (x,,),))) - (x, log (x,) dt (x, duo,
(99)
Note that the term (8, (x))) is due to the fact that the diffusion coefficient (x,) in (19) is function of (cf., equation (16), Appendix A in [34]). This term be transformed as follows
m
V x My (x,)) = Vx,
(g2
Σ ppy= ix,
(100)
m
Σ
81 V. plp ilxx,)
(101)
m
= Σ p(y= Vx, log p(y==(xx))
(102)
27

---

Σ p(y=1|x))8; xx, loggo, + (og - log PMM. (xx))
(103)
Σ pl p ( xx, logg Vx, log PPPM,((X)) 1=1
(104)
Σ (
(x, log PM (x).
(105)
Substituting the last expression in (99), we notice that the term 8 (x,) log (x,) cancels out, and, after simple algebraic manipulations, we arrive at (20).
D.6 Proof of Theorem A.2
We proove Theorem A.2 the assumptions of Section D.5, In Scction D.5 we established that the mixture difflusion process has the forward SDE
dx, (x, + (x, due.
(106)
and the backward SDE
dx, [M,(x,) vx,, (8, (x,)) 8M (x,) log (M, di + (x,) du,
(107)
We apply classifier guidance with classifier 1. ....... yn ,,,,,,)
the mixture diffusion process, following Song et al. [34] (see equations (48)-(49) in [34]). The backward SDE of the classifier-guided mixture diffusion is
dx, (M,((,) V (x,(8M (x,) log (x,) + V,, log plr...... ....,,,,lxl))) di (108)
+ (x,, diy.
(109)
Finally, we arrive at (23) by substituting (105) in the above, canceling out the term 8 (x,) (x, and applying simple algebraic manipulations.
E
Implementation Details
E.1
Classifier Guidance in GFlowNets
Classifier guidance in GelowNets (14) is realized through modification of the base forward policy via the multiplication by the ratio of the classifier outputs porls'/p(tls). The ground truth (theoretically optimal) non-terminal state classifier p(yls) by Proposition 5.2 satisfies (13) which ensures that the
guided policy (14) is valid, i.e. for any state E S
pp(s'1s) plrls') p(yls)
Σ (E
p)))
Σ
(110)
Σ
==(s(1s)p(yss)==11 1.
(111)
p(yls)
of d'(3-s'Ee
=PO(s) by Proposition 5.2
In practice, the ground truth values of p(yls) are unavailable. Instead, an approximation Q (y]s) s s(yls) is learned. Equation (13) might not hold for the learned classifier 2Φ but still wish to Øg for classifier guidance in practice. In order to ensure that the classifier-guidd policy is valid in practice even when the approximation 2 (yls) of the classifier p(yls) is used, we implement guidance as described below.
First, express the guided policy (14) in terms of probaabbiiiitie:
log logpr(s'll (s.ss'ls,)) log ( (s'1s) + log p(yls') log p(yls).
(112)
28

---

Parameterizing distributions through prg-roabbilitiess is common practice in probabilistic modeling: GelowNet forward policies [36, 58] and probabilistic classifiers are typically parameterized by deep neural networks that output logits (unnormalized pgg-pobbabilitie).
Second, in the log-probability parameterization the guided policy (14) can be equivalently expressed as
Dr(s'|s,)) y) = [softmax (og log p(bl))))
(113)
exp ( log Pr log log p(yls))
(114)
(e exp log (s( + log p(rl=") logs)))))
In theory, the softmax operation be replaced with simple exponentiation, i.e. the numerator in (114) is sufficient on its own since Proposition 5.2 ensures that the sum in the denominator equals to 1. However, using the softmax is beneficial in practice when we substitute learned classifier 2g(1ss) instead of the ground truth classifier p(yls). Indeed when Qø(rls) does not satisfy (13), the softmax operation ensures that the guided policy
P (s. y) = |oftmax ( (log (r(()s) + log (+1)) - log (r()))
(115)
valid (i.e. probabilities sum up over s'). The fact the softmax expression is valid in theory ensures that policy (115) guided by (v(()) approaches the ground truth policy (guided by p(yls)) as (y(s)s) approaches p(yls) throughout training.
Experiment details
F.I 2D Distributions with GFlowNets
The base GFlowNet forward policies (s'[s;)) were parameterized MLPs with 2 hidden layers and 256 units each hidden layer. The cell coordinates of a state s on the 2D 32 x x.32 grid were one-hot encoded. The dimensionality of the input was 2 32. The outputs of the forward policy network were the logits of the softmax distribution over 3 action choices: 1) move down; 2) move right; 3) stop.
We trained base GFlowNets with the trajectory balance loss [58]. We fixed the uniform backward policy in the trajectory balance objective. We used Adam optimizer [59] with learning rate 0.001, and pre-train the base models for 200 steps with batch size 16 (16 trajectories per batch). The log of the total flow log Ze was optimized with Adam with learning rate 0.1. In order to promote exploration in trajectory sampling for the trajectory balance objective, we used the sampling policy which takes random actions (uniformly) with probability 0.05 but, otherwise, follows the current learned forward policy.
The classifier was parameterized as MLP with 2 hidden layers and 256 units in each hidden layer. The inputs to the classifier were one-hot encoded cell coordinates, terminal state flag ((0, 1)), and log(s/(1-n)) (in the case of parameterized operations). The classifier outputs were the logits of the joint label distribution @40r,.... ...., ss) for non-terminal states and the logits of the marginal label distribution 1x)x for terminal states x.
We trained the classifier with the loss described in Section 5.2. We used Adam with learning rate 0.001. We performed 15 000 training steps with batch size 64 (64 trajectories sampled from each of the base models per training step). We updated the target network parameters & as the exponential moving average (EMA) of d with the smoothing factor 0.995. We linearly increased the weight (step) of the non non-terminal state loss from 0 to throughout the first 3 000 steps and kept constant y= = I afterward. For the a- d-parameterized version of the classifier (Section B), we used the following sampling scheme for a: U1.3.5,3.55, à T+exp(-2)*
Quantitative evaluation. For each of the composite distributions shown in Figure 2 we evaluated the L1 dis-titt L, (Pietbodd PGT) ExEx (x)ethod - Px)(x)] between the distribution Pmethod induced by the classifier-guidee policy and the ground-truth composition distribution PGr computed from the known base model probabilities p,. The evaluation results are presented below.
29

---

Figure 2 top row. Pi @ P2: L = 0.071; P: P2: L, = 0.086; 0.95 P2: L = 0.167.
Figure 2 bottom row. px 1. = 2): 1,
0.076: ア(,|y = 0.1 0 =
3): /, 0.087;
p(xly) =2, =2) 4
1 0.122
Figure G.6 shows the distance between the composition and the ground truth as function of the number of training steps for the classifier as well as the terminal and non-terminal classifier learning curves.
F.2 Molecule Generation
Domain. In the molecule generation task [36]. the objects x E & are molecular graphs. The non- terminal states E S \X are incomplete molecular graphs. The transitions from a given non-terminal state s are of two types: 1) fragment addition s': new molecular graph a new fragment to the molecular graph 2) stop action s - x: if s * Sp- then the generation process can be terminated the molecular graph corresponding to the current state (note that new terminal state x E X' is different from E S X, but both states correspond to the same molecular graph).
obtained by attaching
Rewards. We trained GelowNets using 3 reward functions: SEH, reward computed by an MPNN [60] that was trained by Bengio et al. [36] to estimate the binding energy of molecule to the soluble epoxide hydrolase protein; SA, an estimate of synthetic accessibility [61] computed with tools from RDKit library [62]; QED, a quantitative estimate of drug-likeness [63] which is also computed with RDKit. We normalized all reward functions to the range [0. 1J. Higher values of SEH. SA, and QED correspond to stronger binding, higher synthetic accessibility, and higher drug likgelikeness respectively. Following Bengio et al. [36], we introduced the parameter & which controls the sharpness (temperature) of the target distribution: p(x) a R(x),, increasing results in distribution skewed towards high-reward objects. We experimented with two & values, 32 and 96 (Figure (d)).
Reward normalization. in the molecule domain.
We used the following normalization rules for SEH, SA, and QED rewards
SEH aw//
SA 10-SAraw
QED QED raw
After training the base GFlowNets with the reward functions described above, we trained classifiers with Algorithm A.I. The classifier parameterized as graph neural network based on a graph transformer architecture [64]. Compared to the 2D grid domain (Section 6, "2D distributions via GFlowNet"), our approach. Instead, we analyzed the samples generated by the composed distributions. We sampled 5 000 molecules from each composed distribution obtained with approach as well as the base GFlowNets. We evaluated the sample collections with the two following strategies. Reward evaluation (Figure 3, Table 1) we analyzed the distributions of rewards across the sample collections. The goal is to see whether the composition of GFlowNets trained for different rewards leads noticeable changes in reward distribution. Distribution distance evaluation (Figure 4, Table G.5) we used the samples to estimate the pairwise distances between the distributions. Specifically, for given pair of distributions represented by two collections of samples D, =
Training and evaluation.
not directly evaluate the distributions obtained by
DB = B. || we computed the earth mover's distance d(D) A- D)) with ground molecule distance given by d(x, x') x (max ( (sxx,x), x'), 10-3))-1 -1, where s(x, x') E [0, 1] is the Tanimoto similarity over Morgan fingerprints of molecules x and x'.
Training details and hyperparameters. The base GFlowNet policies were parameterized as graph neural networks with Graph Transformer architecture [64]. We used 6 transformer layers with embedding of size 128. The input to the Graph Transformer was the graph of fragments with node attributes describing fragments and edge attributes describing attachment points of the edges.
The base GelowNets were trained with trajectory balance loss. We used Adam optimizer. For the policy network ps'('' f)), set the initial learning rate 0.0005 and exponentially decayed with the factor 2000/200000 For the log of the total flow log Zo we set the initial learning rate 0.0005 and
30

---

exponentially decayed with the factor 2 2000/50000 We trained the base GFlowNets for 15 000 steps with batch size 64 (64 trajectories per batch). In order to promote exploration trajectory sampling for the trajectory balance objective, we used the sampling policy which takes random actions (uniformly) with probability 0.1 but, otherwise, follows the current learned forward policy.
The classifier was parameterized as graph neural network with Graph Transformer architecture. We used 4 transformer layers with embedding size 128. The inputs to the classifier were the fragment graph, terminal state flag ((0, I), and log(</(-a)) (in case of parameterized operations). The classifier outputs were the logits of the joint label distribution (' Ynls) for non-terminal states and the logits of the marginal label distribution ( for terminal states x.
We trained the classifier with the loss described in Section 5.2. We used Adam with learning rate 0.001. We performed 15 000 training steps with batch size 8 (8 trajectories sampled from each of the base models per training step). We updated the target network parameters & as the exponential moving average (EMA) of & with the smoothing factor 0.995. We linearly increased the weight y(step) of the non-terminal state loss from 0 to throughout the first 4 000 steps and kept constant y = afterward. For the a a-parameterized version of the classifier (Section B), we used the following sampling scheme for a: U1-5.5,5.5), à
T+)
F.3 Colored MNIST Generation via Diffusion Models
The colored MNIST experiment in Section 6 ("Colored MNIST generation via diffusion models") follows the method for composing diffusion models introduced in Appendix A.2. The three base diffusion models were trained on colored MNIST digits generated from the original MNIST dataset. These colored digits were created by mapping MNIST images from their grayscale representation to either the red or green channel, leaving the other channels set to O. For Figure 5 we post-processed the red and green images generated by the base models and compositions into beige and cyan respectively, which are more accessible colors for colorblind people.
Models, training details, and hyperparameters. The base diffusson models were defined as VE SDEs [34]. Their score models were U-Net [65] networks consisting of 4 convolutional layers with 64, 128, 256, and 256 channels and 4 matching transposed convolutional layers. Time was encoded using 256-dimensional Gaussian random features [66]. The score model was trained using Adam optimizer [59] with a learning rate decreasing exponentially from 1 to 10-4, We performed 200 training steps with batch size 32.
The first classifier ē0,, y2(x,) was a convolutional network consisting of 2 convolutional layers with 64 and 96 channels and three hidden layers with 512, 256 and 256 units. This classifier is time-dependent and used 128-dimensional Gaussian random features to embed the time. The output was a 3x3 matrix encoding the predicted brg-probabilities. The classifier was trained on trajectories sampled from the reverse SDE of the base diffusion models using the AdaDelta optimizer [67] with a learning rate of 1.0. We performed 700 training steps with batch size 128. For the first 100 training steps the classifier was only trained on terminal samples.
The second conditional classifier T033,r F2. x,) was a similar convolutional network with 2 convolu- tional layers with 64 channels and two hidden layers with 256 units. This classifier is conditioned both on time and on (f1 12). The time variable was embedded used 128-dimensional Gaussian random features. The (2) variables were encoded using 1-hot encoding scheme. The output of the classifier was the three predicted log- probbabballitics for F3. Contrary to the first classifier, this one was not trained on the base diffusion models but rather on samples from the posterior p(xly;, F2). It's loss function was:
m
E where Wgg (R)) is estimated using the first classifier. The classifier was trained using the AdaDelta optimizer [67] with a learning rate of 0.I. We performed 200 training steps with batch size 128.
-.(()= =
Σ F==1
W (2)) log (3.35 P.
(116)
Sampling. Sampling from both the individual base models and the composition was done using the Predictor-Corrrcctor sampler [34]. We performed sampling over 500 time steps to generate the
31

---

0 /
/ 0
(a) P1
(b) Pz
P1 P2
P1 P2
(e) P1 D Pz
Figure G.I: Diffusion model composition on colored MNIST. (a.b) samples from base diffusion
models. (c-e) samples from the resulting harmonic mean and contrast compositions.
samples shown in Figure 5, The samples used train the classifier were generated using the same method.
When sampling from the composition we found that using scaling for the classifier guidance was generally necessary to achieve high-quality results. Without scaling, the norm of the gradient over the first and second classifier was too small relative the gradient predicted by the score function, and hence did not sufficiently steer the mixture towards samples from the posterior. Experimentally, we found that scaling factor 10 for the first classifier and scaling factor 75 for the second produced high quality results.
G Additional Results
G.1 Binary Operations for MNIST Digit Generation via Diffusion Models
Here we present a variant of the colored digit generation experiment from Section 6 ("Colored MNIST generation via diffusion models"). using 2 diffusion models. This allows us better illustrate the harmonic mean and contrast operations on this image domain. In a similar fashion to the experiment in Section 6 ("Colored MNIST generation via diffusion models"), ") trained two diffusion models to generate colored MNIST digits. was trained to generate red and green 0 digits and was trained generate green 0 and digits. As before, we used cest-processing map green cyan and red to beige.
Implementation details. The diffusion models used in this experiment and their training procedure exactly the same in Section F.3. The sampling method used to obtain samples from the base models and their compositions was also the same. We found that scaling the classifier guidance was generally required for high- nigh-quality results, and used scaling factor of 20 in this experiment.
The classifier was a convolutional network with convolutional layers consisting of and 64 channels and two hidden layers with 512 and 128 units. The classifier's time input was embedded using 188dimensioonl Gaussian random features. The output was a 2x2 matrix encoding the predicted log- rog-prbbabilities @(r,, y/ I x,). The classifier was trained on trajectories, sampled from the reverse SDE of the base diffusion models, using the AdaDelta optimizer [67] with learning rate of 0.1 and decay rate of 0.97. We performed 200 training steps with batch size 128. For the first 100 training steps, the classifier was only trained on terminal samples.
Results. Figure G.I shows samples obtained from the two trained diffusion models Pl P2 and from the harmonic mean and contrast compositions of these models. We observe that the harmonic mean generates only cyan zero digits, because this is the only type of digit which both p1 and P2 have high density. The contrast P1 1 P2 generates beige zero digits from P1- However, unlike P1 does not generate cyan zero digits, as P2 has high density there. The story is similar for pj O Pz. which generates cyan one digits from p2, but not zero digits due to p1 having high density over those.
G.2 MNIST Subset Generation via Diffusion Models
We report in this section on additional results for composing diffusion models on the standard MNIST dataset. We trained two diffusion models: P|o.....5) trained to generate MNIST digits 0 through 5,
32

---

4
3
3
0
3
7
5
9
4
8
7
4
H 7
4 4
4
5
3 22220000
9
7
9
3
3 4/
5
9
8
4
8
5
9
7
0
332
2
4
4
5
4
5
4
2
8
4
7
9
5
8
4
4
4 S
5 5 4
5
2
202
9
7
6
9
6
9
5
3
1
4
7
6
5
4
8
4 4
S
2
00000000
4
2
2
3
9
4
4
4
9
<6
6
9
4
13
0
7
8
7
7
7
0
d
0
5
5
P
7
7 7
4
6
7
9
4
V
2
0
6
9
8
/
3
5
9
6
8
6
3 2
0
9
7
9
5
4
6
2
O
3
8
6
6
5
9
4
4
2 0
9
9
7
4
6
4
(a) P(o....5)
(b) P(4....9)
(c) Pro.....1 PlA....1 (d) P(o.....) PI4....9)
(e) Pro....5 P P(4....9)
Figure G.2: Diffusion model composition on MNIST. (a.b) samples from base diffusion models. (c-e) samples from the resulting harmonic mean and contrast compositions.
and P) to generate MNIST digits 4 through 9. The training procedure and models used in this experiment were the same as Section G.
Figure G.2 shows samples obtained from the two diffusion models, from the harmonic mean, and from the contrast compositions of these models. We observe that the harmonic mean correctly generates mostly images on which both diffusion models' sampling distributions have high probability, i.c. digits 4 and 5. For the contrasts we see that in both cases digits are generated that have high probability under one model but low probability under the other. We observe some errors, namely some 9's being generated by the harmonic mean and some 4's being generated by the contrast P:o.....S) P P(4....9) This is likely because 4 and 9 are visually similar, causing the guiding classifier to misclassify them, and generate them under the wrong composition.
We also present binary operations between three distributions. In Figure G.3 and G.4, models even digits. lj models odd digits, and P2 models digits that divisible by 3. We color digits (0, 6) purple, {3, 9] blue, [4. 8 orange, and { (1,577 7) beige. In Figure G.3, harmonic mean of and P generates the digit 0 and 6, whereas the contrast of Po with P2 shows even digits non-divisible by
(Po O P2 = [4, 8) )). and odd numbers that are divisible by 3 (Po D = |3. 91). We observe that the samples from Po @ P2 inherit artifacts from the base generator for P2 (the thin digit 0), which shows the impact that the base models have on the composite distribution. In Figure G.4 we present similar
results between odd digits ((5, 3. 5. 7, 9). We noticed that samples from both P1 O P2 and P1 1 P2 includes a small number of the digit 3.
0
S
+
4
Y
2
6
4
3
9
0
7
9
3
0
3
6 0
6
0
6 {
0
2
2
2
2
8
1
3
3
9
3
4
3 3
3
3
9
0
7
8
2
2
4
3
9
0
9
9
9
6
6
0
6
0
0
è 6
0
4
2
8
4
8
8 4
2 8 8
4
9
9
8
2
4
3
8
2
4
3
3
5
9
6
3
3
6
6
6
0
0 ; 0 6
6
7
8
4
2 a
8
3
3
3
9
9
F
6
A
L
6
7
0
3
9
3
b
6
6
6
0
6
0
0
4
2
2
8
4
4
9
9
9
3
4
0
4
6
-
6
9
9
0
3
3
3
0
6 6
6
0
6
0
6
2
2
8
Y
4
2
8
5
3
5 3
3
3
33
2
6
8
6
3
9
0
0
6
0
2
2
2
8
4 2
4 4
8
3
3
3
3
0
2
8
2
3
3
6
9
9
6
6
6
6
0
0
6
8
6
4
2
7
8
3
4
0
2
9
O
3
3
3
0
G
6
0
6
6
2
8
9
2
8
4
3
3
3
(a) Poi Even Digits
(b) P2: (0. (0,3,66 6,
(c) Pa @
(d) Po
P
(c) Po P
Figure G.3: Composing even digits and multiples of three. (a.b) samples from base diffusion
models. (c-e) samples from the resulting harmonic mean and contrast compositions.
9
5
9
3
9
0
/
9
3
0
3
3
3
9
3
3
9
3
7
5
7
7 /
7
0 l
6
6
7
9
/
3
3
9
0
9
9
6
6
3
9
9
3
3
9 9
9
1
5
1
5
1
1
7
O
3
6
6
6
1
9
5
3
3 1,
9
9
6
3
3
9
3
9
9
9
3
4
f
7
1
7
1
S
0
6
0
L
6
6
7
1
6
0
3
9
9
3
f
9
9
7
7
1
7
6
0
6
4
6
6
7
5
7
9
9
3
5
3
3
9
3
1
7
5
7
7
5
0 4
4
6
6
0
9
1
?
3
3
9
9
9
3
3
9
9
1
/
7
5
6 6
5
9
3
6
9
9
6
6
9
9
3
5
3
3
1
5
1
7
5
6
6 6
9
9
3
3
9
9
3
7
5
5
9
O
6
(a) P;: Odd Digits
(b) P2: (0.3,6. 6,9)
(c) P1 8 P2
(d) P:
(e) Pj O
Figure G.4: Compusing odd digits and multiples of three. (a,b) samples from base diffusion models. (c-e) samples from the resulting harmonic mean and contrast compositions.
33

---

G.3 Chaining: Sequential Composition of Multiple Distributions
We present results on chaining binary composition operations sequentially on a custom colored MNIST dataset.
Setup. We start with three base generative models that are trained to generate Pt. P2 and P3 in
Figure G.5. Specifically, Pl is a uniform distribution over digits {0, 1, 2, 3, 4J is a uniform distribution over even digits {0. 2, 4, 6, 8) and P3 a uniform distribution over digits divisible by [0,3, 6, 9). Note that we use different color for each digit consistent across Pt. P2. P3. Our goal is to produce combinations of chained binary operations involving all three distributions, where two of them were combined first, then in a second step, combined with the third distribution through either harmonic mean @ or contrast
Binary Classifier Training. Consider, for example, the operation (p) @ P2) P3- We use the same classifier training procedure for P1 versus P2, as well as the composite model (P) @ P₂) versus P3+ except that in the later case we sample from composite model whole. Our classifier training simultaneously optimizes the terminal classifier and the intermediate state classifier.
Implementation Detains. We adapted diffusion model training code for the base distributions from [33]. Our diffusion model used a UNet backbone with four latent layers on both the contracting and the expansive path. The contracting channel dimensions were [64, 128. 256, 256] with the kernel size 3, and strides 21, The time embedding used a mixture of 128 sine and cosine feature pairs, with total of 256 dimensions. These features were passed through sigmoid activation and then expanded using a different linear head for each layer. The activations were then added to the 2D feature maps at each layer bhannel-wise. We used fixed learning rate of 0.01 with Adam optimizer [59].
We adapted the classifier training code from the MNIST example Pytorch [68]. Our binary classifier has two latent layers with channel dimensions [32, 64), stride and kernel width of 3. We use dropout on both layers: P1 25%, P2 = 50%. We train the network for 200 epochs on data sampled online in batches of 256 from each source model, and treat the composite model in the second step the same way. We use the Adadelta [67] optimizer with the default setting learning rate 1.0.
Sampling. We generate samples according to Appendix A.2 and multiply the gradient by 20.
Results. Row from Figure G.5 contains mostly zeros with few exceptions. This is in agreement with harmonic mean being symmetric. In @ (p2 p3), digits that are outside of the intersection still appear with thin strokes.
34

---

0 00003
2 2
6
Y A 2
0
6
0
3 5 4
0 2
0400
0
5
5
3
2
6
0
O
5
04000
O O
(a) P.
(0
5}
(b) Pa (,62,46,88)
(c) (0,3,6 6,
(d) P1 X P1
(c) P:
(f) X P
2
4
4
+
&
9
6
5
3
8
4
2
6
3
5
3
3
2
2
//
9
6
f
9
4
6
9
R
6
3
3
5
3
2
4
2
9
9 9
A
6 6
5
5
5
3
4
4
2
6
6
4
6 6
8
3
2 2
9
P
4
(g) P2
(h)
P 1 P
i
P
O P3
Dz
(k) P2
(1)
Ps
6 C G
OO 0
0
0 00000000
3 3
0
P1
2 (p & Pb)
P2 & (p)
&
p3)
(p)
& Pb)
5 5
2
S
O
3 /
5
0
5
2
2
8
2 AS/
5
5
/
6
8
5
0
2
0 5 3 5
3
4
3
4 5 3
0
4
2.
5
4
2
0 0
0
O
4
O
5
4
O
8
2
2
O
8
2
2
y
Y
2
O
(p:
P.)
(p>
8 Pb)
(p:
P
Pb
1
(p
Ps)
(p)
X P.)
4
(
Ps)
C
A
A
O 6 0
6
6
9
9
9
0 x
2
2
12
3
3
3
2
2
2
1 0
0
3
6
9
0
2
0 9 6
4
6
3
6
0
7
4
A
0 6 6
A
9
2
4
6 6 0
66666
4
0 100000
3
4
0 3
0
2
4
3
3
4
4
0 OOC
Y
2,00 (pi O P.)
Pa
1
(p)
8 Pb)
(p: Pa)
& (p.
1 Ps)
X (p2 1 P.)
P:
X
(p)
Ps)
0
3
1OOD 0
3 3
3
3
3
0 0
00
V 2
0
3
3
8 3
6 6
3
0
6000
2
2
6
3
3
0
3
0 6
3
0 066000
2
3
D 0600
4
3
0
4
② (p, O Pa)
P:
8
(p.
1 P2)
② (p. Pz)
P
8
(p2 Pg)
(p2 X Pa)
P1
O
(p)
Ps)
Figure G.5: Chaining Binary Operations. (a-c) Samples from pre-trained diffusion models. (d-I) Samples from binary compositions. (row 3) The harmonic mean between all three. (row 4 and beyond) various ways to chain the operations. Parentheses indicate the order of composition.
35

---

Distribution distance (L,)
Non-terminn state loss
Terminal state loss
0.725
54.00
1.8
0.700
Terminal stste loss
52.44
16-
Liffh + jos onfe + fpa))
0.675
Non-temminal state loss
50.89
1.4
L,n @ Giti pall
0.630
09.33
12-
0.000
L,in O., ey GTi Pon)
0.975
0.990
11
04
0.825
0.900
10
1500
10000
2500
5000
7500
10000
12300
15000
Steps
Steps
Figure G.6: Training curves of the classifier 24 F21-) in GelowNet 2D grid domain. The experi- mental setup corresponds to Section 6 ("2D distributions via GFlowNet") and Figure 2 (top row). Left: Terminal state loss and non-terminal state loss (as defined in Algorithm A.1) as functions the number of training steps. Right: L, distance between learned distributions (compositions obtained through classifier-based mixture and guidance) and ground-trun composition distributions as the
function of the number of training steps. 1. q)
ExEx lp(x) (xx)].
Non-termina state Toss
Stale loss
Terminal state loss
0.8
15:00
Terminal state loss
1.250
4.90
077
Terminal
39.76
1.125
Teriinal
4.05
Noc terminal sizte loss
33.75
Non serminal state loss
26:12
00730
2.70
23.50
0.3
16.58
0.2
11.25
0,250
0.00
0.125
2500
5000
7500
10000
12500
15000
50
75
100
125
375
200
Steps
Steps
Figure G.7: Training curves of the classifier
Figure G.8: Training curves of the classifier Qu(y). y21-) in diffusion MNIST image gener- ation domain. The experimental setup corre- sponds Section G.I and Figure G.1. The curves show terminal state loss and non-termin terminal state loss (as defined in equations (26), (28)) as func- tions of the number of training steps. The non- terminal loss optimization begins after the first 100 training steps (shown by the black dashed line).
(211) in GFlowNet molecule generation domain. The experimental setup corresponds to Section ("Molecule generation via GelowNet") and Figure (a-c). The curves show terminal state loss and mon-telminal state loss (as defined in Algorithm A) as functions of the number of training steps.
G.4 Classifier Learning Curves and Training Time
We empirically evaluated classifier training time and learning curves. The results are shown in Figures G.6, G.7, G.8 and Tables G.1, G.2.
Figures G.6, G.7, G.8 show the cross-entropy loss of the classifier for terminal (16) and non-terminal states (18) as a function of the number of training steps for the GFlowNet 2D grid domain, the molecular generation domain, and the Colored MNIST digits domain respectively. They show that the loss drops quickly but remains above O. Figure G.6 further shows the distances between the learned compositions and the ground truth distributions as a function of the number of training steps of the classifier. For all compasitions, as the classifier training progresses, the distance to the ground truth distribution decreases. Compared to the distance at initialization we observe almost an order of magnitude distance reduction by the end of the training.
The runtime of classifier training shown in Tables G.1 and G.2. We report the total runtime, as well as separate measurements for the time spent sampling trajectories and training the classifier. The classifier training time is comparable to the base generative model training time. However, most of the classifier training time (more than 70%, or even 90%) was spent on sampling trajectories from base generative models. Our implementation of the training could improved this regard, e.g. by sampling a smaller set of trajectories once and re-using trajectories for training and by reducing the number of training steps (the loss curves in Figures G.6, G.7, G.8 show that classification losses platcau quickly).
36

---

Table G.I: Summary of base GFlowNet and clas- sifier training time molecule generation do- main. The experimental setup corresponds to Section 6 ("Molecule generation via GFlowNet") and Figure (a-c). All models were trained with a single GeForce RTX 2080 Ti GPU.
Table G.2: Summary of base diffusion and clas- sifier training time in MNIST image generation domain. The experimental setup corresponds to Section G. and Figure G.1. All models were trained with a single Tesla V100 GPU.
Base GelowNet training steps
20000
Base diffcasoon training steps
200
Base GPlowNer batch size
64
Base diffussonn batch size
32
Base Geooweer training elapsed real time
6h 47m 11s
Base diffusion training elapsed real time
10ta 6s
Classlier traiming
15 5000
Chasilier training
200
Classifer basch size
trajectories per model
Chassifer busch
128 trajectoiiss per model
stops per trajectory)
Clessffer triining elassed time
9h 19s
Classifeer traininng elapsed time
300
Chesllier training data generation time
6h 35m S8s (73%)
Chasilier training data generation time
20m (97%)
Table G.3: Average pairwise similarity [36] of molecules generated by GelowNets trained on 'SEH', 'SA', 'QED' rewards at different values of B. For each combination (reward, fi a GFlowNet was trained with the corresponding reward R(x).. Then, 5 500 molecules were generated. The numbers in the table reflect the average pairwise Tanimoto similarity of top 1 0000 molecules (selected according to the target reward function).
SEH
SA
QED
0 = 1
0,527
0.539
0.480
β 4
0.529
0.527
0.464
β = 10
0,535
0.500
0.438
$ 16
0.548
0.465
0.422
0 = 32
0,585
0.411
0.398
β 96
0.618
0.358
0.404
Table G.4: Number of Tanimotoo-spparteed modes found above reward threshold. For each combination (reward, (8 a GFlowNet was trained with the corresponding reward R(x),, and then 000 molecules were generated. Cell format is "A/ B", where A is the number of Tanimoto-seppratee modes found above the reward threshold, and B is the total number of generated molecules above the threshold. Analogously to Figure 14 in [36], we consider having found new mode representative when new molecule has Tanimoto similarity smaller than 0.7 to every previously found mode's representative molecule. Reward thresholds (in [0, 1], normalized values) are 'SEH': 0.875, 'SA': 0.75, 'QED:: 0.75. Note that the normalized threshold of 0.875 for 'SEH' corresponds to the unnormalized threshold of 7 used in [36].
SEH
SA
QED
15 /
37 /
37
0 0/
4
12
17
82 /
82
0
0
A 10
85 /
109
332 / 337
/
16
190 / 280
886 / 910
253
253
B = 32
992 99211
1821
2859
/ /3880
3067
3124
96
1619
4609
4268 / 4983
4470
4980
G.5 Analysis of Sample Diversity of Base GFlowNets in Molecule Generation Domain
In order to assess the effect of the reward exponent & on mode coverage and sample diversity, we evaluated samples generated from GFlowNets pre-trained with different values of B. The results are in Tables G.3 and G.4. The details of the evaluation and the reported metrics are described in the table captions. As expected, larger reward exponents shift the learned distributions towards high-scoring molecules (the total number of molecules with scores above the threshold increases). For 'SA' and 'QED' models we don't observe negative effects of large / on sample diversity and mode coverage: the average pairwise similarity of top 1 000 molecules doesn't grow as f increases and the ratio of Tanimoto-seppaa -separate modes remains high. For 'SEH' models we observe a gradual increase in the average pairwise similarity of top I 000 molecules and gradual decrease in the ratio of Taniaoto-separated modes. However, the total number of separated modes grows as & increases, which indicates that larger reward exponents do not lead to mode dropping.
37

---

G.6
Summary of Pairwise Distribution Distances in Molecule Generation Domain
Table G,5: Estimated pairwise earth mover's distances between distributions shown in Table
y-SA
y-QED
y-SPH,5A
y-SER.QeD
y-SA,QRD
y-SEH.SA.QED
y-SEH y-SAx1
y-QFD 3
5.77
E33A
4.42
5.88
5.15
2.55
3.89
>EQED
5.77
5.88
5.40
4,02
3.85
4.30
5.80
5.90
3.20
SEILSA
3.39
3.26
5.40
4,25
4,19
1.68
3.59
3.30
5.39
SEILQEED
4.20
5.15
4,02
4.25
3.80
3.67
4.22
5.19
4.00
SEILSALEED
S
2,46
4.39
5,80
3.39
4.22
1.91
4,43
5.73
SAX
4.44
2.55
5.90
3.30
5.19
4.59
4.33
4.43
0
3,90
X
573
5.89
3.20
5.39
4,00
3.87
4.30
5.73
5.01
38


Reviews:
[{"review_id": "2XavXbnHpJ", "reviewer": "Reviewer_UsMn", "summary": "This paper introduces a method to combine sequential generative models, in this case GFlowNets, so as to create new distributions from base models. This is done by training classifiers that are then used to guide sampling. The method is tested on a simple grid and a molecular domain (emulating the problem of the paper that introduced GFlowNet), as well as on MNIST with diffusion.", "questions": "l211 and around, I’m not sure I understand the move to approximate $p(y_i|s)$ with $p(y_i|x)$, even when $s$ and $x$ are related by $\\tau =(s_0,…,s,…x)$ a valid trajectory. How or when are these two quantities interchangeable?\nThe objective in 12 involves $w_i$, but the paragraph following (11) seems to suggest that $Q(y_2, .., y_n|x)$ can be replaced by the $w_i$s. So I’m confused whether (12) uses the loss described in (11) or a modified version of it. Another worrying aspect of eq (12) is that there’s an $O(nm)$ sum, which sounds like it can get expensive, and there’s a product of $(n-1)$ probabilities, which sounds like it can get awfully numerically unstable. I’m surprised the authors have been able to train models at all. Are there any tricks involved?\n\nThe appendix is incomplete. In fact, part of the proposed contribution of the paper is to provide a combination method for diffusion methods. This is never quite explained properly; readers are directed to appendix D which is empty. In addition, although the propositions and theorems are analogs of past work, it was quite surprising to find theorems with no proofs in a paper. Even if the proof is almost identical to prior work, reproducing it with proper credit seems like the least thing to do; in this case there are nuances with the GFlowNet framework that are left unexplained.\n\nThe authors already highlight this limitation, but this seems like something quite fundamental that is for some reason not reported: **How expensive is it to train the classifier**? If it’s just as expensive as training a new generative model, then there’s little incentive to use the proposed method. I’d like not to have to take the authors’ word for it and instead see empirical evidence, training curves, wall time, and so on.\n\nAnother missing result is some validation that the learned distributions are the expected ones (e.g. a plot showing that the JS divergence goes to 0 with more training/capacity) at least on a toy setting like the grid environment.\n\nI really appreciate what the paper is trying to accomplish, and the empirical results seem very nice (although lacking some crucial results). I’m not sure though that the proposed method is correct, and with no proofs or extra details of why this should work I’m really inclined to reject this paper. Happy to engage in conversation with the authors of course.", "limitations": "The authors have addressed some of the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is moderately easy to read, although some of its results were not immediately clear to me so I spent quite some time doodling on paper to convince myself that the propositions were reasonable.\n\nWhat the paper proposes and seems to be able to achieve empirically is very interesting. Combining generative models in the ways shown here could be an amazing multiplier of large pretrained models.", "weaknesses": "Generally the paper is not making a good job of convincing the readers that the proposed method should work at the theory level, and that the effort of combining distributions by training a classifier is worth it (compared to retraining a generative model).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and feedback on our paper. \n\n> The appendix is incomplete\n\nThe main-text PDF included an incomplete draft of the appendix by mistake. We apologize for the accidentally caused confusion. The complete appendix is provided in the supplementary zip-archive (can be downloaded from the openreview page). The proofs of all theoretical results as well as the formulation of the approach for diffusion models are in the supplement.\n\n> How expensive is it to train the classifier?\n\nFollowing your suggestion, we empirically evaluated classifier training time and learning curves. The results are shown in Figures R.1, R.2, R.3, and Tables R.1, R.2 in the rebuttal PDF. \n\nThe classifier training time is comparable to the base generative model training time. However, most of the classifier training time (more than 70%, or even 90%) was spent on sampling trajectories from the base generative models. Our implementation of the training could be improved in this regard, e.g. by sampling a smaller set of trajectories once and re-using trajectories for training and by reducing the number of training steps (the loss curves show that classification losses plateau quickly).\n\nWe would like to note that the composite distributions can not be realized by training new generative models for the target composition directly. For diffusion models, training requires data from the target distribution, which is not available without realizing the composition first. For GFlowNets, training the model requires access to the composite reward (which might not be available). Even if the base reward functions ($R_1,R_2$) are available, the compositions (i.e. harmonic mean) can’t be expressed through the rewards only, i.e. $\\frac{R_1(x)R_2(x)}{R_1(x)+R_2(x)}$ is not a valid reward for the harmonic mean, since rewards are unnormalized.\n\nWe also note, that after the classifier is trained once, it can be used to construct multiple compositions of the base distributions.\n\n> Validation that learned distributions are the expected ones\n\nFigure R.1 (right) in the rebuttal PDF, shows the evolution of the distances between learned compositions (realized via classifier) and the ground-truth composition distributions (in 2D grid domain). For all compositions, as the classifier training progresses, the distance to the ground truth distribution decreases. Compared to the distance at initialization we observe almost an order of magnitude distance reduction by the end of the training.\n\n> l211 and around, I’m not sure I understand the move to approximate $p(y_i| s)$ with $p(y_i|x)$. How or when are these two quantities interchangeable?\n\nWe would like to clarify that we do not propose to approximate $p(y_i|s)$ with $p(y_i|x)$. The objective in eq. (12) uses $\\ell(\\hat\\tau,\\hat y_1,\\ldots,\\hat y_n;\\phi)$. The value of $\\ell$ appearing in (12) is exactly the same as given in eq. (11).  We arrive at eq. (12) by combining several ideas.\n\n1. Our goal is to train a classifier $\\tilde Q(y_1,\\dots,y_n|s)$. This classifier can be obtained as the optimal solution of\n$\\min\\limits_\\phi \\operatorname*{\\mathbb{E}}\\limits_{\\hat\\tau,\\hat y_1,\\dots,\\hat y_n\\sim\\tilde p(\\tau,y_1,\\dots,y_n)}\\ell(\\hat\\tau,\\hat y_1,\\dots,\\hat y_n;\\phi),$\nwhere $\\ell$ is defined in eq. (11). We can obtain an unbiased estimate of the loss (and its gradient) by sampling $(\\hat\\tau,\\hat y_1,\\dots,\\hat y_n)$ and evaluating (11) directly. The challenge is not in computing (11), but in the expectation over $(\\tau,y_1,\\dots,y_n)$. The steps described in the paragraphs following (11) were introduced to obtain an estimate of this expectation.\n\n2. The expectation above can be expressed as\n$\\operatorname*{\\mathbb{E}}\\limits_{\\hat\\tau, \\hat y_1\\sim\\tilde p(\\tau,y_1)}\\left[\\sum\\limits\\_{\\hat y_2=1}^m\\dots\\sum\\limits\\_{\\hat y_n=1}^m \\left(\\prod\\limits\\_{i=2}^n \\tilde p(y_i=\\hat y_i|x=\\hat x)\\right)\\ell(\\hat\\tau,\\hat y_1,\\dots,\\hat y_n;\\phi)\\right],$\nwhere we re-wrote the expectation over $(y_2,\\dots,y_n) | \\tau$ in the from “expectation” = sum(“probability” * “value”). The expectation over $(\\tau,y_1)$ can be estimated by sampling pairs $(\\hat\\tau,\\hat y_1)$ as described in the paragraph after eq. (11). The only missing part is the probabilities $\\tilde p(y_i=\\hat y_i|x=\\hat x)$ which are not directly available.\n\n3. Our proposal is to approximate these probabilities as $\\tilde p(y_1=j|x=\\hat x)\\approx w_j(\\hat x;\\phi)=\\tilde Q_\\phi(y_1=j|x=\\hat x)$. The idea here is that the terminal state classifier $\\tilde Q_\\phi(y_1|x)$, when trained to optimality, produces outputs exactly equal to the probabilities $\\tilde p(y_1|x)$.\n\n4. Steps 1-3, give a procedure where the computation of the non-terminal state classification loss requires access to the terminal state classifier. As we described in the paragraph preceding eq. (12), we propose to train non-terminal and terminal classifiers simultaneously and introduce “target network” parameters. The weights $w$ are computed by the target network $\\tilde Q_{\\bar\\phi}$.\n\n> Another worrying aspect of eq (12) is that there’s an $O(nm)$ sum, which sounds like it can get expensive\n\nIn our experiments $n$ and $m$ were at most $3$, and the computational cost of summation over $\\hat y_2,\\dots,\\hat y_n$ was small. In general, one could trade off estimation accuracy for improved speed by replacing the summation with Monte-Carlo estimation. In this case, the values $\\hat y$ are sampled from the categorical distributions $Q_{\\bar \\phi}(y|x)$. Note that labels can be sampled in parallel since $y_i$ are independent given $x$.\n\n> There’s a product of $(n-1)$ probabilities, which sounds like it can get awfully numerically unstable. I’m surprised the authors have been able to train models at all. Are there any tricks involved?\n\nWe only employed the standard techniques for improving the numerical stability of operations on probabilities (re-parameterization in log-probabilities) and did not observe any numerical instability issues."}, {"review_id": "vXpLdCNY6O", "reviewer": "Reviewer_ZgyF", "summary": "The current paper focuses on the challenge of composition generation from pretrained generative models, with a specific focus on GFlowNets and Diffusion models. In comparison to prior literature, two novel compositionality operations are introduced for generating samples that are simultaneously likely according to two generative models or likely per a subset and unlikely per the remaining models. This is a strict generalization of operations introduced in prior work on composition of energy-based models. Practically, the operations are instantiated via a framework motivated by classifier guidance in diffusion models. Experiments are conducted on a molecule generation application and a colored MNIST problem.\n\nRebuttals acknowledgment: I had a good view of the paper before rebuttals and the authors' response to my questions was fair. I continue to keep my score accordingly.", "questions": "As noted in the introduction, use of large scale generative models has been seen in several tasks in the foundation model era of machine learning. Given the limited empirical evaluation, I would like to gather the authors' thoughts on use of their frameworks for, say, controllable generation of language via LLMs or controllable generation of images via text-diffusion models (e.g., what would the model show for known compositionality failures of Dall-e and related methods? See [1]).\n\n[1] Conwell and Ullman, 2022. https://arxiv.org/abs/2208.00005.", "limitations": "- A clear note of what is meant by compositionality would help in this work, since the term is extremely overloaded. The experiments currently reported focus on what would be called systematicity or systematic generalization in my opinion (see Hupkes et al., \"Compositionality decomposed\"), but other valuable forms of compositionality, e.g., productive generalization, will arguably require some \"chaining\" operator that allows prior generated states to be fed into the model for generating the next state. Since the work focuses on GFlowNets and diffusion models, where a notion of sequential generation of intermediate states is present, arguably authors can use their defined operations to perform productive generalization as well? I would appreciate if the scope of this paper is clearly discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "I really like this work! The contributions are simple and straightforward, but very interesting. The formalization, generalized operations, and relation to classifier guidance were exciting to read through. The authors also appropriately acknowledge the limitations of their work, specifically the need for sufficiently strong component models.", "weaknesses": "- My biggest apprehension is limited experimental investigation, which would raise the quality of this paper quite strongly in my opinion. I do not hold this to be a strong weakness though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We address specific questions below:\n\n> I would like to gather the authors' thoughts on use of their frameworks for, say, controllable generation of language via LLMs or controllable generation of images via text-diffusion models (e.g., what would the model show for known compositionality failures of Dall-e and related methods?\n\nThanks for recommending the Conwell *et.al.* paper! The CLIP training objective treats language prompts as a \"bags-of-words\" for compute reasons. Therefore, it is not surprising that the image generated by DALL-E 2 (and family) lack relational understanding. This type of constraint is the norm, not the exception in large model training, which is why we believe finding better ways to model relationships is an important area to study.\n\nThe type of relationships mentioned by Conwell *et.al.* are often mappable to spatial arrangements. It is not difficult to envision learning multiple base models that each model specific relationships. Complex relational queries could then be represented as a composition of appropriate base models, and samples capturing these relations could be generated using the method we have proposed here. This is similar to prior work [27] which used multiple EBMs to model a number of relationships, and found that samples from appropriate compositions of these EBMs reproduced the target relationships significantly more faithfully than StyleGAN2 conditioned on a textual encoding of the relationships. We leave this to follow-up works, as it warrants comprehensive evaluation.\n\n> A clear note of what is meant by compositionality would help. I would appreciate if the scope of this paper is clearly discussed.\n\nSince submission we have worked hard to further clarify our exposition on compositional sculpting and our method. We have clarified that we focus on a narrow but well-defined type of composition where we look to algebraically combine (compose) probability densities in a controllable fashion, such that we can emphasize or de-emphasize regions in the composition where specific base distributions have high density. The harmonic mean and contrast operations we highlight in the paper are specific instance of this. Our paper focuses on a setting where we have access to GFlowNets or diffusion models which can generate samples from those probability distributions we wish to compose.\n\nThe iterative nature of GFlowNets and diffusion models is preserved when composed using our method. Thus, if the base models exhibit productive generalization, so will their compositions. In addition, we would like to highlight that compositions themselves can be chained as well. As the compositions correspond to valid GFlowNets or diffusion models, one can compose these compositions with other GFlowNets or diffusion models."}, {"review_id": "3iCgZB7RUf", "reviewer": "Reviewer_VKR5", "summary": "The paper proposes a method to compose multiple iterative generative models, i.e., either multiple GFlowNets or multiple diffusion models. The idea starts out with a mixture model over the generative models. Then, one can construct a categorical distribution over the generative models that tells us which model a sample originated from. By adapting classifier guidance to GFlowNets, the proposed method can compose multiple models in a way that allows both emphasizing or de-emphasizing specific models by treating the different generative models as different classes. On a diverse set of (toyish) experiments the method is shown to be effective for both GFlowNets and diffusion models.", "questions": "All state-of-the-art diffusion models for images allow for text conditioning. Conditioning on different texts can be viewed as creating multiple different generative models from the viewpoint of this paper. I believe this is a more realistic setting than assuming that one would want to combine multiple diffusion models that were trained entirely separately from one another. It would be interesting to see some evaluations of this setting. A cheap strategy to obtain a classifier for this could be to fine-tune (part of) the CLIP model.", "limitations": "The main limitation is inherited from classifier guidance: the need to train a classifier on intermediate states. This is already mentioned in the paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is very interesting. In particular, the part where the question \"which model was this sampled from\" is treated as a classification task for the purpose of compositional generation.\n\nThe paper addresses a very important problem with high impact. \n\nThe presentation is easy to follow and the text is well-written.\n\n ", "weaknesses": "The experiments clearly demonstrate the effectiveness and versatility of the proposed method. Though, the experiments are limited to toyish settings and I believe more complicated settings would greatly enhance the impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The experiments clearly demonstrate the effectiveness and versatility of the proposed method.\n\nThank you for the positive feedback and insightful comments! \n    \n> The experiments are limited to toyish settings and I believe more complicated settings would greatly enhance the impact of this work.\n    \nWe naturally agree. Model composition with diffusion models and GFlowNets is still new. Our goal for this work was to formulate the problem and motivate further empirical study through our illustrative examples. We consider larger scale problems, such as composing state of the art image generation models to be a natural next step.\n    \n> Conditioning on different texts can be viewed as creating multiple different generative models. It would be interesting to see some evaluations of this setting.\n    \nWe did consider using conditioned models to emulate multiple distributions. One particularly interesting application is in safety, where one could compose safety or moral constraints on an existing text-conditioned generative model to remove harmful content.\n    \nThat said, we believe studies like this warrant separate comprehensive evaluation. In the current work, we focus on the development of a new approach to model composition, its theoretical foundation, and empirical validation."}, {"review_id": "OVnnabbGp9", "reviewer": "Reviewer_eEwT", "summary": "The paper studies the problem of composing independently trained generative processes of diffusion-based generative models and GFlowNets. The paper considers a setting where one has access to $m$ pre-trained samplers for $\\{p_i(x)\\}_{i=1}^m$, and the goal is to obtain a sampler which corresponds to a composition of these processes. Specifically, the authors consider two ways of composing the processes, namely harmonic mean: where the likelihood of the composition is high only where the component processes have high likelihood and contrast The authors frame this as sampling from a conditional distribution $p(x|y)$, where y is an observation which denotes the index of the process a sample is generated from. This results in a procedure analogous to classifier guidance which is popular in the diffusion literature. The authors show how the classifier guidance results in sampling from the desired composition. A critical component of this procedure is learning the classifier $p(y|s)$, for which the authors propose a MLE based procedure using trajectories from the base model. The authors first validate their approach on some synthetic tasks on a 2D grid, followed by experiments on small molecule generation with GFlowNets and colored MNIST with diffusion. ", "questions": "* Could you please address the questions about the analysis of the effect of the classifier along with some runtime details? \n* What are the challenges in applying the method to more complicated tasks with diffusion models? \n* One natural question I had after reading the paper is that since the method encompasses GFlowNets and diffusion models, is it conceivable to be able to compose a mixture of GFlowNets and diffusion models on mixed continuous/discrete tasks? \n* In the experiments, the authors consider a maximum of 3 models being composed. Is there a reason to be limited to composing only 3 models? If not at least in the 2D example an experiment with more models might be helpful. \n\nMinor:\n* The main PDF on OpenReview appears to have some old version of the Appendix at the end. I assume that was by mistake? \n* In several places there is inconsistent usage of “GFLowNet” in place of “GFlowNet”. ", "limitations": "The authors already discuss some limitations of the approach - effect of the classifier as well as the quality of the underlying models. I would also add limited evaluation and lack of comparison to existing approaches. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper studies an interesting question - which is relevant to the community. In particular, methods to leverage pre-trained models for various downstream tasks are becoming increasingly important with the growing adoption of pre-trained models for various domains. \n* Since similar classifier guidance approaches have been studied extensively in the literature on diffusion models, the novelty is relatively limited. Nonetheless, there are several technical aspects of the approach such as the classifier training scheme that are novel (to the best of my knowledge). \n* The proposed method is relatively simple conceptually, and in terms of implementation. \n* The experiments are well designed, and the results are quite promising, albeit with some caveats I mention below. \n* The paper overall is quite well written and easy to follow. I also appreciate the authors including the code with the submission. ", "weaknesses": "* As the authors discuss in Section 3, their theoretical analysis is analogous to classifier guidance in diffusion models. On the other hand, [1] establishes equivalence between GFlowNets and diffusion models. As a results, it seems to me that the insights provided by the theoretical analysis aren’t particularly novel even though the path to achieving them was different (which could be seen as useful on it’s own)\n* The experiments on diffusion are limited to a simple coloured MNIST task, with no baselines from the classifier-guided diffusion literature. \n* The central aspect of the approach is learning the classifiers, however there is no analysis on the classifiers - e.g. how accurate are the classifiers? what is the effect of the classifier performance on the results of the composition? How does the training of the classifier compare to simply training a GFlowNet from scratch in terms of runtime? \n* (Minor) A simple baseline that is missing in the experiments is training a GFlowNet with the appropriate composition from scratch. \n\n[1] Unifying Generative Models with GFlowNets and Beyond. Zhang et al. 2022. arXiv:2209.02606", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper and your insightful feedback. We have addressed the main questions and concerns you have raised below.\n\n> Classifier guidance for GFlowNets is not too novel considering the equivalence to diffusion models.\n\nThis is a fair point. However, despite the known connection, classifier guidance for GFlowNets has not been proposed in prior work as far as we know. Our focus is on efficiently generating samples from compositions of iterative generative models using classifier guidance, which necessitated introducing classifier guidance for GFlowNets.\n\n> No analysis of the classifiers. How accurate are they, how does accuracy affect performance?\n\nThe quality of the classifier is fundamental to the method, as the classifier guides the generative process of the mixture of base models $p_i(s)$ towards sampling from the posterior $\\tilde p(s | y_1, \\dots, y_n)$. If the classifier is poor, the sampling distribution will not match $\\tilde p(x | y_1, \\dots, y_n)$. The primary concern then is that the classifier as a function of the state $\\tilde Q(y_1, \\ldots, y_n \\vert s)$ should be as close as possible to the ground-truth $\\tilde p(y_1, \\ldots, y_n \\vert s)$ rather than the absolute value of the classification loss (though we do want the loss the be as low as possible). In the experiments we have considered, there is a theoretical lower bound on the classification loss as the base distributions we have considered have some overlap.\n\nWe have collected additional empirical results regarding classifier accuracy and its effect on the constructed composition in the rebuttal PDF. We will also add these results to the paper. Figures R.1, R.2, and R.3 show the cross-entropy loss of the classifier for terminal and non-terminal states (eqs (10) and (12)) as a function of the number of training steps for the GFlowNet 2D grid domain, the molecular generation domain, and the Colored MNIST digits domain respectively. They show that the loss drops quickly but remains above 0. Figure R.1 further shows the distance between the composition and the ground truth as a function of the number of training steps for the classifier. This shows that the distance to the ground truth falls quickly in conjunction with the loss.\n\n> How does the training of the classifier compare to simply training a GFlowNet with the appropriate composition from scratch in terms of runtime?\n\nIn addition to the new training curves in figures R.1-R.3, we have added new results regarding the runtime of classifier training in Tables R.1 and R.2 of the rebuttal PDF. These tables show the total runtime, as well as separate measurements for the time spent sampling trajectories and training the classifier.\n\nThe runtime for training the classifier is of the same order of magnitude as training the base models. The main computational expense when training the classifier comes from sampling from the base models, comprising 70%-90% of the runtime, rather than training the classifier itself. In this regard there is certainly room for improvement in the training procedure, e.g. by sampling fewer trajectories, by training on individual trajectories multiple times, and by reducing the number of training steps (Figures R.1-R.3 show that loss plateaus quickly in all cases).\n\nHowever, we would like to stress that training the appropriate composition from scratch is far from trivial, and that our approach is more general. Specifically, in the case of GFlowNets, training the model requires access to the composite reward, which may not be available. Even if the base reward functions ($R_1, R_2$) are available, compositions can’t be expressed through the rewards only because operations on rewards are not analogous to operations on probabilities. For example, $\\frac{R_1(x) R_2(x)}{R_1(x) + R_2(x)}$ is not a valid reward for the harmonic mean. In the case of diffusion models, training requires samples from the composition, which are not available before realizing the composition. In addition, once the classifier has been trained, it can be used to sample from all supported compositions of the base models, rather than a single composition as would be the case for a GFlowNet or diffusion model trained from scratch. We will make these points clear in the paper.\n\n> What are the challenges in applying the method to more complicated tasks with diffusion models?\n\nThe challenges are primarily practical. More complex diffusion models are more expensive to train and sample from. More expensive sampling also makes training the classifier and sampling from the composition more expensive. More complex compositions involving more base models and more observations require either more classifiers or larger classifiers (with more outputs).\n\n> Is it conceivable to be able to compose a mixture of GFlowNets and diffusion models on mixed continuous/discrete tasks?\n\nThis is certainly interesting future work. For the current method, this is not possible as the base models must have a shared domain. However, composing diffusion models with GFlowNets with continuous domain, or GFlowNets with discrete diffusion models is certainly possible.\n\n> Is there a reason to be limited to composing only 3 models?\n\nNo; there is no theoretical limit on the number of base models. Generally, we felt that 3 base models is a reasonably realistic setting.\n\n> The main PDF on OpenReview appears to have some old version of the Appendix at the end. I assume that was by mistake?\n\nYes, this was an oversight on our part. We apologize for any confusion this may have caused. The correct appendix can be found in the supplement."}, {"review_id": "gf16B48iOb", "reviewer": "Reviewer_Ean9", "summary": "The paper describes a way in which, given sequential samplers from multiple probability distributions, a combination of the samplers can be used to sample from a composition of the distributions. To be precise, the sequential samplers are either GFlowNets or diffusion models, the combination of samplers is a weighted combination of action distributions at each intermediate sampling step, and the composition of distributions can be defined by simple soft conjunction and set difference operators. This is demonstrated in toy illustratory experiments and multiobjective molecule synthesis (GFlowNet) and MNIST with digit class and colour attributes (diffusion).", "questions": "Please see \"weaknesses\" above.\n", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "From the perspective of someone who works on both GFlowNets and diffusion models, this is a very well-written paper. \n- The text reads naturally and the right amount of detail is given in the main text. There is a good choice of illustrations to help the reader.\n- The composition of multiple GFlowNets has not been considered before and could be useful, especially in multiobjective problems. The unifying perspective on classifier guidance is also an advantage (but see below).\n- Code is provided, a nice addition to the paper.\n- I checked the GFlowNet-related math and believe it to be sound.", "weaknesses": "- Line 234, 255, 634, 643, maybe others: typo \"GFLowNet\"\n- It would be good to explain why / state as a subclaim that (8) is a policy (i.e., sums to 1 over $s'$), which is not actually obvious from the definition. \n  - It relies on the fact that $p(y|s) = \\sum_{s'}p(y|s')p(s'|s)$, which follows from conditional independence of $y$ (a function of the final state) and $s$ given $s'$. That is a consequence of the Markov property in GFlowNets. A note should be made about this.\n  - The equality may not actually hold in practice, when $p(y|-)$ is a trained classifier, so (8) may not exactly sum to 1. What do you do in this case (in the experiments)?\n- The results on molecule generation raise a few questions:\n  - The reward exponent was set to $\\beta=32$ or $\\beta=96$, which is far larger than in past work, where it was at most 16. Why was such a choice made? This is suspicious, since convergence and mode collapse issues worsen at low temperatures.\n  - Related, with such high exponents, one wonders about mode coverage in the learned distributions. Have you considered the in-sample diversity of the generated molecules (e.g., as measured by average Tanimoto similarity or  diverse top-k metrics)?\n- On related work:\n  - There is no substantial discussion of related work in the main text, even though there is a large body of work on compositional generation and classifier guidance with diffusion models (e.g., the many papers cited in the second paragraph of the introduction).\n  - In the Appendix, the connection with [23] is discussed. The proposed method can easily turn a collection of classifiers for different objectives into a classifier for any convex combination of the objectives. It would be interesting to empirically compare this with conditioning of the model on the linear scalarization weights used in [23].\n  - The paper would be stronger if more explicit unifying connections were made between guidance in diffusion models and in GFlowNets. Note that diffusion models in a fixed time discretization are actually GFlowNets of a certain structure (cf. \"A theory of continuous generative flow networks\" [arXiv:2301.12594, ICML 2023] and \"Unifying generative models with GFlowNets and beyond\" [arXiv:2209.02606]). Classifier guidance using the gradient of $p_t(y|x_t)$ should be the continuous-time limit of equation (8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review of our paper and thoughtful feedback. Please find our response to the raised questions below.\n\n> It would be good to explain why / state as a subclaim that (8) is a policy (i.e., sums to 1 over $s’$).\n\nFollowing your suggestion, in the subsequent revision of the paper we will include a note explaining why (8) is a valid policy (sums to 1 over $s’$). The preliminary text of the note including your suggested proof sketch is below:\n\n```\nNote that (8) is a valid forward policy, i.e. the distribution sums up to 1 over $s'$. This property follows from the relationship $p(y \\vert s) = \\sum_{s'} p(y \\vert s') p_F(s' \\vert s)$ which is implied by the probabilistic model: $y$ is a (stochastic) function of the terminal state $x$, $y$ and $s$ are independent given $s'$.\n```\n\nWe will provide more detailed proof in the appendix.\n\n> The equality may not actually hold in practice. What do you do in this case (in the experiments)?\n\nIt is true that in practice, when only an approximation of the classifier is available, the policy constructed according to eq (8) may not exactly sum to 1. In our experiments, we expressed the conditional policy in terms of log-probabilities (logits) and computed the probabilities as $\\operatorname{softmax}_{s’}(\\log p_F(s’ \\vert s) + \\log p(y \\vert s’) - \\log p (y \\vert s))$, where the softmax operation ensures that the obtained distribution sums up to 1 over $s’$. In theory (assuming the perfectly learned classifier), the softmax operation can be replaced with simple element-wise exponentiation, but using the softmax is also correct.\n\n> The reward exponent was set to $\\beta = 32$ or $\\beta = 64$. Why was such a choice made?\n\nWe chose to set the reward exponent $\\beta$ to $32$ and $96$ due to the following reasons:\n* $\\beta = 96$ was used in “Multi-objective GFlownets” [23] for a similar task (c.f. parameters in Table 12, Section D.4 of [23])\n* composition of models concentrated on high-scoring molecules is a more challenging and application-relevant task\n\n> Have you considered the in-sample diversity of the generated molecules\n\nWe evaluated samples generated from GFlowNets pre-trained with different reward exponents $\\beta$, in order to assess the effect of $\\beta$ on mode coverage and sample diversity. The results are in Tables R.3 and R.4 in the rebuttal PDF. The details of the evaluation and the reported metrics are described in the table captions. As expected, larger reward exponents shift the learned distributions towards high-scoring molecules (the total number of molecules with scores above the threshold increases). For ‘SA’ and ‘QED’ models we don’t observe negative effects of large $\\beta$ on sample diversity and mode coverage: the average pairwise similarity of top 1000 molecules doesn’t grow as $\\beta$ increases and the ratio of Tanimoto-separated modes remains high. For ‘SEH’ models we observe a gradual increase in the average pairwise similarity of the top 1000 molecules and a gradual decrease in the ratio of Tanimoto-separated modes. However, the total number of separated modes grows as $\\beta$ increases, which indicates that larger reward exponents don’t lead to mode dropping. \n\n> There is no substantial discussion of related work in the main text. The paper would be stronger if more explicit unifying connections were made.\n\nWe plan to utilize the additional content page to expand the discussion of the related work on compositional generation and guidance in diffusion models. In particular, the papers cited in the introduction. We also will make a note that would help a reader to better position our work in the view of unifying connections between guidance in diffusion models and GFlowNets (and continuous GFlowNets).\n\n> The proposed method can easily turn a collection of classifiers for different objectives into a classifier for any convex combination of the objectives. It would be interesting to empirically compare with conditioning of the model on the linear scalarization weights used in [23]\n\nWe understand the first part of the question in the following way: “The proposed method can easily turn a collection of GFlowNets for different objectives into a GFlowNet for any convex combination of the objectives”. Please let us know if our understanding is correct.\n\nConvex combination of the rewards $R(x | w) = \\sum_i w_i R_i(x) = \\sum_i w_i Z_i  p_i(x)$ corresponds to a mixture distribution $p(x | w) =\\sum_i \\left(  \\frac{w_i Z_i}{\\sum_j w_j Z_j}  p_i(x) \\right)$ which can be realized by both Multi-objective GFlowNets [23] and our approach. \n\nWe mainly focus on harmonic mean, contrast, and other compositions beyond mixtures. Note that in our scenario (access to forward policies of pre-trained GFlowNets), sampling from the mixture can be realized without training a classifier: it is sufficient to sample an index of the base model from a categorical distribution, and then run the forward process of the selected base GFlowNet. In our approach, we represent the mixture as an individual GFlowNet forward policy (expressed through a classifier) because at the next step, we apply classifier guidance to this policy."}, {"review_id": "4ZTk83XeK0", "reviewer": "Reviewer_m7vZ", "summary": "This paper proposes an approach of Compositional Sculpting for iterative generative models, including GFlowNets and defusion models.\nThe model uses classifier guidance to sample from the target posterior distribution composed of pre-trained base models.\nThe paper also proposes a training algorithm for the classifier.\nThe approach is validated by empirical analyses of an image dataset and molecular generation.", "questions": "Please refer to the weakness section.\n\nTypos - Line 217: train train", "limitations": "The paper has a limitation section.\nIt seems not to have a dedicated section for social impact.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is general enough for different GFlowNets and defusion models.\n\n- The empirical results validate the method.", "weaknesses": "(1) It would be convincing to have more complicated experiments, especially for image data.\nColored MNIST might be too small and simple.\n\n(2) More clarity might be helpful for the following points.\n\n(2A) What is the model in line 193? \"Under the model we have introduced the variables y_1, ... y_n are dependent given a state s in S, but, are independent given a terminal state x in X.\"\n\n(2B) I am confused while reading line 209. The sampling scheme is 1) sample y from its prior 2) sample tau given y. But the following sentence says sampling y given tau.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper and your feedback. We’ve provided some clarification in response to your questions below.\n\n> What is the model on line 193?\n\nEquations (2)-(4) constitute a graphical model over the variables $x$ and $y_1, \\dots, y_n$. We introduce a specialization of this model for GFlowNets in the first paragraph of section 3.1. This is the model we refer to on line 193 in section 3.2. We will clarify in the text that this refers specifically to the specialized model for GFlowNets.\n\n> Confused when reading line 209. The sampling scheme is 1) sample y from its prior 2) sample tau given y. But the following sentence says sampling y given tau.\n\nWe allow conditioning on multiple observations. Thus, there are multiple i.i.d. variables $y_1, \\dots y_n$, one for each observation. Line 209 explains that when generating samples to train on, the first observation $\\widehat y_1$ is sampled from its prior, $\\widehat \\tau$ is sampled given $\\widehat y_1$, and the remaining observations $\\widehat y_2, \\dots \\widehat y_n$ are then sampled given $\\widehat{\\tau}$.\n\n> It seems not to have a dedicated section for social impact.\n\nMost societal impact issues are common to other work on generative models. However, our work has a positive impact in terms of opening the door towards efficient compositions of existing models, enabling wider applicability and computational savings. We will expand on this in a dedicated paragraph in the conclusion."}]

Author rebuttal:
We thank all reviewers for the time and effort dedicated to review of our work and for the helpful and constructive feedback.

## Motivation and focus of the paper

Our work is motivated by the growing costs of general-purpose pre-training of generative models as well as the need for model reuse and control of the generation post-training. The problem of model composition is still new and the iterative nature of the generation processes in diffusion models and GFlowNets necessitates special methods. 

We have developed a formal approach where composition is defined as meaningful mathematical operations on a set of base probability distributions. These compositions are highly controllable, allowing us to emphasize or de-emphasize regions in the composition where specific base distributions have high density. We assume that we have access to a number of GFlowNets or diffusion models that generate samples from these base distributions, and provide a method to construct processes that generate samples from the composite distributions. Compared to training GFlowNets or diffusion models from scratch to reproduce these compositions, which is generally impractical or impossible, our method is both practical and more flexible, as after training it can generate samples from all supported compositions. Further, we derived generalized variants of the theoretical results on diffusion mixture and guidance for the case of GFlowNets. Following theoretical justification, we empirically validate the approach in a range of experimental settings including a practically-relevant molecule generation task.

We believe that our work will motivate and support future research on principled approaches to generative model composition with potential for scale.

## Additional experimental results

We have collected additional empirical results to support our response to reviewers’ comments. The new figures and tables are in the rebuttal PDF document (attached to this comment). We list the new results below, and discuss them in detail within the individual responses to reviewers.

### Classifier training time. Analysis of classifier and learned distributions.

Following the suggestions made by reviewers UsMN and eEwT, we provide classifier training curves (Figures R.1, R.2, R.3) and a summary of the training time (Tables R.1, R.2).  Figure R.1 also shows the distance between the learned compositions (obtained via our method) and ground-truth compositions in the 2D GFlowNet domain.

### Effect of reward exponent $\beta$ on the GFlowNet models: mode coverage and diversity

Following reviewer Eean9’s suggestions, we have evaluated the sample diversity of the base GFlowNets in the molecule generation domain at different reward exponents $\beta. The results are shown in Tables R.3 and R.4.

## Clarity improvement and subsequent revision
Based on the feedback from the reviewers, we will expand the paper and incorporate a number of clarifications. We list the most important changes below:

- We will add the additional experimental results (presented in the rebuttal PDF) and the corresponding discussion.
- Following reviewer UsMn’s suggestion, we will elaborate on the details of the derivation of the non-terminal state classification loss (12), as well as computational complexity and numerical stability.
- Following reviewer ZgyF’s suggestion, we will extend the discussion of the application of the method for controllable text generation and image generation via text-diffusion models.
- Following reviewer ZgyF’s suggestion, we will discuss the scope of the paper and clarify the notion of compositionality in the context of our work as well as the relation to other forms of compositionality.
- Following reviewer VKR5’s suggestion, we will extend the discussion of the application of the method for the composition of models obtained by conditioning on different text prompts in the context of text-conditioned diffusion models.
- Following the suggestions of reviewers eEwT and Ean9, we will expand the discussion of the related work on composition and guidance in generative models as well as unifying connections between GFlowNets and diffusion models.
- Following reviewer Ean9’s suggestion, we will add a subclaim explaining why the classifier-guided policy (8) is valid (sums up to 1 over $s’$) and explain the details of the practical implementation of the policy.
- Following reviewer m7vZ’s suggestion, we will add “Broader impact” section.

Meta-review: This paper proposes the composition of pretrained generative component models to
 achieve compositional generalization, and it's applicable to iterative generative processes such as GFlowNets and diffusion models.
The main contribution is the introduction of two new composition operators: harmonic mean and contrast.
After rebuttal, the authors addressed most of the concerns on missing details and extra experiments.
Six reviewers reviewed this paper and all recommended acceptance.
Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.
Decision: Accept (poster)