Title: Compositional Sculpting of Iterative Generative Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: generative model composition

GFlowNets

diffusion models

classifier guidance

probabilistic methods
Abstract: High training costs of generative models and the need to fine-tune them for specific tasks have created a strong interest in model reuse and composition.
A key challenge in composing iterative generative processes, such as GFlowNets and diffusion models, is that to realize the desired target distribution, all steps of the generative process need to be coordinated, and satisfy delicate balance conditions.
In this work, we propose Compositional Sculpting: a general approach for defining compositions of iterative generative processes. We then introduce a method for sampling from these compositions built on classifier guidance.
We showcase ways to accomplish compositional sculpting in both GFlowNets and diffusion models. We highlight two binary operations $\\unicode{x2014}$ the $\\textit{harmonic mean}\\unicode{x00A0}(p_1 \\otimes p_2$) and the $\\textit{contrast}\\unicode{x00A0}(p_1 \\,\\unicode{x25D1}\\,\\, p_2$) between pairs, and the generalization of these operations to multiple component distributions.
We offer empirical results on image and molecular generation tasks. Project codebase: https://github.com/timgaripov/compositional-sculpting.

Compositional Sculpting of Iterative Generative Processes

Abstract

High training costs of generative models and the need to fine-tune them for specific tasks have created a strong interest in model reuse and composition. A key challenge in composing iterative generative processes, such as GFlowNets and diffusion models, is that to realize the desired target distribution, all steps of the generative process need to be coordinated, and satisfy delicate balance conditions. In this work, we propose Compositional Sculpting: a general approach for defining compositions of iterative generative processes. We then introduce a method for sampling from these compositions built on classifier guidance. We showcase ways to accomplish compositional sculpting in both GFlowNets and diffusion models. We highlight two binary operations - the harmonic mean ( 𝑝 1 ⊗𝑝 2 ) and the contrast ( 𝑝 1 ◑ 𝑝 2 ) between pairs, and the generalization of these operations to multiple component distributions. We offer empirical results on image and molecular generation tasks. Project codebase: https://github.com/timgaripov/compositional-sculpting .

1 Introduction

Large-scale general-purpose pre-training of machine learning models has produced impressive results in computer vision [1-3], image generation [4-6], natural language processing [7-11], robotics [1214], and basic sciences [15]. By distilling vast amounts of data, such models can produce powerful inferences that lead to emergent capabilities beyond the specified training objective [16]. However, generic pre-trained models are often insufficient for specialized tasks in engineering and basic sciences. Field-adaptation via techniques such as explicit fine-tuning on bespoke datasets [17], human feedback [18], or cleverly designed prompts [19, 20] is therefore often required. Alternatively, capabilities of pre-trained models can be utilized and extended via model composition.

Prior work on generative model composition [21, 22, 24] has developed operations for piecing together Energy-Based Models (EBMs) via algebraic manipulations of their energy functions. For example, consider two distributions 𝑝 1 ( 𝑥 ) ∝ exp(𝐸 1 ( 𝑥 )) and 𝑝 2 ( 𝑥 ) ∝ exp(𝐸 2 ( 𝑥 )) induced by energy

Compositional generation [21-28] views a complex target distribution in terms of simpler pre-trained building blocks which can be mixed and matched into a tailored solution to a specialized task. Given a set of base models capturing different properties of the data, composition provides a way to fuse these models into a single composite model with capacity greater than any individual base model. In this way it allows one to specify distributions over examples that exhibit multiple desired properties simultaneously [22]. The need to construct complex distributions adhering to multiple constraints arises in numerous practical multi-objective design problems such as molecule generation [29-31]. In this context, compositional modeling provides mechanisms for control of the resulting distribution and exploration of different trade-offs between the objectives and constraints.

(a)

𝑝

1

(b)

𝑝

2

(c)

𝑝

1

⊗𝑝

2

(d)

𝑝

1

◑

𝑝

2

(e)

𝑝

1

◐

𝑝

2

High

Low

O

0

0

O

functions 𝐸 1 and 𝐸 2 . Their product 𝑝 prod ( 𝑥 ) ∝ 𝑝 1 ( 𝑥 ) 𝑝 2 ( 𝑥 ) ∝ exp(-( 𝐸 1 ( 𝑥 ) + 𝐸 2 ( 𝑥 ))) and negation 𝑝 neg ( 𝑥 ) ∝ 𝑝 1 ( 𝑥 )∕( 𝑝 2 ( 𝑥 )) 𝛾 ∝ exp(-( 𝐸 1 ( 𝑥 ) -𝛾𝐸 2 ( 𝑥 ))) , 𝛾 > 0 correspond to operations on the underlying energy functions. The product assigns high likelihood to points 𝑥 that have high likelihood under both base distributions but assigns low likelihood to points that have close-to-zero likelihood under one (or both). The negation distribution assigns high likelihood to points that are likely under 𝑝 1 but unlikely under 𝑝 2 and assigns low likelihood to points that are likely under 𝑝 2 but unlikely under 𝑝 1 .

Jain et al. [31] develop Multi-Objective GFlowNets (MOGFNs), an extension of GFlowNets for multi-objective optimization tasks. While a vanilla GFlowNet captures a distribution induced by a single reward (objective) function 𝑝 ( 𝑥 ) ∝ 𝑅 ( 𝑥 ) (see Section 2 for details), an MOGFN aims to learn a single conditional model that can realize distributions corresponding to various combinations (e.g. a convex combination) of multiple reward functions. Though a single MOGFN realizes a spectrum of compositions of base reward functions, the approach assumes access to the base rewards at training time. Moreover, MOFGNs require the set of possible composition operations to be specified at training time. In this work, we address post hoc composition of pre-trained GFlowNets (or diffusion models) and provide a way to create compositions that need not be specified in advance.

Iterative generative processes including diffusion models [5, 32-34] and GFlowNets [35, 36] progressively refine coarse objects into cleaner ones over multiple steps. Realizing effective compositions of these models is complicated by the fact that simple alterations in their generation processes result in non-trivial changes in the distributions of the final objects. Unlike for EBMs, products and negations of diffusion models cannot be realized through simple algebraic operations on their score functions. Du et al. [28] show that the result of the addition of score functions is not equal to the score of the diffused product distribution and develop a method that corrects the sum-of-scores sampling via additional MCMC steps nested under each step of the diffusion time loop.

In this work, we introduce Compositional Sculpting, a general approach for the composition of pretrained models. We highlight two special examples of binary operations harmonic mean : ( 𝑝 1 ⊗𝑝 2 ) and contrast : ( 𝑝 1 ◑ 𝑝 2 ). More general compositions are obtained as conditional distributions in a probabilistic model constructed on top of pre-trained base models. We show that these operations can be realized via classifier guidance. We provide results of empirical verification of our method on molecular generation (with GFlowNets) and image generation (with diffusion models).

2 Background

Generative flow networks (GFlowNets). GFlowNets [35, 36] are an approach for generating structured objects (e.g. graphs) from a discrete space  . Given a 'reward function' 𝑅 ( 𝑥 ) ≥ 0 , a GFlowNet seeks to sample from 𝑝 ( 𝑥 ) = 𝑅 ( 𝑥 ) ∕ 𝑍 , where 𝑍 = ∑ 𝑥 𝑅 ( 𝑥 ) , i.e. the model assigns larger probabilities to high-reward objects.

Starting at a fixed initial state 𝑠 0 , objects 𝑥 are generated through a sequence of changes corresponding to a trajectory of incomplete states 𝜏 = ( 𝑠 0 → 𝑠 1 → … → 𝑠 𝑛 -1 → 𝑥 ) . The structure of possible trajectories corresponds to by a DAG (  ,  ) where  is a set of states (both complete and incomplete) and  is the set of directed edges (actions) 𝑠 → 𝑠 ′ . The set of complete objects (terminal states)  is a subset of  . The generation process starts at 𝑠 0 and follows a parameterized stochastic 'forward policy' 𝑃𝐹 ( 𝑠 ′ | 𝑠 ; 𝜃 ) which for each state 𝑠 ∈  ⧵  specifies a probability distribution over all possible successor states 𝑠 ′ ∶( 𝑠 → 𝑠 ′ ) ∈  . The process terminates once a terminal state is reached.

Diffusion models. Diffusion models [5, 32-34, 37] are a family of generative models developed for continuous domains. Given an empirical (data) distribution ̂ 𝑝 ( 𝑥 ) = 1 ∕ 𝑛 ∑ 𝑖 𝛿 ̂ 𝑥𝑖 ( 𝑥 ) in  = ℝ 𝑑 , diffusion models seek to approximate ̂ 𝑝 ( 𝑥 ) via a generative process 𝑝 ( 𝑥 ) .

| Song et al. [34] invoke a result from the theory of stochastic processes [38] which gives the expression for the reverse-time process or 'backward SDE': 𝑑𝑥𝑡 = [ 𝑓𝑡 ( 𝑥𝑡 ) -𝑔 2 𝑡 ∇ 𝑥 log 𝑝𝑡 ( 𝑥𝑡 ) ] 𝑑𝑡 + 𝑔𝑡 𝑑𝑤𝑡, where 𝑤𝑡 is the standard Wiener process in reversed time. This SDE involves the known coefficients 𝑓𝑡 , 𝑔𝑡 and the unknown score function ∇ 𝑥 log 𝑝𝑡 ( ⋅ ) of the marginal distribution 𝑝𝑡 ( ⋅ ) at time 𝑡 . A 'score network' 𝑠 𝑡 ( 𝑥 ; 𝜃 ) (a deep neural network with parameters 𝜃 ) is trained to approximate ∇ 𝑥 log 𝑝𝑡 ( 𝑥 ) . Once 𝑠 𝑡 ( ⋅ ; 𝜃 ) is trained, sampling reduces to numerical integration of the backward SDE.

A diffusion process is a noising process that gradually destroys the original 'clean' data 𝑥 . Viewed as a stochastic differential equation (SDE) [34], it is a time-indexed collection of random variables { 𝑥𝑡 } 𝑇 𝑡 =0 in  = ℝ 𝑑 which interpolates between the data distribution 𝑝 0 ( 𝑥 ) = ̂ 𝑝 ( 𝑥 ) at 𝑡 = 0 and a prior distribution 𝑝𝑇 ( 𝑥 ) at 𝑡 = 𝑇 . The evolution of 𝑥𝑡 is described by the 'forward SDE' 𝑑𝑥𝑡 = 𝑓𝑡 ( 𝑥𝑡 ) 𝑑𝑡 + 𝑔𝑡 𝑑𝑤𝑡 with drift coefficient 𝑓𝑡 ∶ ℝ 𝑑 → ℝ 𝑑 and diffusion coefficient 𝑔𝑡 ∈ ℝ . Here, 𝑤𝑡 is the standard Wiener process. Crucially, the coefficients 𝑓𝑡 , 𝑔𝑡 are generally chosen such that the prior 𝑝𝑇 and the transition probabilities 𝑝𝑠𝑡 ( 𝑥𝑡 𝑥𝑠 ) , 0 ≤ 𝑠 < 𝑡 ≤ 𝑇 have a closed form (see [34]).

Classifier guidance in diffusion models. Classifier guidance [32, 39] is a technique for controllable generation in diffusion models. Suppose that each example 𝑥 0 is accompanied by a discrete class label 𝑦 . The goal is to sample from the conditional distribution 𝑝 0 ( 𝑥 0 | 𝑦 ) . The Bayes rule 𝑝𝑡 ( 𝑥𝑡 | 𝑦 ) ∝ 𝑝𝑡 ( 𝑥𝑡 ) 𝑝𝑡 ( 𝑦 | 𝑥𝑡 ) implies the score-function decomposition ∇ 𝑥𝑡 log 𝑝𝑡 ( 𝑥𝑡 | 𝑦 ) = ∇ 𝑥𝑡 log 𝑝𝑡 ( 𝑥𝑡 ) + ∇ 𝑥𝑡 log 𝑝𝑡 ( 𝑦 | 𝑥𝑡 ) , where the first term is already approximated by a pre-trained unconditional diffusion model and the second term can be derived from a time-dependent classifier 𝑝𝑡 ( 𝑦 | 𝑥𝑡 ) . Therefore, the stated goal can be achieved by first training the classifier 𝑝𝑡 ( 𝑦 | 𝑥𝑡 ) using noisy samples 𝑥𝑡 from the intermediate steps of the process, and then plugging in the expression for the conditional score into the backward SDE sampling process [34].

3 Related Work

Generative model composition. In Section 1 we reviewed prior work on energy-based composition operations and Multi-Objective GFlowNets. Learning mixtures of Generative Adversarial Networks has been addressed in [40], where the mixture components are learned simultaneously, and in [41], where the components are learned one by one in an adaptive boosting fashion. Algorithms for additive and multiplicative boosting of generative models have been developed in [42].

This work focuses on the composition of pre-trained models. Assuming that each pre-trained model represents the distribution of examples demonstrating certain concepts (e.g. molecular properties), the composition of models is equivalent to concept composition (e.g. property 'A' AND property 'B'). The inverse problem is known as 'unsupervised concept discovery', where the goal is to automatically discover composable concepts from data. Unsupervised concept discovery and concept composition methods have been proposed for energy-based models [25] and text-to-image diffusion models [43].

Controllable generation. Generative model composition is a form of post-training control of the generation process - an established area of research. A simple approach to control is training a conditional generative model 𝑝 ( 𝑥 | 𝑐 ) on pairs ( 𝑥, 𝑐 ) of objects 𝑥 and conditioning information 𝑐 . Annotations 𝑐 can be class labels [39], text prompts [4, 6, 44], semantic maps, and images [4]. Different from out work, this assumes that generation control operations are specified at training time. Dhariwal and Nichol [39] apply classifier guidance [32] on top of (conditional) diffusion models to improve their fidelity. Ho and Salimans [45] develop classifier-free guidance by combining conditional and unconditional score functions. In ControlNet [17], an additional network is trained to enable a pre-trained diffusion model to incorporate previously unavailable conditioning information. Meng et al. [46] and Couairon et al. [47] develop semantic image editing methods which first partially noise and then denoise an image to generate an edited version, possibly conditioned on a segmentation mask [47]. Similar to conditional diffusion models, conditional GFlowNets have been used to condition generation on reward exponents [36] or combinations of multiple predefined reward functions [31]. Note that the methods developed in this work can be combined with conditional diffusion models and GFlowNets: 𝑝 ( 𝑥 | 𝑐 1 ) , … , 𝑝 ( 𝑥 | 𝑐𝑚 ) can act as base generative models to be composed.

Compositional generalization. The notion of compositionality has a broad spectrum of interpretations across a variety of disciplines including linguistics, cognitive science, and philosophy. Hupkes et al. [48] collect a list of aspects of compositionality from linguistic and philosophical theories and design practical tests for neural language models. Conwell and Ullman [49] empirically examine the relational understanding of DALL-E 2 [50], a text-guided image generation model, and point out limitations in the model's ability to capture relations such as 'in', 'on', 'hanging over', etc. In this work, we focus on a narrow but well-defined type of composition where we seek to algebraically compose probability densities in a controllable fashion, such that we can emphasize or de-emphasize regions in the data space where specific base distributions have high density.

Connections between GFlowNets and diffusion models. Our method is applicable to compositions of both GFlowNets and diffusion models. This is due to deep connections between these two model families. GFlowNets were initially developed for generating discrete (structured) data [36] and diffusion models were initially developed for continuous data [5, 32]. Lahlou et al. [51] develop an extension of GFlowNets for DAGs with continuous state-action spaces. Zhang et al. [52] point out unifying connections between GFlowNets and other generative model families, including diffusion models. In this work, we articulate another aspect of the relation between GFlowNets and diffusion models: in Section 5.2 we derive the expressions for mixture GFlowNet policies and classifier-guided GFlowNet policies analogous to those derived for diffusion models in [32, 39, 53, 54].

4 Compositional Sculpting of Generative Models

Suppose we can access a number of pre-trained generative models { 𝑝𝑖 ( 𝑥 )} 𝑚 𝑖 =1 over a common domain  . We may wish to compose these distributions such that we can, say, draw samples that are likely to arise from 𝑝 1 ( 𝑥 ) and 𝑝 2 ( 𝑥 ) , or that are likely to arise from 𝑝 1 ( 𝑥 ) but not from 𝑝 2 ( 𝑥 ) . In other words, we wish to specify a distribution that we can shape to emphasize and de-emphasize specific base models.

4.1 Binary Composition Operations

Let us first focus on composing two base models. We could specify the composition as a weighted sum 𝑝 ̃ ( 𝑥 ) = ∑ 2 𝑖 =1 𝜔𝑖𝑝𝑖 ( 𝑥 ) with weights 𝜔 1 , 𝜔 2 ≥ 0 summing to one. The weights determine the prevalence of each base model in the composition, but beyond that our control is limited. We cannot emphasize regions where 𝑝 1 and 𝑝 2 both have high density, or de-emphasize regions where 𝑝 2 has high density.

An alternative is to use conditioning to shape a prior 𝑝 ̃ ( 𝑥 ) based on the base models. When we condition 𝑥 on some observation 𝑦 1 , the resulting posterior takes the form 𝑝 ̃ ( 𝑥 | 𝑦 1 ) ∝ 𝑝 ̃ ( 𝑦 1 | 𝑥 ) 𝑝 ̃ ( 𝑥 ) . Points 𝑥 that match 𝑦 1 according to 𝑝 ̃ ( 𝑦 1 | 𝑥 ) will have increased density, and the density of points that do not match it decreases. Intuitively, by defining 𝑦 1 ∈ {1 , 2} as the event that 𝑥 was generated by a specific base model, we can shape a prior 𝑝 ̃ ( 𝑥 ) based on the densities of the base models. To this end we define a uniform prior over 𝑦𝑘 and define the conditional density 𝑝 ( 𝑥 | 𝑦 1 = 𝑖 ) to represent the fact that 𝑥 was generated from 𝑝𝑖 ( 𝑥 ) . This gives us the following model:

Our goal is to realize compositions which generate samples likely to arise from both 𝑝 1 ( 𝑥 ) and 𝑝 2 ( 𝑥 ) or from 𝑝 1 ( 𝑥 ) but not 𝑝 2 ( 𝑥 ) . Thus we introduce a second observation 𝑦 2 ∈ {1 , 2} such that 𝑦 1 and 𝑦 2 are independent and identically distributed given 𝑥 . The resulting model and inferred posterior are:

| | implied by this model tells us how likely it is that 𝑥 was generated by 𝑝 1 ( 𝑥 ) rather than 𝑝 2 ( 𝑥 ) . In fact, it corresponds to the output of an optimal classifier trained to tell 𝑝 1 ( 𝑥 ) and 𝑝 2 ( 𝑥 ) apart.

| | | The above posterior shows clearly how conditioning on observations 𝑦 1 = 𝑖, 𝑦 2 = 𝑗 has shaped the prior mixture to accentuate regions in the posterior where the observed base models 𝑖, 𝑗 have high density.

Conditioning on observations 𝑦 1 =1 and 𝑦 2 =2 , or equivalently 𝑦 1 =2 , 𝑦 2 =1 , results in the posterior

Conditioning on observations 𝑦 1 =1 and 𝑦 2 =1 results in the posterior

| We refer to this posterior as the ' harmonic mean of 𝑝 1 and 𝑝 2 ', and denote it as a binary operation 𝑝 1 ⊗𝑝 2 . Its value is high only at points that have high likelihood under both 𝑝 1 ( 𝑥 ) and 𝑝 2 ( 𝑥 ) at the same time (Figure 1(c)). Thus, the harmonic mean is an alternative to the product operation for EBMs. The harmonic mean is commutative ( 𝑝 1 ⊗𝑝 2 = 𝑝 2 ⊗𝑝 1 ) and is undefined when 𝑝 1 and 𝑝 2 have disjoint supports, since then the RHS of (5) is zero everywhere.

| We refer to this binary operation, providing an alternative to the negation operation in EBMs, as the ' contrast of 𝑝 1 and 𝑝 2 ', and denote it as 𝑝 1 ◑ 𝑝 2 . The ratio (6) is high when 𝑝 1 ( 𝑥 ) is high and 𝑝 2 ( 𝑥 ) is low (Figure 1(d)). The contrast is not commutative ( 𝑝 1 ◑ 𝑝 2 ≠ 𝑝 2 ◑ 𝑝 1 , unless 𝑝 1 = 𝑝 2 ). We denote the reverse contrast as 𝑝 1 ◐ 𝑝 2 = 𝑝 2 ◑ 𝑝 1 . Appendix C provides a detailed comparison between the contrast and negation operations, and between the harmonic mean and product operations.

Controlling the individual contributions of 𝑝 1 and 𝑝 2 to the composition. In order to provide more control over the extent of individual contributions of 𝑝 1 and 𝑝 2 to the composition, we modify model (3). Specifically, we introduce an interpolation parameter 𝛼 and change the likelihood of observation 𝑦 2 in (3): 𝑝 ̃ ( 𝑦 2 = 𝑖 | 𝑥 ; 𝛼 ) = ( 𝛼𝑝 1 ( 𝑥 )) [ 𝑖 =1] ⋅ ((1𝛼 ) 𝑝 2 ( 𝑥 )) [ 𝑖 =2] ∕( 𝛼𝑝 1 ( 𝑥 ) + (1 𝛼 ) 𝑝 2 ( 𝑥 )) , where 𝛼 ∈ (0 , 1) and [ ⋅ ] denotes the indicator function. Conditional distributions in this model give harmonic interpolation 2 and parameterized contrast :

Operation chaining. As the operations we have introduced result in proper distributions, we can create new 𝑁 -ary operations by chaining binary (and 𝑁 -ary) operations together. For instance, chaining binary harmonic means gives the harmonic mean of three distributions

4.2 Compositional Sculpting: General Approach

The above approach for realizing compositions of two base models can be generalized to compositions of 𝑚 base models 𝑝 1 ( 𝑥 ) , … , 𝑝𝑚 ( 𝑥 ) controlled by 𝑛 observations. Though operator chaining can also realize compositions of 𝑚 base models, our generalized method allows us to specify compositions more flexibly and results in different compositions. We introduce an augmented probabilistic model 𝑝 ̃ ( 𝑥, 𝑦 1 , … , 𝑦 𝑛 ) as a joint distribution over the original objects 𝑥 ∈  and 𝑛 observation variables 𝑦 1 ∈  , … , 𝑦 𝑛 ∈  where  = {1 , … , 𝑚 } . By defining appropriate conditionals 𝑝 ( 𝑦𝑘 | 𝑥 ) we can controllably shape a prior 𝑝 ̃ ( 𝑥 ) into a posterior 𝑝 ̃ ( 𝑥 𝑦 1 , … , 𝑦 𝑛 ) .

| As in the binary case, we propose to use a uniformly-weighted mixture of the base models 𝑝 ̃ ( 𝑥 ) = ( 1 ∕ 𝑚 ) ⋅ ∑ 𝑚 𝑖 =1 𝑝𝑖 ( 𝑥 ) . The support of this mixture is the union of the supports of the base models: ⋃ 𝑚 𝑖 =1 supp{ 𝑝𝑖 ( 𝑥 )} = supp{ 𝑝 ̃ ( 𝑥 )} . This is essential as the prior can only be shaped in places where it has non-zero density. As before we define the conditionals 𝑝 ( 𝑦𝑘 = 𝑖 | 𝑥 ) to correspond to the observation that 𝑥 was generated by base model 𝑖 . The resulting full model is

Note that under this model the mixture can be represented as 𝑝 ̃ ( 𝑥 ) = ∑ 𝑚 𝑦𝑘 =1 𝑝 ̃ ( 𝑥 | 𝑦𝑘 ) 𝑝 ̃ ( 𝑦𝑘 ) for any 𝑘 . The inferred posterior over 𝑥 for this model is

The posterior 𝑝 ̃ ( 𝑥 | 𝑦 1 = 𝑖 1 , … , 𝑦 𝑛 = 𝑖 𝑛 ) is a composition of distributions { 𝑝𝑖 ( 𝑥 )} 𝑚 𝑖 =1 that can be adjusted by choosing values for 𝑦 1 , … , 𝑦 𝑛 . By adding or omitting an observation 𝑦𝑘 = 𝑖 we can sculpt the posterior to our liking, emphasizing or de-emphasizing regions of  where 𝑝𝑖 has high density. The observations can be introduced with multiplicities (e.g., 𝑦 1 = 1 , 𝑦 2 = 1 , 𝑦 3 = 2 ) to further strengthen the effect. Moreover, one can choose to introduce all observations simultaneously as in (10) or sequentially as in (11). As we show below (Section 5.1 for GFlowNets; Appendix A.2 for diffusion models), the composition (10) can be realized by a sampling policy that can be expressed as a function of the pre-trained (base) sampling policies.

Special instances and general formulation. The general approach outlined in this section is not limited to choices we made to construct the model in equation (9), i.e. 𝑝 ̃ ( 𝑥 ) does not have to be a uniformly weighted mixture of the base distributions, 𝑦 1 , … , 𝑦 𝑛 do not have to be independent and identically distributed given 𝑥 , and different choices of the likelihood 𝑝 ̃ ( 𝑦 = 𝑖 | 𝑥 ) are possible. For instance, in the model for parameterized operations (7) the likelihoods of observations 𝑝 ̃ ( 𝑦 1 | 𝑥 ) , 𝑝 ̃ ( 𝑦 2 | 𝑥 ) differ. 5 Compositional Sculpting of Iterative Generative Processes

In this Section, we show how to apply the model above to compose GFlowNets, and how one can use classifier guidance to sample from the composition. The similar method for diffusion model composition is described in Appendix A.2.

5.1 Composition of GFlowNets

Besides a sample 𝑥 from 𝑝𝑖 ( 𝑥 ) , a GFlowNet also generates a trajectory 𝜏 which ends in the state 𝑥 . Thus, we extend the model 𝑝 ̃ ( 𝑥, 𝑦 1 , … , 𝑦 𝑛 ) , described above, and introduce 𝜏 as a variable with conditional distribution 𝑝 ̃ ( 𝜏 | 𝑦𝑘 = 𝑖 ) = ∏ | 𝜏 | -1 𝑡 =0 𝑝𝑖,𝐹 ( 𝑠 𝑡 +1 | 𝑠 𝑡 ) , where 𝑝𝑖,𝐹 is the forward policy of the GFlowNet that samples from 𝑝𝑖 .

Proposition 5.1 (GFlowNet mixture policy) . Suppose distributions 𝑝 1 ( 𝑥 ) , … , 𝑝𝑚 ( 𝑥 ) are realized by GFlowNets with forward policies 𝑝 1 ,𝐹 ( ⋅ | ⋅ ) , … , 𝑝 𝑚,𝐹 ( ⋅ | ⋅ ) . Then, the mixture distribution 𝑝 M ( 𝑥 ) = ∑ 𝑚 𝑖 =1 𝜔𝑖𝑝𝑖 ( 𝑥 ) with 𝜔 1 , … , 𝜔𝑚 ≥ 0 and ∑ 𝑚 𝑖 =1 𝜔𝑖 = 1 is realized by the GFlowNet forward policy

Our approach for sampling from the composition is conceptually simple. Given 𝑚 base GFlowNets that sample from 𝑝 1 , … , 𝑝𝑚 respectively, we start by defining the prior 𝑝 ̃ ( 𝑥 ) as the uniform mixture of these GFlowNets. Proposition 5.1 shows that this mixture can be realized by a policy constructed from the forward policies of the base GFlowNets. We then apply classifier guidance to this mixture to sample from the composition. Proposition 5.2 shows that classifier guidance results in a new policy which can be constructed directly from the GFlowNet being guided.

Proposition 5.2 (GFlowNet classifier guidance) . Consider a joint distribution 𝑝 ( 𝑥, 𝑦 ) over a discrete space  ×  such that the marginal 𝑝 ( 𝑥 ) is realized by a GFlowNet with forward policy 𝑝𝐹 ( ⋅ | ⋅ ) . Assume that the joint distribution of 𝑥 , 𝑦 , and GFlowNet trajectories 𝜏 = ( 𝑠 0… 𝑠 𝑛 = 𝑥 ) decomposes as 𝑝 ( 𝜏, 𝑥, 𝑦 ) = 𝑝 ( 𝜏, 𝑥 ) 𝑝 ( 𝑦 | 𝑥 ) , i.e. 𝑦 is independent of the intermediate states { 𝑠 𝑖 } 𝑛 -1 𝑖 =0 in 𝜏 given 𝑥 . Then, 1. For all non-terminal nodes 𝑠 ∈  ⧵  , the probabilities 𝑝 ( 𝑦 𝑠 ) satisfy where 𝑦 is a random variable such that the joint distribution of a GFlowNet trajectory 𝜏 and 𝑦 is given by 𝑝 ( 𝜏, 𝑦 = 𝑖 ) = 𝜔𝑖𝑝𝑖 ( 𝜏 ) for 𝑖 ∈ {1 , … , 𝑚 } .

The conditional distribution 𝑝 ( 𝑥 𝑦 ) is realized by the classifier-guided policy

| | Note that (13) ensures that ∑ 𝑠 ′ ∶( 𝑠 → 𝑠 ′ )∈  𝑝𝐹 ( 𝑠 ′ | 𝑠, 𝑦 ) = 1 .

Proposition 5.1 is analogous to results on mixtures of diffusion models (Theorem 1 of Peluchetti [53], Theorem 1 of Lipman et al. [54]). Proposition 5.2 is analogous to classifier guidance for diffusion models [32, 39]. To the best of our knowledge, our work is the first to derive both results for GFlowNets.

Theorem 5.3 summarizes our approach. The propositions and the theorem are proved in Appendix D. Theorem 5.3. Suppose distributions 𝑝 1 ( 𝑥 ) , … , 𝑝𝑚 ( 𝑥 ) are realized by GFlowNets with forward policies 𝑝 1 ,𝐹 ( ⋅ | ⋅ ) , … , 𝑝𝑚,𝐹 ( ⋅ | ⋅ ) respectively. Let 𝑦 1 , … , 𝑦 𝑛 be random variables defined by (9) . Then, the conditional 𝑝 ̃ ( 𝑥 | 𝑦 1 , … , 𝑦 𝑛 ) is realized by the forward policy

| Note that the result of conditioning on observations 𝑦 1 , … , 𝑦 𝑛 is just another GFlowNet policy. Therefore, to condition on more observations, one can apply classifier guidance repeatedly.

5.2 Classifier Training (GFlowNets)

The evaluation of policy (15) requires knowledge of the probabilities 𝑝 ̃ ( 𝑦 1 , … , 𝑦 𝑛 | 𝑠 ) , 𝑝 ̃ ( 𝑦 | 𝑠 ) . These probabilities can be estimated by a classifier fitted to trajectories sampled from the base GFlowNets.

Learning the terminal state classifier. For a terminal state 𝑥 , the variables 𝑦 1 , … , 𝑦 𝑛 are independent, hence we can use the factorization ̃ 𝑄𝜙 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑥 ) = ∏ 𝑛 𝑘 =1 ̃ 𝑄𝜙 ( 𝑦𝑘 | 𝑥 ) . Moreover, all distributions on the r.h.s. must be the same, i.e. it is enough to learn just ̃ 𝑄𝜙 ( 𝑦 1 | 𝑥 ) . This marginal classifier can be learned by minimizing the cross-entropy loss

Let ̃ 𝑄𝜙 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑠 ) be a classifier with parameters 𝜙 that we wish to train to approximate the groundtruth conditional 𝑝 ̃ ( 𝑦 1 , … , 𝑦 𝑛 | 𝑠 ) . Under model (9), 𝑦 1 , … , 𝑦 𝑛 are dependent given a state 𝑠 ∈  ⧵  , but, are independent given a terminal state 𝑥 ∈  . This motivates separate treatment of terminal and non-terminal states.

| Here 𝑦 ̂ 1 is sampled from 𝑝 ̃ ( 𝑦 1 ) , which is uniform under our choice of 𝑝 ̃ ( 𝑥 ) . Then, 𝑥 ̂ | ( 𝑦 1 = 𝑦 ̂ 1 ) is generated from the base GFlowNet 𝑝̂ 𝑦 1 , since (9) implies that 𝑝 ̃ ( 𝑥 𝑦 = 𝑦 ̂ 1 ) = 𝑝̂ 𝑦 1 ( 𝑥 ) .

| Learning the non-terminal state classifier. Given a non-terminal state 𝑠 ∈  ⧵  , we need to model 𝑦 1 , … , 𝑦 𝑛 jointly, and training requires sampling tuples ( ̂ 𝑠, ̂ 𝑦 1 , … , ̂ 𝑦𝑛 ) . Non-terminal states 𝑠 can be generated as intermediate states in trajectories 𝜏 = ( 𝑠 0 → 𝑠 1 → … → 𝑥 ) . Given a sampled trajectory 𝜏 ̂ and a set of labels 𝑦 ̂ 1 , … , ̂ 𝑦𝑛 we denote the trajectory cross-entropy by

Pairs ( ̂ 𝜏, ̂ 𝑦 1 ) can be generated in the same way as in the terminal classifier training above: 1) 𝑦 ̂ 1 ∼ 𝑝 ̃ ( 𝑦 1 ) ; 2) 𝜏 ̂ ∼ 𝑝̂ 𝑦 1 ( 𝜏 ) . Sampling 𝑦 ̂ 2 , … , ̂ 𝑦𝑛 given 𝑥 ̂ (the terminal state of 𝜏 ̂ ) requires access to values 𝑝 ̃ ( 𝑦𝑘 = ̂ 𝑦𝑘 | 𝑥 ̂ ) , which are not directly available. However, if the terminal classifier is learned as described above, the estimates 𝑤𝑖 ( 𝑥 ̂ ; 𝜙 ) = ̃ 𝑄𝜙 ( 𝑦 1 = 𝑖 | 𝑥 = 𝑥 ̂ ) can be used instead. In this case, the loss and the sampling procedure for the non-terminal classifier rely on the outputs of the terminal classifier. In order to train two classifiers simultaneously, and avoid the instability due to a feedback loop, we employ the 'target network' technique developed in the context of deep Q-learning [55]. We introduce a 'target network' parameter vector 𝜙 which is used to produce the estimates 𝑤𝑖 ( 𝑥 ̂ ; 𝜙 ) for the non-terminal loss. We update 𝜙 as the exponential moving average of the recent iterates of 𝜙 .

After putting all components together the training loss for the non-terminal state classifier is

⎣ ⎦ We refer the reader to Appendix D.4 for a more detailed derivation of the loss (18). Algorithm A.1 shows the complete classifier training procedure.

𝑝

1

𝑝

2

𝑝

1

⊗𝑝

2

𝑝

1

◑

𝑝

2

𝑝

1

◑

0

.

95

𝑝

2

High

Low

2

𝑝

3

𝑝 ̃

𝑥

𝑦

1 =1

𝑦

2 =2

𝑝 ̃

(

𝑥

𝑦

1 =1

𝑦

2 =2

𝑦

3 =3

)

𝑝 ̃

𝑥

𝑦

1 =2

𝑦

2 =2

𝑝 ̃

(

𝑥

𝑦

1 =2

𝑦

2 =2

𝑦

3 =2

)

(a) Base at

= 32

(b) Harmonic mean

(c) Contrasts

(d) Base at

= 96

(e) Harmonic mean

PSA

PSEH

PSEH  PSA

PSA O09s PSEH

PSEHDo9s PSA

PSA

PSEH

PSEH  PSA

0.9

0.9

0.9

0.9

0.9

0.8

0.8

0.8

0.8

0.8

S

0.7

S

0.7

S

0.7

S

0.7

A

S

0.7

0.6

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.5

0.4

'0.0

0.2

0.40.6

0.8

1.0

0.4

0.0

0.2

0.40.6

0.8

1.0

0.4

0.0

0.2

0.40.6

0.8

1.0

0.4

0.0

0.2

0.40.6

0.8

1.0

0.4

0.0

0.2

0.40.6

0.8

1.0

SEH

SEH

SEH

SEH

SEH

𝛽

6 Experiments

2D distributions via GFlowNet. We tested our GFlowNet composition method on 2D grid [36], a controlled domain, where the ground-truth composite distributions can be evaluated directly.

Figure 2 (top row) shows the distributions obtained by composing two pre-trained GFlowNets (top row; left). The harmonic mean 𝑝 1 ⊗𝑝 2 , covers the regions that have high probability under both 𝑝 1 and 𝑝 2 and excludes locations where either of the probabilities is low. 𝑝 1 ◑ 𝑝 2 resembles 𝑝 1 but the relative masses of the modes of 𝑝 1 are modulated by 𝑝 2 : regions with high 𝑝 2 have lower probability under contrast. The parameterized contrast 𝑝 1 ◑ 0 . 95 𝑝 2 with 𝛼 = 0 . 05 magnifies the contrasting effect: high 𝑝 2 ( 𝑥 ) implies very low ( 𝑝 1 ◑ 0 . 95 𝑝 2 )( 𝑥 ) . The bottom row of Figure 2 shows operations on 3 distributions. The conditional 𝑝 ̃ ( 𝑥 | 𝑦 1=1 , 𝑦 2 =2) is concentrated on the points that have high likelihood under both 𝑝 1 and 𝑝 2 . Similarly, the value 𝑝 ̃ ( 𝑥 | 𝑦 1=1 , 𝑦 2=2 , 𝑦 3 =3) is high if 𝑥 is likely to be observed under all three distributions at the same time. The conditionals 𝑝 ̃ ( 𝑥 | 𝑦 1=2 , 𝑦 2 =2) and 𝑝 ̃ ( 𝑥 | 𝑦 1=2 , 𝑦 2=2 , 𝑦 3 =2) highlight the points with high 𝑝 2 ( 𝑥 ) but low 𝑝 1 ( 𝑥 ) and 𝑝 3 ( 𝑥 ) . Conditioning on three labels results in a sharper distribution compared to double-conditioning. We provide quantitative results and further details in Appendix F.1. The classifier learning curves are provided in Appendix G.4.

In the 2D grid domain, the states are the cells of an 𝐻 × 𝐻 grid. The starting state is the upper-left cell 𝑠 0 = (0 , 0) . At each state, the allowed actions are: 1) move right; 2) move down; 3) terminate the trajectory at the current position. We first trained GFlowNets 𝑝𝑖 ( 𝑥 ) ∝ 𝑅𝑖 ( 𝑥 ) with reward functions 𝑅𝑖 ( 𝑥 ) > 0 , and then trained classifiers and constructed compositions following Theorem 5.3.

Molecule generation via GFlowNet. Next, we evaluated our method for GFlowNet composition on a large and highly structured data space, and assessed the effect that composition operations have

𝛽

In each row, the numbers show the percentage of the samples from the respective model that fall into one of 8 bins according to rewards. The 'low' and 'high' categories are decided by thresholding SEH: 0.5, SA: 0.6, QED: 0.25.

-0

.

5

0

.

0

0

.

5

1

.

0

-0

.

5

0

.

0

0

.

5

1

.

0

/u1D45D

SEH

/u1D45D

SA

/u1D45D

QED

(a)

(b)

(c)

(d)

(e)

(f)

(g)

on resulting data distributions in a practical setting. To that end, we conducted experiments with GFlowNets trained for the molecular generation task proposed by Bengio et al. [36]. We train base GFlowNets with 3 reward functions 𝑅 SEH ( 𝑥 ) , 𝑅 SA ( 𝑥 ) , and 𝑅 QED ( 𝑥 ) measuring 3 distinct molecular properties (details in Appendix F.2). Following Bengio et al. [36], we introduced the parameter 𝛽 which controls the sharpness (temperature) of the target distribution: 𝑝 ( 𝑥 ) ∝ 𝑅 ( 𝑥 ) 𝛽 , increasing 𝛽 results in a distribution skewed towards high-reward objects. We experimented with 𝛽 = 32 and 𝛽 = 96 (Figures 3(a),3(d)). After training base GFlowNets with the respective reward functions, we trained classifiers with Algorithm A.1. In order to evaluate the base models and the composed GFlowNet policies, we generated 5 000 samples with each policy and anaylzed the samples. Further details are in Appendix F.2. Classifier learning curves are provided in Appendix G.4. Sample diversity statistics of base GFlowNets at different values of 𝛽 are provided in Appendix G.5.

Weshow reward distribution statistics of three GFlowNets (trained with SEH, SA, and QED at 𝛽 = 32 ) and their compositions in Table 1. Each row of the table gives a percentage breakdown of the samples from a given model into one of 2 3 = 8 bins according to rewards. For all three base models, the majority of the samples fall into the 'high' category according to the respective reward, while the rewards that were not used for training show variation. Conditioning on two different labels (e.g. 𝑦 ={ SEH , QED } ) results in concentration on examples that score high in two selected rewards, but not necessarily scoring high in the reward that was not selected. The conditional 𝑦 ={ SEH , QED , SA } shifts the focus to examples that have all three properties.

Figure 3 shows reward distributions of base GFlowNets (trained with rewards 𝑅 SEH ( 𝑥 ) 𝛽 , 𝑅 SA ( 𝑥 ) 𝛽 , 𝛽 ∈ {32 , 96} ) and their compositions. Base GFlowNet distributions are concentrated on examples that score high in their respective rewards. For each model, there is considerable variation in the reward that was not used for training. The harmonic mean operation (Figures 3(b), 3(e)) results in distributions that are concentrated on the samples scoring high in both rewards. The contrast operation (Figure 3(c)) has the opposite effect: the distributions are skewed towards the examples scoring high in only one of the original rewards. Note that the tails of the contrast distributions are retreating from the area covered by the harmonic mean.

Figure 4 shows 2D embeddings of the distributions appearing in Table 1. The embeddings were computed with t-SNE based on the pairwise earth mover's distances (details in Appendix F.2, complete summary of distribution distances in Table G.5). The configuration of the embeddings provides insight into the relative positions of the base models and conditionals in the distribution space. The points corresponding to the pairwise conditionals lie in between the two base models selected for conditioning. The conditional 𝑦 ={ SEH , SA , QED } appears to be near the centroid of the triangle ( 𝑝 SEH , 𝑝 SA , 𝑝 QED ) and lies close the the pairwise conditionals. The distributions obtained by repeated conditioning on the same label (e.g. 𝑦 ={ SEH , SEH , SEH } ) are spread out to the boundary, lying closer to the respective base distributions and relatively far from pairwise conditionals.

Colored MNIST generation via diffusion models. Finally, we empirically tested our method for the composition of diffusion models on an image generation task. In this experiment, we composed

𝑝

1

𝑝

2

𝑝

3

𝑝 ̃

(

𝑥

|

|

|

𝑦

1 =1

𝑦

2 =1

)

𝑝 ̃

(

𝑥

|

|

|

𝑦

1 =2

𝑦

2 =2

)

𝑝 ̃

(

𝑥

|

|

|

𝑦

1 =3

𝑦

2 =3

)

𝑝 ̃

(

𝑥

𝑦

1 =1

𝑦

2 =2

𝑦

3 =3

)

𝑝 ̃

(

𝑥

𝑦

1 =1

𝑦

2 =2

𝑦

3 =1

)

𝑝 ̃

(

𝑥

𝑦

1 =1

𝑦

2 =2

𝑦

3 =2

)

081031

2

a23200

33333333

2

2222222

3

2

23220

0

20223

2

3333337

3333

3333333

2111

2

222222

32

333

2

20

272

3

33

3

333

122

22222

3

3

2

0

3

2

2

2

3

3

333

33

3

3

2

2

22

222

2

2

32233

33

3

2

022

0

Q

333333

3

33333

3

3

e

2222222

2

222223

3

13032

20232

Q

3

6333333

3

2

20222

22222222

2

223232

32

2222

2

7

222

2

2

222

2

20222

C

22222223

2

2

2222232

three diffusion models trained to generate MNIST [56] digits {0, 1, 2, 3} in two colors: cyan and beige. Each model was trained to generate digits with a specific property: 𝑝 1 generated cyan digits, 𝑝 2 generated digits less than 2, and 𝑝 3 generated even digits.

Figure 5 shows samples from the pre-trained models and from selected compositions. The negating effect of not conditioning on observations is clearly visible in the compositions using two variables. For example, 𝑝 ̃ ( 𝑥 | 𝑦 1 =1 , 𝑦 2 =1) only generates cyan 3's. Because there we do not condition on 𝑝 2 or 𝑝 3 , the composition excludes digits that have high probability under 𝑝 2 or 𝑝 3 , i.e. those that are less than 2 or even. In 𝑝 ̃ ( 𝑥 | 𝑦 1 =1 , 𝑦 2 =3) , cyan even digits have high density under both 𝑝 1 and 𝑝 3 , but because 𝑝 2 is not conditioned on, the composition excludes digits less than two (i.e. cyan 0's). Finally, 𝑝 ̃ ( 𝑥 | 𝑦 1 =1 , 𝑦 2 =2 , 𝑦 3 =3) generates only cyan 0's, on which all base models have high density. 7 Conclusion

We built the composition iteratively via the factorization 𝑝 ̃ ( 𝑥 | 𝑦 1 ,𝑦 2 , 𝑦 3 ) ∝ 𝑝 ̃ ( 𝑥 ) 𝑝 ̃ ( 𝑦 1 ,𝑦 2 | 𝑥 ) 𝑝 ̃ ( 𝑦 3 | 𝑥,𝑦 1 ,𝑦 2 ) . To this end, we first trained a classifier ̃ 𝑄 ( 𝑦 1 ,𝑦 2 | 𝑥𝑡 ) on trajectories sampled from the base models. This allowed us to generate samples from 𝑝 ̃ ( 𝑥 | 𝑦 1 ,𝑦 2 ) . Wethen trained an additional classifier ̃ 𝑄 ( 𝑦 3 | 𝑥𝑡 ,𝑦 1 ,𝑦 2 ) on trajectories from compositions defined by ( 𝑦 1 ,𝑦 2 ) to allow us to sample from 𝑝 ̃ ( 𝑥 | 𝑦 1 ,𝑦 2 ,𝑦 3 ) . Further details can be found in Appendix F.3.

Weintroduced Compositional Sculpting, a general approach for composing iterative generative models. Compositions are defined through 'observations', which enable us to emphasize or de-emphasize the density of the composition in regions where specific base models have high density. We highlighted two binary compositions, harmonic mean and contrast, which are analogous to the product and negation operations defined on EBMs. A crucial feature of the compositions we have introduced is that we can sample from them directly. By extending classifier guidance we are able to leverage the generative capabilities of the base models to produce samples from the composition. Through empirical experiments, we validated our approach for composing diffusion models and GFlowNets on toy domains, molecular generation, and image generation.

Broader impact. We proposed a mathematical framework and methods for the composition of pre-trained generative models. While the primary emphasis of our work is on advancing foundational research on generative modeling methodology and principled sampling techniques, our work inherits ethical concerns associated with generative models such as creation of deepfake content and misinformation dissemination, as well as reproduction of biases present in the datasets used for model training. If not carefully managed, these models can perpetuate societal biases, exacerbating issues of fairness and equity. Our work further contributes to research on the reuse of pre-trained models. This research direction promotes eco-friendly AI development, with the long-term goal of reducing energy consumption and carbon emissions associated with large-scale generative model training.

References

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ArXiv , abs/2304.02643, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2818-2829, 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2018.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

OpenAI. Chatgpt (mar 14, 2023 version). Large language model, 2023. URL https://chat. openai.com/chat .

Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alexandru Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier García, Jianmo Ni, Andrew Chen, Kathleen Kenealy, J. Clark, Stephan Lee, Daniel H Garrette, James Lee-Thorp, Colin Raffel, Noam M. Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy B. Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio . ArXiv , abs/2203.17189, 2022.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Anand Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Ho Vuong, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. ArXiv , abs/2212.06817, 2022.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv , abs/2204.02311, 2022.

Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-e: An embodied multimodal language model. ArXiv , abs/2303.03378, 2023.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as i say: Grounding language in robotic affordances. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning , volume 205 of Proceedings of Machine Learning Research , pages 287-318. PMLR, 2023.

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan,

Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv , abs/2108.07258, 2021.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ArXiv , abs/2302.05543, 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems , volume 35, pages 27730-27744, 2022.

D'idac Sur'is, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. ArXiv , abs/2303.08128, 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837, 2022.

Geoffrey E Hinton. Products of experts. In Ninth International Conference on Artificial Neural Networks , volume 1, 1999.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=HkCsm6lRb .

Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems , 33:6637-6647, 2020.

Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems , 34: 15608-15620, 2021.

Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose visual relations. Advances in Neural Information Processing Systems , 34:23166-23178, 2021.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 423-439. Springer, 2022.

Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International Conference on Machine Learning , pages 8489-8510. PMLR, 2023.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference on machine learning , pages 4849-4859. PMLR, 2020.

Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS: Markov Molecular Sampling for Multi-objective Drug Discovery. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=kHSu4ebxFXY .

Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective GFlownets. In International Conference on Machine Learning , pages 14631-14653. PMLR, 2023.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems , 33:12438-12448, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=PxTIG12RRHS .

Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research , 24(210):1-55, 2023. URL http://jmlr.org/papers/v24/22-0364.html .

Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems , 34:27381-27394, 2021.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Mgan: Training generative adversarial nets with multiple generators. In International conference on learning representations , 2018.

Ilya O Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, and Bernhard Schölkopf. Adagan: Boosting generative models. Advances in neural information processing systems , 30, 2017.

Aditya Grover and Stefano Ermon. Boosted generative models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Nan Liu, Yilun Du, Shuang Li, Joshua B Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-image generative models. ArXiv , abs/2306.05357, 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021. URL https://openreview. net/forum?id=qw8AKxfYbI .

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=aBsCjcPu_tE .

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= 3lge0p5o-M-.

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research , 67:757-795, 2020.

Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. ArXiv , abs/2208.00005, 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv , abs/2204.06125, 2022.

Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández-Garcıa, Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning , pages 18269-18300. PMLR, 2023.

Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. ArXiv , abs/2209.02606, 2022.

Stefano Peluchetti. Non-denoising forward-time diffusions, 2022. URL https://openreview. net/forum?id=oVfIKuhqfC .

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t .

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Yann LeCun. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/ , 1998.

Damiano Brigo. The general mixture-diffusion sde and its relationship with an uncertainvolatility option model with volatility-asset decorrelation. ArXiv , abs/0812.4052, 2008.

Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlownets. In Advances in Neural Information Processing Systems , volume 35, pages 5955-5967, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ArXiv , abs/1412.6980, 2014.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics , 1:1-11, 2009.

Greg Landrum. Rdkit: Open-source cheminformatics, 2010. URL https://www.rdkit.org/ .

G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry , 4(2):90-98, 2012.

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in neural information processing systems , 32, 2019.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Matthew D Zeiler. Adadelta: an adaptive learning rate method. ArXiv , abs/1212.5701, 2012.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

A Compositional Sculpting of Iterative Generative Processes

(Section 5, continued)

A.1 Classifier Training (GFlowNets): Algorithm

A.2 Composition of Diffusion Models

In this section, we show how the method introduced above can be applied to diffusion models. First, we adapt the model we introduced in (9)-(11) to diffusion models. A diffusion model trained to sample from 𝑝𝑖 ( 𝑥 ) generates a trajectory 𝜏 = { 𝑥𝑡 } 𝑇 𝑡 =0 over a range of time steps which starts with a randomly sampled state 𝑥𝑇 and ends in 𝑥 0 , where 𝑥 0 has distribution 𝑝𝑖,𝑡 =0 ( 𝑥 ) = 𝑝𝑖 ( 𝑥 ) . Thus, we must adapt our model to reflect this. We introduce a set of mutually dependent variables 𝑥𝑡 for 𝑡 ∈ (0 , 𝑇 ] with as conditional distribution the transition kernel of the diffusion model 𝑝𝑖 ( 𝑥𝑡 𝑥 0 ) .

Proposition A.1 (Diffusion mixture SDE) . Suppose distributions 𝑝 1 ( 𝑥 ) , … , 𝑝𝑚 ( 𝑥 ) are realized by diffusion models with forward SDEs 𝑑𝑥𝑖,𝑡 = 𝑓𝑖,𝑡 ( 𝑥𝑖,𝑡 ) 𝑑𝑡 + 𝑔𝑖,𝑡 𝑑𝑤𝑖,𝑡 and score functions 𝑠 𝑖,𝑡 ( ⋅ ) , respectively. Then, the mixture distribution 𝑝 M ( 𝑥 ) = ∑ 𝑚 𝑖 =1 𝜔𝑖𝑝𝑖 ( 𝑥 ) with 𝜔 1… 𝜔𝑚 ≥ 0 and ∑ 𝑚 𝑖 =1 𝜔𝑖 = 1 is realized by a diffusion model with forward SDE

| Given 𝑚 base diffusion models that sample from 𝑝 1 , … , 𝑝𝑚 respectively, we define the prior 𝑝 ̃ ( 𝑥 ) as a mixture of these diffusion models. Proposition A.1 shows that this mixture is a diffusion model that can be constructed directly from the base diffusion models. We then apply classifier guidance to this mixture to sample from the composition. We present an informal version of the proposition below. The required assumptions and the proof are provided in Appendix D.5.

and backward SDE

with

If the base diffusion models have a common forward SDE 𝑑𝑥𝑖,𝑡 = 𝑓𝑡 ( 𝑥𝑖,𝑡 ) 𝑑𝑡 + 𝑔𝑡 𝑑𝑤𝑖,𝑡 , equations (19)-(20) simplify to

Theorem A.2 summarizes the overall approach.

Theorem A.2. Suppose distributions 𝑝 1 ( 𝑥 ) , … , 𝑝𝑚 ( 𝑥 ) are realized by diffusion models with forward SDEs 𝑑𝑥𝑖,𝑡 = 𝑓𝑖,𝑡 ( 𝑥𝑖,𝑡 ) 𝑑𝑡 + 𝑔𝑖,𝑡 𝑑𝑤𝑖,𝑡 and score functions 𝑠 𝑖,𝑡 ( ⋅ ) , respectively. Let 𝑦 1 , … 𝑦𝑛 be random variables defined by (9) . Then, the conditional 𝑝 ̃ ( 𝑥 | 𝑦 1 , … , 𝑦 𝑛 ) is realized by a classifier-guided diffusion with backward SDE 𝑑𝑥𝑡 = 𝑣𝐶,𝑡 ( 𝑥𝑡 , 𝑦 1 , … , 𝑦 𝑛 ) 𝑑𝑡 + 𝑔𝐶,𝑡 ( 𝑥𝑡 ) 𝑑𝑤𝑡, (23)

with

The proof of Theorem A.2 is provided in Appendix D.6.

A.3 Classifier Training (Diffusion Models)

We approximate the inferential distributions in equations (22) and (23) with a time-conditioned classifier ̃ 𝑄𝜙 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑥𝑡 ) with parameters 𝜙 . Contrary to GFlowNets, which employed a terminal and non-terminal state classifier, here we only need a single time-dependent classifier. The classifier is trained with different objectives on terminal and non-terminal states. The variables 𝑦 1 , … , 𝑦 𝑛 are dependent given a state 𝑥𝑡 for 𝑡 ∈ [0 , 𝑇 ) , but are independent given the terminal state 𝑥𝑇 . Thus, when training on terminal states we can exploit this independence. Furthermore, we generally found it beneficial to initially train only on terminal states. The loss for the non-terminal states depends on classifications of the terminal state of the associated trajectories, thus by minimizing the classification error of terminal states first, we reduce noise in the loss calculated for the non-terminal states later.

For a terminal state 𝑥 0 , the classifier ̃ 𝑄𝜙 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑥𝑡 ) can be factorized as ∏ 𝑛 𝑘 =1 ̃ 𝑄𝜙 ( 𝑦𝑘 | 𝑥 0 ) . Hence we can train ̃ 𝑄 by minimizing the cross-entropy loss

Samples 𝑝 ̃ ( 𝑥 0 , 𝑦 1 ) can be generated according to the factorization 𝑝 ̃ ( 𝑦 1 ) 𝑝 ̃ ( 𝑥 0 | 𝑦 1 ) . First, 𝑦 ̂ 1 is sampled from 𝑝 ̃ ( 𝑦 1 ) , which is uniform under our choice of 𝑝 ̃ ( 𝑥 0 ) . Then, 𝑥 ̂ 0 | ( 𝑦 1 = 𝑦 ̂ 1 ) is generated from the reverse SDE of base diffusion model 𝑝̂ 𝑦 1 ( 𝑥 ) . Note that equation (9) implies that all observations have the same conditional distribution given 𝑥 . Thus, ̃ 𝑄𝜙 ( 𝑦 1 | 𝑥 0 ) is also a classifier for observations 𝑦 2 , … , 𝑦 𝑛 .

For a non-terminal state 𝑥𝑡 with 𝑡 ∈ (0 , 𝑇 ] , we must train ̃ 𝑄 to predict 𝑦 1 , … , 𝑦 𝑛 jointly. For a non-terminal state ̂ 𝑥𝑡 and observations 𝑦 ̂ 1 , … , ̂ 𝑦𝑛 , the cross-entropy loss is

| Tuples ( ̂ 𝑥𝑡 , ̂ 𝑦 1 , … , ̂ 𝑦𝑛 ) are obtained as follows: 1) 𝑦 ̂ 1 ∼ 𝑝 ̃ ( 𝑦 1 ) ; 2) A trajectory 𝜏 = { 𝑥𝑡 } 𝑇 𝑡 =0 is sampled from the reverse SDE of diffusion model 𝑦 1 . At this point, we would ideally sample 𝑦 ̂ 2 , … , ̂ 𝑦𝑛 given 𝑥 ̂ 0 but this requires access to 𝑝 ̃ ( 𝑦𝑘 = ̂ 𝑦𝑘 | 𝑥 ̂ 0 ) . Instead, we approximate this with 𝑤𝑖 ( 𝑥 ̂ ; 𝜙 ) = ̃ 𝑄𝜙 ( 𝑦 1 = 𝑖 | 𝑥 0 = 𝑥 ̂ 0 ) and marginalize over 𝑦 ̂ 2 , … , ̂ 𝑦𝑛 to obtain the cross-entropy loss

B Classifier Guidance for Parameterized Operations

This section covers the details of classifier guidance and classifier training for the parameterized operations (Section 4.1).

The complete probabilistic model for the parameterized operations on two distributions is given by

While in the probabilistic model (3) all observations 𝑦𝑖 are exchangeable, in the parameterized model (29) 𝑦 1 and 𝑦 2 are not symmetric. This difference requires changes in the classifier training algorithm for the parameterized operations.

We develop the method for the parameterized operations based on two observations:

𝑦 1 appears in (29) in the same way as in (3);

the likelihood 𝑝 ̃ ( 𝑦 2 𝑥 ; 𝛼 ) of 𝑦 2 given 𝑥 can be expressed as the function of 𝑝 ̃ ( 𝑦 1 𝑥 ) and 𝛼 :

| | These two observations combined suggest the training procedure where 1) the terminal state classifier is trained to approximate 𝑝 ̃ ( 𝑦 1 = 𝑖 | 𝑥 ) in the same way as in Section 5.2; 2) the probability estimates 𝑤𝑖 ( ̂ 𝑥, ̂ 𝛼 ; 𝜙 ) ≈ 𝑝 ̃ ( 𝑦 2 = 𝑖 ; 𝛼 ̂ ) are expressed through the learned terminal state classifier 𝑝 ̃ ( 𝑦 1 = 𝑖 | 𝑥 ) via (30). Below we provide details of this procedure for the case of GFlowNet composition.

Learning the terminal state classifier. The marginal 𝑦 1 classifier ̃ 𝑄𝜙 ( 𝑦 1 | 𝑥 ) is learned by minimizing the cross-entropy loss

Then, the joint classifier ̃ 𝑄𝜙 ( 𝑦 1 , 𝑦 2 𝑥 ; 𝛼 ) is constructed as

where ̃ 𝑄𝜙 ( 𝑦 2 | 𝑥 ; 𝛼 ) can be expressed through the marginal ̃ 𝑄𝜙 ( 𝑦 1 | 𝑥 ) via (30).

Learning the non-terminal state classifier. The non-terminal state classifier ̃ 𝑄 ( 𝑦 1 , 𝑦 2 | 𝑠 ; 𝛼 ) models 𝑦 1 and 𝑦 2 jointly. Note that 𝛼 is one of the inputs to the classifier model. Given a sampled trajectory 𝜏 ̂ , labels 𝑦 ̂ 1 , ̂ 𝑦 2 , and 𝛼 ̂ , the total cross-entropy loss of all non-terminal states in 𝜏 ̂ is

-4

-3

-2

-1

0

1

2

3

4

/u1D465

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

/u1D45D

(

/u1D465

)

/u1D45D

1

/u1D45D

2

(a) Base distributions

𝑝

1

,

𝑝

2

-4

-3

-2

-1

0

1

2

3

4

/u1D465

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

/u1D45D

(

/u1D465

)

/u1D45D

1

⊗/u1D45D

2

/u1D45D

1

prod

/u1D45D

2

(b) HM

(ours)

, Product

-4

-3

-2

-1

0

1

2

3

4

/u1D465

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

/u1D45D

(

/u1D465

)

/u1D6FC

= 0

.

001

/u1D6FC

= 0

.

050

/u1D6FC

= 0

.

500

(c) Contrasts

(ours)

𝑝

1

◑

( 1-

)

𝑝

2

-4

-3

-2

-1

0

1

2

3

4

/u1D465

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

/u1D45D

(

/u1D465

)

/u1D6FE

= 0

.

50

/u1D6FE

= 0

.

25

/u1D6FE

= 0

.

10

(d) Negations

𝑝

1

𝑝

2

𝛼

𝛾

The pairs ( ̂ 𝜏, ̂ 𝑦 1 ) can be generated via a sampling scheme similar to the one used for the terminal state classifier loss above: 1) 𝑦 ̂ 1 ∼ 𝑝 ̃ ( 𝑦 1 ) and 2) 𝜏 ̂ ∼ 𝑝̂ 𝑦 1 ( 𝜏 ) . An approximation of the distribution of 𝑦 ̂ 2 given 𝜏 ̂ is constructed using (30):

| | Since these expressions involve outputs of the terminal state classifier which is being trained simultaneously, we again (see Section 5.2) introduce the target network parameters 𝜙 that are used to compute the probability estimates (34).

The training loss for the non-terminal state classifier is

⎣ ⎦ where 𝑝 ( 𝛼 ) is sampling distribution over 𝛼 ∈ (0 , 1) . In our experiments, we used the following sampling scheme for 𝛼 :

C Analysis of Compositional Sculpting and Energy Operations

The harmonic mean and contrast operations we have introduced are analogous to the product and negation operations for EBMs respectively. Although the harmonic mean and product operations are quite similar in practice, unlike the negation operation our proposed contrast operation always results in a valid probability distribution. Figure C.1 shows the results of these operations applied to two Gaussian distributions. The harmonic mean and product, shown in panel (b), are both concentrated on points that have high probability under both Gaussians. Figure C.1(c) shows parameterized contrasts 𝑝 1 ◑ (1𝛼 ) 𝑝 2 at different values of 𝛼 , and panel (d) shows negations 𝑝 1 neg 𝛾 𝑝 2 at different values of 𝛾 . The effect of negation at 𝛾 = 0 . 1 resembles the effect of the contrast operation: the density retreats from the high likelihood region of 𝑝 2 . However, as 𝛾 increases to 0 . 5 the distribution starts to concentrate excessively on the values 𝑥 < -3 . This is due to the instability of division 𝑝 1 ( 𝑥 ) ∕( 𝑝 2 ( 𝑥 )) 𝛾 in regions where 𝑝 2 ( 𝑥 ) → 0 . Proposition C.1 shows that negation 𝑝 1 neg 𝛾 𝑝 2 in many cases results in an improper (non-normalizable) distribution.

Mathematical analysis of operations. Harmonic mean and product are not defined for pairs of distributions 𝑝 1 , 𝑝 2 which have disjoint supports. In such cases, attempts at evaluation of the expressions for 𝑝 1 ⊗𝑝 2 and 𝑝 1 prod 𝑝 2 will lead to impossible probability distributions that have zero

probability mass (density) everywhere 3 . The result of both harmonic mean and product are correctly defined for any pair of distributions 𝑝 1 , 𝑝 2 that have non-empty support intersection.

Notably, contrast is well-defined for any input distributions while negation is ill-defined for some input distributions 𝑝 1 , 𝑝 2 as formally stated below (see Figure C.1 (d) for a concrete example).

Proposition C.1.

For any 𝛼 ∈ (0 , 1) the parameterized contrast operation 𝑝 1 ◑ (1𝛼 ) 𝑝 2 (7) is well-defined: gives a proper distribution for any pair of distributions 𝑝 1 , 𝑝 2 .

For any 𝛾 ∈ (0 , 1) there are infinitely many pairs of distributions 𝑝 1 , 𝑝 2 such that the negation

results in an improper (non-normalizable) distribution.

Proof. Without loss of generality, we prove the claims of the proposition assuming absolutely continuous distributions 𝑝 1 , 𝑝 2 with probability density functions 𝑝 1 ( ⋅ ) , 𝑝 2 ( ⋅ ) .

Claim 1. For any two distributions 𝑝 1 , 𝑝 2 we have 𝑝 1 ( 𝑥 ) ≥ 0 , 𝑝 2 ( 𝑥 ) ≥ 0 , ∫  𝑝 1 ( 𝑥 ) 𝑑𝑥 = ∫  𝑝 2 ( 𝑥 ) 𝑑𝑥 = 1 < ∞ . Then, the RHS of the expression for the parameterized contrast operation 𝑝 1 ◑ (1𝛼 ) 𝑝 2 (7) satisfies

For points 𝑥 ∉ supp( 𝑝 1 ) ∪ supp( 𝑝 2 ) , we set 𝑝 1 ( 𝑥 ) 2 𝛼𝑝 1 ( 𝑥 )+(1𝛼 ) 𝑝 2 ( 𝑥 ) = 0 since by construction the composite distributions do not have probability mass outside of the union of the supports of the original distributions. The above implies that

Therefore, the RHS of the expression for the parameterized contrast operation 𝑝 1 ◑ (1𝛼 ) 𝑝 2 (7) can be normalized, and the distribution 𝑝 1 ◑ (1𝛼 ) 𝑝 2 is well-defined.

Claim 2. For any 𝛾 ∈ (0 , 1) we provide an infinite collection of distribution pairs 𝑝 1 and 𝑝 2 such that negation 𝑝 1 neg 𝛾 𝑝 2 results in a non-normalizable distribution.

For the given 𝛾 ∈ (0 , 1) we select four numbers 𝜇 1 ∈ ℝ , 𝜇 2 ∈ ℝ , 𝜎 1 > 0 , 𝜎 2 > 0 such that

Consider univariate normal distributions 𝑝 1 ( 𝑥 ) =  ( 𝑥 ; 𝜇 1 , 𝜎 2 1 ) , 𝑝 2 ( 𝑥 ) =  ( 𝑥 ; 𝜇 2 , 𝜎 2 2 ) with density functions

For such 𝑝 1 and 𝑝 2

Condition (40) implies that the quadratic function under the exponent above has a non-negative coefficient for 𝑥 2 . Therefore this function either grows unbounded as 𝑥 → ∞ (if the coefficients for the quadratic and linear terms are not zero), or constant (if the coefficients for quadratic and linear terms are zero). In either case, ∫ ℝ 𝑝 1 ( 𝑥 ) ∕( 𝑝 2 ( 𝑥 )) 𝛾 𝑑𝑥 = ∞ .

D Proofs and Derivations

D.1 Proof of Proposition 5.1

Our goal is to show that the policy (12) induces the mixture distribution 𝑝𝑀 ( 𝑥 ) = ∑ 𝑚 𝑖 =1 𝜔𝑖𝑝𝑖 ( 𝑥 ) .

Preliminaries. In our proof below we use the notion of 'the probability of observing a state 𝑠 ∈  on a GFlowNet trajectory'. Following Bengio et al. [35], we abuse the notation and denote this probability by where  𝑠 0 ,𝑠 is the set of all (sub)trajectories starting at 𝑠 0 and ending at 𝑠 . The probabilities induced by the policy (12) are denoted by 𝑝𝑀 ( 𝑠 ) . Note that 𝑝𝑖 ( 𝑠 ) and 𝑝𝑀 ( 𝑠 ) should not be interpreted as probability mass functions over the set of states  . In particular 𝑝𝑖 ( 𝑠 0 ) = 𝑝𝑀 ( 𝑠 0 ) = 1 and sums ∑ 𝑠 ∈  𝑝𝑖 ( 𝑠 ) , ∑ 𝑠 ∈  𝑝𝑀 ( 𝑠 ) are not equal to 1 (unless  = { 𝑠 0 } ). However, the functions 𝑝𝑖 ( ⋅ ) , 𝑝𝑀 ( ⋅ ) restricted to the set of terminal states  give valid probability distributions over  : ∑ 𝑥 ∈  𝑝𝑖 ( 𝑥 ) = ∑ 𝑥 ∈  𝑝𝑀 ( 𝑥 ) = 1 . By definition 𝑝𝑖 ( ⋅ ) and 𝑝𝑀 ( ⋅ ) satisfy the recurrent relationship

The joint distribution of 𝑦 and 𝜏 described in the statement of Proposition 5.1 is 𝑝 ( 𝜏, 𝑦 = 𝑖 ) = 𝑤𝑖𝑝𝑖 ( 𝜏 ) . This joint distribution over 𝑦 and trajectories implies the following expressions for the distributions involving intermediate states 𝑠 .

Proof. Using the notation introduced above, we can formally state our goal. We need to show that 𝑝𝑀 ( 𝑥 ) induced by 𝑝𝑀,𝐹 gives the mixture distribution

We prove a more general equation for all states 𝑠 ∈ 

by induction over the DAG (  ,  ) .

Base case. Consider the initial state 𝑠 0 ∈  . By definition 𝑝𝑖 ( 𝑠 0 ) = 𝑝𝑀 ( 𝑠 0 ) = 1 which implies

Inductive step. Consider a state 𝑠 such that (51) holds for all predecessor states 𝑠 ∗ ∶ ( 𝑠 ∗ → 𝑠 ) ∈  . For such a state we have which proves (51) for 𝑠 .

D.2 Proof of Proposition 5.2

Claim 1 . Our goal is to prove the relationship (13) for all non-terminal states 𝑠 ∈  ⧵  . To prove this relationship, we invoke several important properties of Markovian probability flows on DAGs [35].

By Proposition 16 of Bengio et al. [35] for the given GFlowNet forward policy 𝑝𝐹 ( ⋅ | ⋅ ) there exists a unique backward policy 𝑝𝐵 ( ⋅ | ⋅ ) such that the probability of any complete trajectory 𝜏 = ( 𝑠 0 → … → 𝑠 | 𝜏 | = 𝑥 ) in DAG (  ,  ) can be expressed as and the probability of observing a state 𝑠 ∈  on a trajectory can be expressed as

| | By the statement of Proposition 5.2, in the probabilistic model 𝑝 ( 𝑥, 𝑦 ) , the marginal distribution 𝑝 ( 𝑥 ) is realized by the GFlowNet forward policy 𝑝𝐹 ( ⋅ | ⋅ ) and 𝑦 is independent of intermediate states 𝑠 . The joint distribution 𝑝 ( 𝑠, 𝑦 ) is given by where  𝑠,𝑥 is the set of all (sub)trajectories starting at 𝑠 and ending at 𝑥 . Moreover, 𝑝𝐹 ( ⋅ | ⋅ ) and 𝑝𝐵 ( ⋅ | ⋅ ) are related through the 'detailed balance condition' [35, Proposition 21] 𝑝 ( 𝑠 ) 𝑝𝐹 ( 𝑠 ′ 𝑠 ) = 𝑝 ( 𝑠 ′ ) 𝑝𝐵 ( 𝑠 𝑠 ′ ) , ∀( 𝑠 → 𝑠 ′ ) ∈  . (62)

Expressing the conditional probability 𝑝 ( 𝑦 𝑠 ) through the joint 𝑝 ( 𝑠, 𝑦 ) we obtain

which proves (13).

| We know (Section D.1) that the state probabilities induced by the marginal GFlowNet policy 𝑝𝐹 ( ⋅ | ⋅ ) satisfy the recurrence

Claim 2 . Our goal is to show that the classifier-guided policy (14) induces the conditional distribution 𝑝 ( 𝑦 𝑥 ) .

Let 𝑝𝑦 ( ⋅ ) denote the state probabilities induced by the classifier-guided policy (14). These probabilities by definition (Section D.1) satisfy the recurrence

We show that by induction over DAG (  ,  ) .

Base case. Consider the initial state 𝑠 0 . By definition 𝑝𝑦 ( 𝑠 0 ) = 1 . At the same time 𝑝 ( 𝑠 0 | 𝑦 ) = 𝑝 ({ 𝜏 ∶ 𝑠 0 ∈ 𝜏 } 𝑦 ) = 1 . Therefore 𝑝𝑦 ( 𝑠 0 ) = 𝑝 ( 𝑠 0 𝑦 ) .

| | Inductive step. Consider a state 𝑠 such that (73) holds for all predecessor states 𝑠 ∗ ∶ ( 𝑠 ∗ → 𝑠 ) ∈  . For such a state we have

which proves (73) for state 𝑠 .

D.3 Proof of Theorem 5.3

By Proposition 5.1 we have that the policy

In the probabilistic model 𝑝 ̃ ( 𝑥, 𝑦 1… , 𝑦 𝑛 ) the marginal distribution 𝑝 ̃ ( 𝑥 ) = 𝑝𝑀 ( 𝑥 ) is realized by the mixture policy 𝑝𝑀,𝐹 . Therefore, 𝑝 ̃ ( 𝑥, 𝑦 1 , … , 𝑦 𝑛 ) satisfies the conditions of Proposition 5.2 which states that the conditional distribution 𝑝 ̃ ( 𝑥 𝑦 1 , … , 𝑦 𝑛 ) is realized by the classifier-guided policy generates the mixture distribution 𝑝𝑀 ( 𝑥 ) = 1 𝑚 ∑ 𝑚 𝑖 =1 𝑝𝑖 ( 𝑥 ) .

This section provides a more detailed step-by-step derivation of the non-terminal state classifier training objective (18).

Step 1. Our goal is to train a classifier ̃ 𝑄 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑠 ) . This classifier can be obtained as the optimal solution of

where 𝓁 ( ⋅ ) is defined in equation (17). An unbiased estimate of the loss (and its gradient) can be obtained by sampling ( ̂ 𝜏, ̂ 𝑦 1 , … , ̂ 𝑦𝑛 ) and evaluating (17) directly. However sampling tuples ( 𝜏, 𝑦 1 , … , 𝑦 𝑛 ) is not straightforward. The following steps describe our proposed approach to the estimation of expectation in (83).

Step 2. The expectation in (83) can be expressed as

⎣ ⎦ where we re-wrote the expectation over ( 𝑦 2 , … , 𝑦 𝑛 ) | 𝜏 as the explicit sum of the form 𝔼 𝑞 ( 𝑧 ) [ 𝑔 ( 𝑧 )] = ∑ 𝑧 ∈  𝑞 ( 𝑧 ) 𝑔 ( 𝑧 ) . The expectation over ( 𝜏, 𝑦 1 ) can be estimated by sampling pairs ( ̂ 𝜏, ̂ 𝑦 1 ) as described in the paragraph after equation (17): 1) 𝑦 ̂ 1 ∼ 𝑝 ̃ ( 𝑦 1 ) and 2) 𝜏 ̂ ∼ 𝑝̂ 𝑦 1 ( 𝜏 ) . The only missing part is the probabilities 𝑝 ̃ ( 𝑦𝑖 = ̂ 𝑦𝑖 𝑥 = 𝑥 ̂ ) which are not directly available.

| Step 3. Our proposal is to approximate these probabilities as 𝑝 ̃ ( 𝑦 1 = 𝑗 | 𝑥 = 𝑥 ̂ ) ≈ 𝑤𝑗 ( 𝑥 ̂ ; 𝜙 ) = ̃ 𝑄𝜙 ( 𝑦 1 = 𝑗 | 𝑥 = 𝑥 ̂ ) . The idea here is that the terminal state classifier ̃ 𝑄𝜙 ( 𝑦 1 | 𝑥 ) , when trained to optimality, produces outputs exactly equal to the probabilities 𝑝 ̃ ( 𝑦 1 | 𝑥 ) , and the more the classifier is trained the better is the approximation of the probabilities.

Combining all the steps above, we arrive at objective (18) which we use to estimate the expectation in (83).

Step 4. Steps 1-3, give a procedure where the computation of the non-terminal state classification loss requires access to the terminal state classifier. As we described in the paragraph preceding equation (18), we propose to train non-terminal and terminal classifiers simultaneously and introduce 'target network' parameters. The weights 𝑤 are computed by the target network ̃ 𝑄 𝜙 .

Note that equation (18) involves summation over 𝑦 ̂ 2 , … ̂ 𝑦𝑛 with 𝑚 𝑛 -1 terms in the sum. If values of 𝑛 and 𝑚 are small, the sum can be evaluated directly. In general, one could trade off estimation accuracy for improved speed by replacing the summation with Monte Carlo estimation. In this case, the values ̂ 𝑦𝑘 are sampled from the categorical distributions 𝑄 𝜙 ( 𝑦 | 𝑥 ) . Note that labels can be sampled in parallel since 𝑦𝑖 are independent given 𝑥 .

D.5 Assumptions and Proof of Proposition A.1

This subsection provides a formal statement of the assumptions and a more detailed formulation of Proposition A.1.

We found an error in the statement and the proof of Theorem 1 (Appendix A.2 of [53]). The error makes the result of [53] for 𝐷 -dimensional diffusion processes disagree with the result of [57] for 1 -dimensional diffusion processes.

The assumptions, the formulation of the result, and the proof below closely follow those of Theorem 1 of Peluchetti [53]. Theorem 1 in [53] generalizes the result of Brigo [57] (Corollary 1.3), which derives the SDE for mixtures of 1D diffusion processes.

Here we provide a corrected version of Theorem 1 of [53] in a modified notation and a simplified setting (mixture of finite rather than infinite number of diffusion processes). Most of the content is directly adapted from [53].

for a vector-valued 𝑓 ∶ ℝ 𝐷 → ℝ 𝐷 , the divergence of 𝑓 is denoted as ∇ ⋅ ( 𝑓 ( 𝑥 )) = ∑ 𝐷 𝑑 =1 𝜕 𝜕𝑥𝑑 𝑓𝑑 ( 𝑥 ) ,

Notation:

for a scalar-values 𝑎 ∶ ℝ 𝐷 → ℝ , the divergence of the gradient of 𝑎 (the Laplace operator) is denoted by Δ( 𝑎 ( 𝑥 )) = ∇ ⋅ (∇ 𝑎 ( 𝑥 )) = ∑ 𝐷 𝑑 =1 𝜕 2 𝜕𝑥 2 𝑑 𝑎 ( 𝑥 ) .

Assumption 1 (SDE solution). A given 𝐷 -dimensional SDE ( 𝑓, 𝑔 ) :

with associated initial distribution 𝑝 0 ( 𝑥 ) and integration interval [0 , 𝑇 ] admits a unique strong solution on [0 , 𝑇 ] .

Assumption 2 (SDE density). A given 𝐷 -dimensional SDE ( 𝑓, 𝑔 ) with associated initial distribution 𝑝 0 ( 𝑥 ) and integration interval [0 , 𝑇 ] admits a marginal density on (0 , 𝑇 ) with respect to the 𝐷 -dimensional Lebesgue measure that uniquely satisfies the Fokker-Plank (Kolmogorov-forward) partial differential equation (PDE):

Assumption 3 (positivity). For a given stochastic process, all finite-dimensional densities, conditional or not, are strictly positive.

Theorem D.1 (Diffusion mixture representation) . Consider the family of 𝐷 -dimensional SDEs on 𝑡 ∈ [0 , 𝑇 ] indexed by 𝑖 ∈ {1 , … , 𝑚 } , where the initial distributions 𝑝𝑖, 0 and the Wiener processes 𝑤𝑖,𝑡 are all independent. Let 𝑝𝑖,𝑡 , 𝑡 ∈ (0 , 𝑇 ) denote the marginal density of 𝑥𝑖,𝑡 . For mixing weights { 𝜔𝑖 } 𝑚 𝑖 =1 , 𝜔𝑖 ≥ 0 , ∑ 𝑚 𝑖 =1 𝜔𝑖 = 1 , define the mixture marginal density 𝑝𝑀,𝑡 for 𝑡 ∈ (0 , 𝑇 ) and the mixture initial distribution 𝑝𝑀, 0 by

Consider the 𝐷 -dimensional SDE on 𝑡 ∈ [0 , 𝑇 ] defined by

It is assumed that all diffusion processes 𝑥𝑖,𝑡 and the diffusion process 𝑥𝑀,𝑡 satisfy the regularity Assumptions 1, 2, and 3. Then the marginal distribution of the diffusion 𝑥𝑀,𝑡 is 𝑝𝑀,𝑡 .

Proof. For 0 < 𝑡 < 𝑇 we have that

The second is an exchange of the order of summation and differentiation, the third line is the application of the Fokker-Planck PDEs for processes 𝑥𝑖,𝑡 , the fourth line is a rewriting in terms of 𝑝𝑀,𝑡 , the fifth line is another exchange of the order of summation and differentiation. The result follows by noticing that 𝑝𝑀,𝑡 ( 𝑥 ) satisfies the Fokker-Planck equation of SDE ( 𝑓𝑀,𝑔𝑀 ) .

Proof of Proposition A.1. Below, we show that the result of Proposition A.1 follows from Theorem D.1.

First, we rewrite 𝑓𝑀,𝑡 ( 𝑥𝑡 ) and 𝑔𝑀,𝑡 ( 𝑥𝑡 ) in (89) in terms of the classifier probabilities (21):

From the forward process, we derive the backward process following Song et al. [34]. Using the result of Anderson [38], the backward process for (19) is given by

With these expressions, we apply the result of Theorem D.1 to the base forward processes 𝑑𝑥𝑖,𝑡 = 𝑓𝑖,𝑡 ( 𝑥𝑖,𝑡 ) 𝑑𝑡 + 𝑔𝑖,𝑡 𝑑𝑤𝑖,𝑡 and obtain the mixture forward process in equation (19).

Note that the term ∇ 𝑥𝑡 ( 𝑔 2 𝑀,𝑡 ( 𝑥𝑡 )) is due to the fact that the diffusion coefficient 𝑔𝑀,𝑡 ( 𝑥𝑡 ) in (19) is a function of 𝑥 (cf., equation (16), Appendix A in [34]). This term can be transformed as follows

Substituting the last expression in (99), we notice that the term 𝑔 2 𝑀,𝑡 ( 𝑥𝑡 )∇ 𝑥𝑡 log 𝑝𝑀,𝑡 ( 𝑥𝑡 ) cancels out, and, after simple algebraic manipulations, we arrive at (20).

D.6 Proof of Theorem A.2

We proove Theorem A.2 in the assumptions of Section D.5. In Section D.5 we established that the mixture diffusion process has the forward SDE

and the backward SDE

Weapply classifier guidance with classifier 𝑝 ̃ ( 𝑦 1 , … , 𝑦 𝑛 | 𝑥𝑡 ) to the mixture diffusion process, following Song et al. [34] (see equations (48)-(49) in [34]). The backward SDE of the classifier-guided mixture diffusion is

Finally, we arrive at (23) by substituting (105) in the above, canceling out the term 𝑔 2 𝑀,𝑡 ( 𝑥𝑡 )∇ 𝑥𝑡 log 𝑝𝑀,𝑡 ( 𝑥𝑡 ) , and applying simple algebraic manipulations.

E Implementation Details

E.1 Classifier Guidance in GFlowNets

Classifier guidance in GFlowNets (14) is realized through modification of the base forward policy via the multiplication by the ratio of the classifier outputs 𝑝 ( 𝑦 | 𝑠 ′ ) ∕ 𝑝 ( 𝑦 | 𝑠 ) . The ground truth (theoretically optimal) non-terminal state classifier 𝑝 ( 𝑦 | 𝑠 ) by Proposition 5.2 satisfies (13) which ensures that the guided policy (14) is valid, i.e. for any state 𝑠 ∈ 

In practice, the ground truth values of 𝑝 ( 𝑦 | 𝑠 ) are unavailable. Instead, an approximation 𝑄𝜙 ( 𝑦 | 𝑠 ) ≈ 𝑝 ( 𝑦 | 𝑠 ) is learned. Equation (13) might not hold for the learned classifier 𝑄𝜙 , but we still wish to use ̃ 𝑄𝜙 for classifier guidance in practice. In order to ensure that the classifier-guided policy is valid in practice even when the approximation 𝑄𝜙 ( 𝑦 | 𝑠 ) of the classifier 𝑝 ( 𝑦 | 𝑠 ) is used, we implement guidance as described below. First, we express the guided policy (14) in terms of log-probabilities:

Parameterizing distributions through log-probabilities is common practice in probabilistic modeling: GFlowNet forward policies [36, 58] and probabilistic classifiers are typically parameterized by deep neural networks that output logits (unnormalized log-probabilities).

Second, in the log-probability parameterization the guided policy (14) can be equivalently expressed as

| | | In theory, the softmax operation can be replaced with simple exponentiation, i.e. the numerator in (114) is sufficient on its own since Proposition 5.2 ensures that the sum in the denominator equals to 1 . However, using the softmax is beneficial in practice when we substitute learned classifier 𝑄𝜙 ( 𝑦 | 𝑠 ) instead of the ground truth classifier 𝑝 ( 𝑦 | 𝑠 ) . Indeed when 𝑄𝜙 ( 𝑦 | 𝑠 ) does not satisfy (13), the softmax operation ensures that the guided policy

| | | | is valid (i.e. probabilities sum up to 1 over 𝑠 ′ ). The fact the softmax expression is valid in theory ensures that policy (115) guided by 𝑄𝜙 ( 𝑦 | 𝑠 ) approaches the ground truth policy (guided by 𝑝 ( 𝑦 | 𝑠 ) ) as 𝑄𝜙 ( 𝑦 | 𝑠 ) approaches 𝑝 ( 𝑦 | 𝑠 ) throughout training. F Experiment details

F.1 2D Distributions with GFlowNets

The base GFlowNet forward policies 𝑝𝑖,𝐹 ( 𝑠 ′ | 𝑠 ; 𝜃 ) were parameterized as MLPs with 2 hidden layers and 256 units in each hidden layer. The cell coordinates of a state 𝑠 on the 2D 32 × 32 grid were one-hot encoded. The dimensionality of the input was 2 ⋅ 32 . The outputs of the forward policy network were the logits of the softmax distribution over 3 action choices: 1) move down; 2) move right; 3) stop.

The classifier was parameterized as MLP with 2 hidden layers and 256 units in each hidden layer. The inputs to the classifier were one-hot encoded cell coordinates, terminal state flag ( {0 , 1} ), and log( 𝛼 ∕(1𝛼 )) (in the case of parameterized operations). The classifier outputs were the logits of the joint label distribution ̃ 𝑄𝜙 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑠 ) for non-terminal states 𝑠 and the logits of the marginal label distribution ̃ 𝑄𝜙 ( 𝑦 1 𝑥 ) for terminal states 𝑥 .

We trained base GFlowNets with the trajectory balance loss [58]. We fixed the uniform backward policy in the trajectory balance objective. We used Adam optimizer [59] with learning rate 0 . 001 , and pre-train the base models for 20 000 steps with batch size 16 ( 16 trajectories per batch). The log of the total flow log 𝑍𝜃 was optimized with Adam with a learning rate 0 . 1 . In order to promote exploration in trajectory sampling for the trajectory balance objective, we used the sampling policy which takes random actions (uniformly) with probability 0 . 05 but, otherwise, follows the current learned forward policy.

| We trained the classifier with the loss described in Section 5.2. We used Adam with learning rate 0 . 001 . We performed 15 000 training steps with batch size 64 ( 64 trajectories sampled from each of the base models per training step). We updated the target network parameters 𝜙 as the exponential moving average (EMA) of 𝜙 with the smoothing factor 0 . 995 . We linearly increased the weight 𝛾 ( step ) of the non-terminal state loss from 0 to 1 throughout the first 3 000 steps and kept constant 𝛾 = 1 afterward. For the 𝛼 -parameterized version of the classifier (Section B), we used the following sampling scheme for 𝛼 : 𝑧 ̂ ∼ 𝑈 [-3 . 5 , 3 . 5] , 𝛼 ̂ = 1 .

Quantitative evaluation. For each of the composite distributions shown in Figure 2 we evaluated the L1-distance 𝐿 1 ( 𝑝 method , 𝑝 GT ) = ∑ 𝑥 ∈  | 𝑝 method ( 𝑥 ) -𝑝 GT ( 𝑥 ) | between the distribution 𝑝 method induced by the classifier-guided policy and the ground-truth composition distribution 𝑝 GT computed from the known base model probabilities 𝑝𝑖 . The evaluation results are presented below.

Figure 2 top row . 𝑝 1 ⊗𝑝 2 : 𝐿 1 = 0 . 071 ; 𝑝 1 ◑ 𝑝 2 : 𝐿 1 = 0 . 086 ; 𝑝 1 ◑ 0 . 95 𝑝 2 : 𝐿 1 = 0 . 167 .

| | Figure G.6 shows the distance between the composition and the ground truth as a function of the number of training steps for the classifier as well as the terminal and non-terminal classifier learning curves.

Figure 2 bottom row . 𝑝 ̃ ( 𝑥 | 𝑦 1 =1 , 𝑦 2 =2) : 𝐿 1 = 0 . 076 ; 𝑝 ̃ ( 𝑥 | 𝑦 1 =1 , 𝑦 2 =2 , 𝑦 3 =3) : 𝐿 1 = 0 . 087 ; 𝑝 ̃ ( 𝑥 𝑦 1 =2 , 𝑦 2 =2) : 𝐿 1 = 0 . 112 ; 𝑝 ̃ ( 𝑥 𝑦 1 =2 , 𝑦 2 =2 , 𝑦 3 =2) : 𝐿 1 = 0 . 122 .

F.2 Molecule Generation

Domain. In the molecule generation task [36], the objects 𝑥 ∈  are molecular graphs. The nonterminal states 𝑠 ∈  ⧵  are incomplete molecular graphs. The transitions from a given non-terminal state 𝑠 are of two types: 1) fragment addition 𝑠 → 𝑠 ′ : new molecular graph 𝑠 ′ is obtained by attaching a new fragment to the molecular graph 𝑠 ; 2) stop action 𝑠 → 𝑥 : if 𝑠 ≠ 𝑠 0 , then the generation process can be terminated at the molecular graph corresponding to the current state (note that new terminal state 𝑥 ∈  is different from 𝑠 ∈  ⧵  , but both states correspond to the same molecular graph).

Rewards. We trained GFlowNets using 3 reward functions: SEH , a reward computed by an MPNN [60] that was trained by Bengio et al. [36] to estimate the binding energy of a molecule to the soluble epoxide hydrolase protein; SA , an estimate of synthetic accessibility [61] computed with tools from RDKit library [62]; QED , a quantitative estimate of drug-likeness [63] which is also computed with RDKit . We normalized all reward functions to the range [0 , 1] . Higher values of SEH, SA, and QED correspond to stronger binding, higher synthetic accessibility, and higher drug-likeness respectively. Following Bengio et al. [36], we introduced the parameter 𝛽 which controls the sharpness (temperature) of the target distribution: 𝑝 ( 𝑥 ) ∝ 𝑅 ( 𝑥 ) 𝛽 , increasing 𝛽 results in a distribution skewed towards high-reward objects. We experimented with two 𝛽 values, 32 and 96 (Figure 3(a),3(d)).

Reward normalization. We used the following normalization rules for SEH, SA, and QED rewards in the molecule domain.

SEH = SEHraw ∕8 ;

QED = QEDraw .

SA = 10SAraw 9 ;

Training and evaluation. After training the base GFlowNets with the reward functions described above, we trained classifiers with Algorithm A.1. The classifier was parameterized as a graph neural network based on a graph transformer architecture [64]. Compared to the 2D grid domain (Section 6, '2D distributions via GFlowNet'), we can not directly evaluate the distributions obtained by our approach. Instead, we analyzed the samples generated by the composed distributions. We sampled 5 000 molecules from each composed distribution obtained with our approach as well as the base GFlowNets. We evaluated the sample collections with the two following strategies. Reward evaluation (Figure 3, Table 1) we analyzed the distributions of rewards across the sample collections. The goal is to see whether the composition of GFlowNets trained for different rewards leads to noticeable changes in reward distribution. Distribution distance evaluation (Figure 4, Table G.5) : we used the samples to estimate the pairwise distances between the distributions. Specifically, for a given pair of distributions represented by two collections of samples  𝐴 = { 𝑥𝐴,𝑖 } 𝑛 𝑖 =1 ,  𝐵 = { 𝑥𝐵,𝑖 } 𝑛 𝑖 =1 we computed the earth mover's distance 𝑑 (  𝐴,  𝐵 ) with ground molecule distance given by 𝑑 ( 𝑥, 𝑥 ′ ) = (max{ 𝑠 ( 𝑥, 𝑥 ′ ) , 10 -3 }) -1 - 1 , where 𝑠 ( 𝑥, 𝑥 ′ ) ∈ [0 , 1] is the Tanimoto similarity over Morgan fingerprints of molecules 𝑥 and 𝑥 ′ .

Training details and hyperparameters. The base GFlowNet policies were parameterized as graph neural networks with Graph Transformer architecture [64]. We used 6 transformer layers with an embedding of size 128. The input to the Graph Transformer was the graph of fragments with node attributes describing fragments and edge attributes describing attachment points of the edges.

The base GFlowNets were trained with trajectory balance loss. We used Adam optimizer. For the policy network 𝑝𝐹 ( 𝑠 ′ | 𝑠 ; 𝜃 ) , we set the initial learning rate 0 . 0005 and exponentially decayed with the factor 2 step ∕200000 . For the log of the total flow log 𝑍𝜃 we set the initial learning rate 0 . 0005 and

exponentially decayed with the factor 2 step ∕50000 . We trained the base GFlowNets for 15 000 steps with batch size 64 ( 64 trajectories per batch). In order to promote exploration in trajectory sampling for the trajectory balance objective, we used the sampling policy which takes random actions (uniformly) with probability 0 . 1 but, otherwise, follows the current learned forward policy.

| We trained the classifier with the loss described in Section 5.2. We used Adam with learning rate 0 . 001 . We performed 15 000 training steps with batch size 8 ( 8 trajectories sampled from each of the base models per training step). We updated the target network parameters 𝜙 as the exponential moving average (EMA) of 𝜙 with the smoothing factor 0 . 995 . We linearly increased the weight 𝛾 ( step ) of the non-terminal state loss from 0 to 1 throughout the first 4 000 steps and kept constant 𝛾 = 1 afterward. For the 𝛼 -parameterized version of the classifier (Section B), we used the following sampling scheme for 𝛼 : 𝑧 ̂ ∼ 𝑈 [-5 . 5 , 5 . 5] , 𝛼 ̂ = 1 .

The classifier was parameterized as a graph neural network with Graph Transformer architecture. We used 4 transformer layers with embedding size 128 . The inputs to the classifier were the fragment graph, terminal state flag ( {0 , 1} ), and log( 𝛼 ∕(1𝛼 )) (in case of parameterized operations). The classifier outputs were the logits of the joint label distribution ̃ 𝑄𝜙 ( 𝑦 1 , … , 𝑦 𝑛 | 𝑠 ) for non-terminal states 𝑠 and the logits of the marginal label distribution ̃ 𝑄𝜙 ( 𝑦 1 𝑥 ) for terminal states 𝑥 .

F.3 Colored MNIST Generation via Diffusion Models

The colored MNIST experiment in Section 6 ('Colored MNIST generation via diffusion models') follows the method for composing diffusion models introduced in Appendix A.2. The three base diffusion models were trained on colored MNIST digits generated from the original MNIST dataset. These colored digits were created by mapping MNIST images from their grayscale representation to either the red or green channel, leaving the other channels set to 0. For Figure 5 we post-processed the red and green images generated by the base models and compositions into beige and cyan respectively, which are more accessible colors for colorblind people.

Models, training details, and hyperparameters. The base diffusion models were defined as VE SDEs [34]. Their score models were U-Net [65] networks consisting of 4 convolutional layers with 64, 128, 256, and 256 channels and 4 matching transposed convolutional layers. Time was encoded using 256-dimensional Gaussian random features [66]. The score model was trained using Adam optimizer [59] with a learning rate decreasing exponentially from 10 -2 to 10 -4 . We performed 200 training steps with batch size 32.

The second conditional classifier ̃ 𝑄 ( 𝑦 3 | 𝑦 1 , 𝑦 2 , 𝑥 𝑡 ) was a similar convolutional network with 2 convolutional layers with 64 channels and two hidden layers with 256 units. This classifier is conditioned both on time and on ( 𝑦 1 , 𝑦 2 ) . The time variable was embedded used 128-dimensional Gaussian random features. The ( 𝑦 1 , 𝑦 2 ) variables were encoded using a 1-hot encoding scheme. The output of the classifier was the three predicted log-probabilities for 𝑦 3 . Contrary to the first classifier, this one was not trained on the base diffusion models but rather on samples from the posterior 𝑝 ̃ ( 𝑥 | 𝑦 1 , 𝑦 2 ) . It's loss function was:

The first classifier ̃ 𝑄 ( 𝑦 1 , 𝑦 2 | 𝑥𝑡 ) was a convolutional network consisting of 2 convolutional layers with 64 and 96 channels and three hidden layers with 512, 256 and 256 units. This classifier is time-dependent and used 128-dimensional Gaussian random features to embed the time. The output was a 3x3 matrix encoding the predicted log-probabilities. The classifier was trained on trajectories sampled from the reverse SDE of the base diffusion models using the AdaDelta optimizer [67] with a learning rate of 1 . 0 . We performed 700 training steps with batch size 128. For the first 100 training steps the classifier was only trained on terminal samples.

⎣ ⎦ where 𝑤̂ 𝑦 3 ( 𝑥 ̂ 0 ) is estimated using the first classifier. The classifier was trained using the AdaDelta optimizer [67] with a learning rate of 0 . 1 . We performed 200 training steps with batch size 128.

Sampling. Sampling from both the individual base models and the composition was done using the Predictor-Corrector sampler [34]. We performed sampling over 500 time steps to generate the

(a)

𝑝

1

(b)

𝑝

2

(c)

𝑝

1

⊗𝑝

2

(d)

𝑝

1

◑

𝑝

2

(e)

𝑝

1

◐

𝑝

2

samples shown in Figure 5. The samples used to train the classifier were generated using the same method.

When sampling from the composition we found that using scaling for the classifier guidance was generally necessary to achieve high-quality results. Without scaling, the norm of the gradient over the first and second classifier was too small relative to the gradient predicted by the score function, and hence did not sufficiently steer the mixture towards samples from the posterior. Experimentally, we found that scaling factor 10 for the first classifier and scaling factor 75 for the second produced high quality results.

G Additional Results

G.1 Binary Operations for MNIST Digit Generation via Diffusion Models

Here we present a variant of the colored digit generation experiment from Section 6 ('Colored MNIST generation via diffusion models'). using 2 diffusion models. This allows us to better illustrate the harmonic mean and contrast operations on this image domain. In a similar fashion to the experiment in Section 6 ('Colored MNIST generation via diffusion models'), we trained two diffusion models to generate colored MNIST digits. 𝑝 1 was trained to generate red and green 0 digits and 𝑝 2 was trained to generate green 0 and 1 digits. As before, we used post-processing to map green to cyan and red to beige.

Implementation details. The diffusion models used in this experiment and their training procedure were exactly the same as in Section F.3. The sampling method used to obtain samples from the base models and their compositions was also the same. We found that scaling the classifier guidance was generally required for high-quality results, and used a scaling factor of 20 in this experiment.

The classifier was a convolutional network with 2 convolutional layers consisting of 32 and 64 channels and two hidden layers with 512 and 128 units. The classifier's time input was embedded using 128-dimensional Gaussian random features. The output was a 2x2 matrix encoding the predicted log-probabilities ̃ 𝑄 ( 𝑦 1 , 𝑦 2 | 𝑥𝑡 ) . The classifier was trained on trajectories, sampled from the reverse SDE of the base diffusion models, using the AdaDelta optimizer [67] with a learning rate of 0 . 1 and a decay rate of 0 . 97 . We performed 200 training steps with batch size 128. For the first 100 training steps, the classifier was only trained on terminal samples.

Results. Figure G.1 shows samples obtained from the two trained diffusion models 𝑝 1 , 𝑝 2 and from the harmonic mean and contrast compositions of these models. We observe that the harmonic mean generates only cyan zero digits, because this is the only type of digit on which both 𝑝 1 and 𝑝 2 have high density. The contrast 𝑝 1 ◑ 𝑝 2 generates beige zero digits from 𝑝 1 . However, unlike 𝑝 1 , it does not generate cyan zero digits, as 𝑝 2 has high density there. The story is similar for 𝑝 1 ◐ 𝑝 2 , which generates cyan one digits from 𝑝 2 , but not zero digits due to 𝑝 1 having high density over those.

G.2 MNIST Subset Generation via Diffusion Models

We report in this section on additional results for composing diffusion models on the standard MNIST dataset. We trained two diffusion models: 𝑝 {0 , … , 5} was trained to generate MNIST digits 0 through 5,

(a)

𝑝

{0

,

…

,

5}

(b)

𝑝

{4

,

…

,

9}

(c)

𝑝

{0

,

…

,

5}

⊗𝑝

{4

,

…

,

9}

(d)

𝑝

{0

,

…

,

5}

◑

𝑝

{4

,

…

,

9}

(e)

𝑝

{0

,

…

,

5}

◐

𝑝

{4

,

…

,

9}

4

4

7

330133

3

3

4

5879

5

98

7

8b8

5

5

9

9

7

7

4

4

444444

7

h

5

S

S

3331

3

2

0

3222

3

20

3

3

2

D

998

9

7

9779

y

6

y99

5

4

4

4

79

8

8

h

5

了

2

33

3

2

7

7

7

S

4

5242

6

S

8

6

G

4

5

5

5

L

h

5

3

3

9

a

1

3

6

5

b

4

3

3

7

9

7

7

981

d

5

5

5

7

9

5

4

4

H

3

D

3

33

S

6

9

3

2

2

1

b

7

8

6

5

山

5

4

4

5

2

2

8

9

7

7

6

7

7

4

521

40

3

S

8

6

S

5

y

4

力

4

力

2

23320

了

2

9

9

9

b

6

4

and 𝑝 {4 , … , 9} to generate MNIST digits 4 through 9. The training procedure and models used in this experiment were the same as in Section G.1.

We also present binary operations between three distributions. In Figure G.3 and G.4, 𝑝 0 models even digits, 𝑝 1 models odd digits, and 𝑝 2 models digits that are divisible by 3 . We color digits {0 , 6} purple, {3 , 9} blue, {4 , 8} orange, and {1 , 5 , 7} beige. In Figure G.3, harmonic mean of 𝑝 0 and 𝑝 2 generates the digit 0 and 6 , whereas the contrast of 𝑝 0 with 𝑝 2 shows even digits non-divisible by 3 ( 𝑝 0 ◑ 𝑝 2 = {4 , 8} ), and odd numbers that are divisible by 3 ( 𝑝 0 ◐ 𝑝 2 = {3 , 9} ). We observe that the samples from 𝑝 0 ⊗𝑝 2 inherit artifacts from the base generator for 𝑝 2 (the thin digit 0 ), which shows the impact that the base models have on the composite distribution. In Figure G.4 we present similar results between odd digits ( {5 , 3 , 5 , 7 , 9} . We noticed that samples from both 𝑝 1 ◑ 𝑝 2 and 𝑝 1 ◐ 𝑝 2 includes a small number of the digit 3 .

Figure G.2 shows samples obtained from the two diffusion models, from the harmonic mean, and from the contrast compositions of these models. We observe that the harmonic mean correctly generates mostly images on which both diffusion models' sampling distributions have high probability, i.e. digits 4 and 5. For the contrasts we see that in both cases digits are generated that have high probability under one model but low probability under the other. We observe some errors, namely some 9's being generated by the harmonic mean and some 4's being generated by the contrast 𝑝 {0 , … , 5} ◐ 𝑝 {4 , … , 9} . This is likely because 4 and 9 are visually similar, causing the guiding classifier to misclassify them, and generate them under the wrong composition.

(a)

𝑝

0

: Even Digits

(b)

𝑝

2

:

{0

,

3

,

6

,

9}

(c)

𝑝

0

⊗𝑝

2

(d)

𝑝

0

◑

𝑝

2

(e)

𝑝

0

◐

𝑝

2

8

G

4

3

9

9

3

0

3

2

4

2228

H

3

9

6

3

9333

2

9

b

9

b

6

0

4

2

2

力

339

9

3

E66

3

8

2

3

S

9

3

33

？

８４８

9

8

4

2

8

8

b

3

330

h

3

9

8

8

？

9

9

9

8

4

8

3

3

993

9

9

2

S

8

8

4

3

9

9

9

3

9

3

3

3

3

9

3

3

2

2

Y

8

4

4

2

4

8

8

31

3

3

3

33993

S

3339

6

3

3

2

3

G

9

a

6

9

6

2

4

2

9

93336

3

4

8

D

h

2

9

?

333

O

4

889284

8

E

333394.

3

(a)

𝑝

1

: Odd Digits

(b)

𝑝

2

:

{0

,

3

,

6

,

9}

(c)

𝑝

1

⊗𝑝

2

(d)

𝑝

1

◑

𝑝

2

(e)

𝑝

1

◐

𝑝

2

9

9

5

b

6

5

b

7

3

9

9

3

0

3

3

3

93134

G

3

7

O

飞

6

5

6

6

3

3

9

D

9

9

b

6

3

9

93

33

9

9

9

S

9

q

5

7

7

3

3

S

9

3

3

？

99

3

6

99

9

3

？

7

9

S

q

5

9

9

3

9

3

，

9

3

3

993

7

S

3

7

9

9

9

3

3

3

3

3

3

3

9

3

3

5

7

3

9

3

3

9

3

9

3

5

9

9

3

9

9

1

5

6

3

3

3

9

a

6

6

6

3

9333

3

9

5

3

7

5

q

5

S

3

9

?

39333

O

3

933999

3

557

D

G.3 Chaining: Sequential Composition of Multiple Distributions

We present results on chaining binary composition operations sequentially on a custom colored MNIST dataset.

Setup. We start with three base generative models that are trained to generate 𝑝 1 , 𝑝 2 and 𝑝 3 in Figure G.5. Specifically, 𝑝 1 is a uniform distribution over digits {0 , 1 , 2 , 3 , 4 , 5} , 𝑝 2 is a uniform distribution over even digits {0 , 2 , 4 , 6 , 8} , and 𝑝 3 is a uniform distribution over digits divisible by 3 : {0 , 3 , 6 , 9} . Note that we use a different color for each digit consistent across 𝑝 1 , 𝑝 2 , 𝑝 3 . Our goal is to produce combinations of chained binary operations involving all three distributions, where two of them were combined first, then in a second step, combined with the third distribution through either harmonic mean ⊗ or contrast ◑ .

Binary Classifier Training. Consider, for example, the operation ( 𝑝 1 ⊗𝑝 2 ) ◑ 𝑝 3 . We use the same classifier training procedure for 𝑝 1 versus 𝑝 2 , as well as the composite model ( 𝑝 1 ⊗𝑝 2 ) versus 𝑝 3 , except that in the later case we sample from composite model as a whole. Our classifier training simultaneously optimizes the terminal classifier and the intermediate state classifier.

Implementation Detains. We adapted diffusion model training code for the base distributions from [33]. Our diffusion model used a UNet backbone with four latent layers on both the contracting and the expansive path. The contracting channel dimensions were [64 , 128 , 256 , 256] with the kernel size 3 , and strides [1 , 2 , 2 , 2] . The time embedding used a mixture of 128 sine and cosine feature pairs, with a total of 256 dimensions. These features were passed through a sigmoid activation and then expanded using a different linear head for each layer. The activations were then added to the 2D feature maps at each layer channel-wise. We used a fixed learning rate of 0 . 01 with Adam optimizer [59].

Weadapted the classifier training code from the MNIST example in Pytorch [68]. Our binary classifier has two latent layers with channel dimensions [32 , 64] , stride 1 and kernel width of 3 . We use dropout on both layers: 𝑝 1 = 25% , 𝑝 2 = 50% . We train the network for 200 epochs on data sampled online in batches of 256 from each source model, and treat the composite model in the second step the same way. We use the Adadelta [67] optimizer with the default setting of learning rate 1 . 0 .

Sampling. We generate samples according to Appendix A.2 and multiply the gradient by 𝛼 = 20 .

Results. Row 3 from Figure G.5 contains mostly zeros with a few exceptions. This is in agreement with harmonic mean being symmetric. In 𝑝 1 ⊗ ( 𝑝 2 ⊗𝑝 3 ) , digits that are outside of the intersection still appear with thin strokes.

(a)

𝑝

1

{0 - 5}

(b)

𝑝

2

{0

,

2

,

4

,

6

,

8}

(c)

𝑝

3

{0

,

3

,

6

,

9}

(d)

𝑝

1

⊗𝑝

2

(e)

𝑝

2

⊗𝑝

3

(f)

𝑝

1

⊗𝑝

3

(g)

𝑝

1

◑

𝑝

2

(h)

𝑝

2

◑

𝑝

3

(i)

𝑝

1

◐

𝑝

3

(j)

𝑝

1

◐

𝑝

2

(k)

𝑝

2

◐

𝑝

3

(l)

𝑝

1

◑

𝑝

3

𝑝

1

⊗

(

𝑝

2

⊗𝑝

3

)

𝑝

2

⊗

(

𝑝

1

⊗𝑝

3

)

𝑝

3

⊗

(

𝑝

1

⊗𝑝

2

)

𝑝

1

◑

(

𝑝

2

◑

𝑝

3

)

𝑝

1

◑

(

𝑝

2

⊗𝑝

3

)

𝑝

1

◑

(

𝑝

2

◐

𝑝

3

)

𝑝

2

◑

(

𝑝

1

◑

𝑝

3

)

𝑝

2

◑

(

𝑝

1

⊗𝑝

3

)

𝑝

2

◑

(

𝑝

1

◐

𝑝

3

)

𝑝

3

◑

(

𝑝

1

◑

𝑝

2

)

𝑝

3

◑

(

𝑝

1

⊗𝑝

2

)

𝑝

3

◑

(

𝑝

1

◐

𝑝

2

)

𝑝

1

⊗

(

𝑝

1

◑

𝑝

3

)

𝑝

1

⊗

(

𝑝

2

◐

𝑝

3

)

𝑝

2

⊗

(

𝑝

1

◑

𝑝

3

)

𝑝

2

⊗

(

𝑝

1

◐

𝑝

3

)

𝑝

3

⊗

(

𝑝

1

◑

𝑝

2

)

𝑝

3

⊗

(

𝑝

1

◐

𝑝

2

)

𝑝

1

⊗

(

𝑝

2

◑

𝑝

3

)

𝑝

1

◐

(

𝑝

2

⊗𝑝

3

)

𝑝

2

◐

(

𝑝

1

⊗𝑝

3

)

0

3

3

O

3

3

oo

2

003

300003

3

5

3

3

4

人

3

h

4

8

5

9

q

3

3

4

2

3

22

244

9

6

了力

Y44

O

D

1

S

3

B

2

8

2

2

02

L

人

3

3

3

2

2

4

A

2

6

9

3

S

3

2

3

4

3

3

4

4

Y

2の0

6

Terminal state loss

0

2500

5000

7500

10000

12500

15000

Steps

0

.

500

0

.

525

0

.

550

0

.

575

0

.

600

0

.

625

0

.

650

0

.

675

0

.

700

0

.

725

Terminal state loss

Non-terminal state loss

40

.

00

41

.

56

43

.

11

44

.

67

46

.

22

47

.

78

49

.

33

50

.

89

52

.

44

54

.

00

1

2

/u1D45D

1

2

+ 1

(

/u1D43F

/u1D43F

/u1D43F

1

1

1

(

/u1D45D

(

1

⊗/u1D45D

(

/u1D45D

1

/u1D45D

2

◐

2

,

,

2

/u1D45D

/u1D43F

1

/u1D45D

1

◐

7500

Steps

GT

,

05

GT

GT

(

(

/u1D45D

(

1

2

1

/u1D45D

1

/u1D45D

+ 1

⊗/u1D45D

1

/u1D45D

2

,

GT

10000

2

/u1D45D

2

)

◐

(

2

/u1D45D

/u1D45D

1

)

)

◐

)

0

.

05

2

)

/u1D45D

12500

)

2

)

)

15000

Non-terminal state loss

)

1

/u1D43F

1

1

1

1

1

0

0

0

0

0

Distribution distance (

.

.

.

.

.

.

.

.

.

.

8

6

4

2

0

8

6

4

2

0

0

2500

5000

0

2500

5000

7500

10000

12500

15000

Steps

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Terminal state loss

Non-terminal state loss

0

.

00

5

.

62

11

.

25

16

.

88

22

.

50

28

.

12

33

.

75

39

.

38

45

.

00

Non-terminal state loss

Terminal state loss

0

25

50

75

100

125

150

175

200

Steps

0

.

000

0

.

125

0

.

250

0

.

375

0

.

500

0

.

625

0

.

750

0

.

875

1

.

000

1

.

125

1

.

250

0

.

00

0

.

45

0

.

90

1

.

35

1

.

80

2

.

25

2

.

70

3

.

15

3

.

60

4

.

05

4

.

50

Terminal state loss

Non-terminal state loss

Terminal state loss

Non-terminal state loss

G.4 Classifier Learning Curves and Training Time

Weempirically evaluated classifier training time and learning curves. The results are shown in Figures G.6, G.7, G.8 and Tables G.1, G.2.

The runtime of classifier training is shown in Tables G.1 and G.2. We report the total runtime, as well as separate measurements for the time spent sampling trajectories and training the classifier. The classifier training time is comparable to the base generative model training time. However, most of the classifier training time (more than 70%, or even 90%) was spent on sampling trajectories from base generative models. Our implementation of the training could be improved in this regard, e.g. by sampling a smaller set of trajectories once and re-using trajectories for training and by reducing the number of training steps (the loss curves in Figures G.6, G.7, G.8 show that classification losses plateau quickly).

Figures G.6, G.7, G.8 show the cross-entropy loss of the classifier for terminal (16) and non-terminal states (18) as a function of the number of training steps for the GFlowNet 2D grid domain, the molecular generation domain, and the Colored MNIST digits domain respectively. They show that the loss drops quickly but remains above 0. Figure G.6 further shows the distances between the learned compositions and the ground truth distributions as a function of the number of training steps of the classifier. For all compositions, as the classifier training progresses, the distance to the ground truth distribution decreases. Compared to the distance at initialization we observe almost an order of magnitude distance reduction by the end of the training.

0

.

G.5 Analysis of Sample Diversity of Base GFlowNets in Molecule Generation Domain

In order to assess the effect of the reward exponent 𝛽 on mode coverage and sample diversity, we evaluated samples generated from GFlowNets pre-trained with different values of 𝛽 . The results are in Tables G.3 and G.4. The details of the evaluation and the reported metrics are described in the table captions. As expected, larger reward exponents shift the learned distributions towards high-scoring molecules (the total number of molecules with scores above the threshold increases). For 'SA' and 'QED' models we don't observe negative effects of large 𝛽 on sample diversity and mode coverage: the average pairwise similarity of top 1 000 molecules doesn't grow as 𝛽 increases and the ratio of Tanimoto-separated modes remains high. For 'SEH' models we observe a gradual increase in the average pairwise similarity of top 1 000 molecules and a gradual decrease in the ratio of Tanimoto-separated modes. However, the total number of separated modes grows as 𝛽 increases, which indicates that larger reward exponents do not lead to mode dropping.

G.6 Summary of Pairwise Distribution Distances in Molecule Generation Domain

Peer reviews:
comments: SUMMARY:
This paper proposes an approach of Compositional Sculpting for iterative generative models, including GFlowNets and defusion models.
The model uses classifier guidance to sample from the target posterior distribution composed of pre-trained base models.
The paper also proposes a training algorithm for the classifier.
The approach is validated by empirical analyses of an image dataset and molecular generation.

STRENGTHS:
- The method is general enough for different GFlowNets and defusion models.

- The empirical results validate the method.

WEAKNESSES:
(1) It would be convincing to have more complicated experiments, especially for image data.
Colored MNIST might be too small and simple.

(2) More clarity might be helpful for the following points.

(2A) What is the model in line 193? "Under the model we have introduced the variables y_1, ... y_n are dependent given a state s in S, but, are independent given a terminal state x in X."

(2B) I am confused while reading line 209. The sampling scheme is 1) sample y from its prior 2) sample tau given y. But the following sentence says sampling y given tau.

QUESTIONS:
Please refer to the weakness section.

Typos - Line 217: train train

LIMITATIONS:
The paper has a limitation section.
It seems not to have a dedicated section for social impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a way in which, given sequential samplers from multiple probability distributions, a combination of the samplers can be used to sample from a composition of the distributions. To be precise, the sequential samplers are either GFlowNets or diffusion models, the combination of samplers is a weighted combination of action distributions at each intermediate sampling step, and the composition of distributions can be defined by simple soft conjunction and set difference operators. This is demonstrated in toy illustratory experiments and multiobjective molecule synthesis (GFlowNet) and MNIST with digit class and colour attributes (diffusion).\n\nSTRENGTHS:\nFrom the perspective of someone who works on both GFlowNets and diffusion models, this is a very well-written paper. \n- The text reads naturally and the right amount of detail is given in the main text. There is a good choice of illustrations to help the reader.\n- The composition of multiple GFlowNets has not been considered before and could be useful, especially in multiobjective problems. The unifying perspective on classifier guidance is also an advantage (but see below).\n- Code is provided, a nice addition to the paper.\n- I checked the GFlowNet-related math and believe it to be sound.\n\nWEAKNESSES:\n- Line 234, 255, 634, 643, maybe others: typo \"GFLowNet\"\n- It would be good to explain why / state as a subclaim that (8) is a policy (i.e., sums to 1 over $s'$), which is not actually obvious from the definition. \n  - It relies on the fact that $p(y|s) = \\sum_{s'}p(y|s')p(s'|s)$, which follows from conditional independence of $y$ (a function of the final state) and $s$ given $s'$. That is a consequence of the Markov property in GFlowNets. A note should be made about this.\n  - The equality may not actually hold in practice, when $p(y|-)$ is a trained classifier, so (8) may not exactly sum to 1. What do you do in this case (in the experiments)?\n- The results on molecule generation raise a few questions:\n  - The reward exponent was set to $\\beta=32$ or $\\beta=96$, which is far larger than in past work, where it was at most 16. Why was such a choice made? This is suspicious, since convergence and mode collapse issues worsen at low temperatures.\n  - Related, with such high exponents, one wonders about mode coverage in the learned distributions. Have you considered the in-sample diversity of the generated molecules (e.g., as measured by average Tanimoto similarity or  diverse top-k metrics)?\n- On related work:\n  - There is no substantial discussion of related work in the main text, even though there is a large body of work on compositional generation and classifier guidance with diffusion models (e.g., the many papers cited in the second paragraph of the introduction).\n  - In the Appendix, the connection with [23] is discussed. The proposed method can easily turn a collection of classifiers for different objectives into a classifier for any convex combination of the objectives. It would be interesting to empirically compare this with conditioning of the model on the linear scalarization weights used in [23].\n  - The paper would be stronger if more explicit unifying connections were made between guidance in diffusion models and in GFlowNets. Note that diffusion models in a fixed time discretization are actually GFlowNets of a certain structure (cf. \"A theory of continuous generative flow networks\" [arXiv:2301.12594, ICML 2023] and \"Unifying generative models with GFlowNets and beyond\" [arXiv:2209.02606]). Classifier guidance using the gradient of $p_t(y|x_t)$ should be the continuous-time limit of equation (8).\n\nQUESTIONS:\nPlease see \"weaknesses\" above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of composing independently trained generative processes of diffusion-based generative models and GFlowNets. The paper considers a setting where one has access to $m$ pre-trained samplers for $\\{p_i(x)\\}_{i=1}^m$, and the goal is to obtain a sampler which corresponds to a composition of these processes. Specifically, the authors consider two ways of composing the processes, namely harmonic mean: where the likelihood of the composition is high only where the component processes have high likelihood and contrast The authors frame this as sampling from a conditional distribution $p(x|y)$, where y is an observation which denotes the index of the process a sample is generated from. This results in a procedure analogous to classifier guidance which is popular in the diffusion literature. The authors show how the classifier guidance results in sampling from the desired composition. A critical component of this procedure is learning the classifier $p(y|s)$, for which the authors propose a MLE based procedure using trajectories from the base model. The authors first validate their approach on some synthetic tasks on a 2D grid, followed by experiments on small molecule generation with GFlowNets and colored MNIST with diffusion.\n\nSTRENGTHS:\n* The paper studies an interesting question - which is relevant to the community. In particular, methods to leverage pre-trained models for various downstream tasks are becoming increasingly important with the growing adoption of pre-trained models for various domains. \n* Since similar classifier guidance approaches have been studied extensively in the literature on diffusion models, the novelty is relatively limited. Nonetheless, there are several technical aspects of the approach such as the classifier training scheme that are novel (to the best of my knowledge). \n* The proposed method is relatively simple conceptually, and in terms of implementation. \n* The experiments are well designed, and the results are quite promising, albeit with some caveats I mention below. \n* The paper overall is quite well written and easy to follow. I also appreciate the authors including the code with the submission.\n\nWEAKNESSES:\n* As the authors discuss in Section 3, their theoretical analysis is analogous to classifier guidance in diffusion models. On the other hand, [1] establishes equivalence between GFlowNets and diffusion models. As a results, it seems to me that the insights provided by the theoretical analysis aren’t particularly novel even though the path to achieving them was different (which could be seen as useful on it’s own)\n* The experiments on diffusion are limited to a simple coloured MNIST task, with no baselines from the classifier-guided diffusion literature. \n* The central aspect of the approach is learning the classifiers, however there is no analysis on the classifiers - e.g. how accurate are the classifiers? what is the effect of the classifier performance on the results of the composition? How does the training of the classifier compare to simply training a GFlowNet from scratch in terms of runtime? \n* (Minor) A simple baseline that is missing in the experiments is training a GFlowNet with the appropriate composition from scratch. \n\n[1] Unifying Generative Models with GFlowNets and Beyond. Zhang et al. 2022. arXiv:2209.02606\n\nQUESTIONS:\n* Could you please address the questions about the analysis of the effect of the classifier along with some runtime details? \n* What are the challenges in applying the method to more complicated tasks with diffusion models? \n* One natural question I had after reading the paper is that since the method encompasses GFlowNets and diffusion models, is it conceivable to be able to compose a mixture of GFlowNets and diffusion models on mixed continuous/discrete tasks? \n* In the experiments, the authors consider a maximum of 3 models being composed. Is there a reason to be limited to composing only 3 models? If not at least in the 2D example an experiment with more models might be helpful. \n\nMinor:\n* The main PDF on OpenReview appears to have some old version of the Appendix at the end. I assume that was by mistake? \n* In several places there is inconsistent usage of “GFLowNet” in place of “GFlowNet”.\n\nLIMITATIONS:\nThe authors already discuss some limitations of the approach - effect of the classifier as well as the quality of the underlying models. I would also add limited evaluation and lack of comparison to existing approaches.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to compose multiple iterative generative models, i.e., either multiple GFlowNets or multiple diffusion models. The idea starts out with a mixture model over the generative models. Then, one can construct a categorical distribution over the generative models that tells us which model a sample originated from. By adapting classifier guidance to GFlowNets, the proposed method can compose multiple models in a way that allows both emphasizing or de-emphasizing specific models by treating the different generative models as different classes. On a diverse set of (toyish) experiments the method is shown to be effective for both GFlowNets and diffusion models.\n\nSTRENGTHS:\nThe method is very interesting. In particular, the part where the question \"which model was this sampled from\" is treated as a classification task for the purpose of compositional generation.\n\nThe paper addresses a very important problem with high impact. \n\nThe presentation is easy to follow and the text is well-written.\n\nWEAKNESSES:\nThe experiments clearly demonstrate the effectiveness and versatility of the proposed method. Though, the experiments are limited to toyish settings and I believe more complicated settings would greatly enhance the impact of this work.\n\nQUESTIONS:\nAll state-of-the-art diffusion models for images allow for text conditioning. Conditioning on different texts can be viewed as creating multiple different generative models from the viewpoint of this paper. I believe this is a more realistic setting than assuming that one would want to combine multiple diffusion models that were trained entirely separately from one another. It would be interesting to see some evaluations of this setting. A cheap strategy to obtain a classifier for this could be to fine-tune (part of) the CLIP model.\n\nLIMITATIONS:\nThe main limitation is inherited from classifier guidance: the need to train a classifier on intermediate states. This is already mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe current paper focuses on the challenge of composition generation from pretrained generative models, with a specific focus on GFlowNets and Diffusion models. In comparison to prior literature, two novel compositionality operations are introduced for generating samples that are simultaneously likely according to two generative models or likely per a subset and unlikely per the remaining models. This is a strict generalization of operations introduced in prior work on composition of energy-based models. Practically, the operations are instantiated via a framework motivated by classifier guidance in diffusion models. Experiments are conducted on a molecule generation application and a colored MNIST problem.\n\nRebuttals acknowledgment: I had a good view of the paper before rebuttals and the authors' response to my questions was fair. I continue to keep my score accordingly.\n\nSTRENGTHS:\nI really like this work! The contributions are simple and straightforward, but very interesting. The formalization, generalized operations, and relation to classifier guidance were exciting to read through. The authors also appropriately acknowledge the limitations of their work, specifically the need for sufficiently strong component models.\n\nWEAKNESSES:\n- My biggest apprehension is limited experimental investigation, which would raise the quality of this paper quite strongly in my opinion. I do not hold this to be a strong weakness though.\n\nQUESTIONS:\nAs noted in the introduction, use of large scale generative models has been seen in several tasks in the foundation model era of machine learning. Given the limited empirical evaluation, I would like to gather the authors' thoughts on use of their frameworks for, say, controllable generation of language via LLMs or controllable generation of images via text-diffusion models (e.g., what would the model show for known compositionality failures of Dall-e and related methods? See [1]).\n\n[1] Conwell and Ullman, 2022. https://arxiv.org/abs/2208.00005.\n\nLIMITATIONS:\n- A clear note of what is meant by compositionality would help in this work, since the term is extremely overloaded. The experiments currently reported focus on what would be called systematicity or systematic generalization in my opinion (see Hupkes et al., \"Compositionality decomposed\"), but other valuable forms of compositionality, e.g., productive generalization, will arguably require some \"chaining\" operator that allows prior generated states to be fed into the model for generating the next state. Since the work focuses on GFlowNets and diffusion models, where a notion of sequential generation of intermediate states is present, arguably authors can use their defined operations to perform productive generalization as well? I would appreciate if the scope of this paper is clearly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to combine sequential generative models, in this case GFlowNets, so as to create new distributions from base models. This is done by training classifiers that are then used to guide sampling. The method is tested on a simple grid and a molecular domain (emulating the problem of the paper that introduced GFlowNet), as well as on MNIST with diffusion.\n\nSTRENGTHS:\nThe paper is moderately easy to read, although some of its results were not immediately clear to me so I spent quite some time doodling on paper to convince myself that the propositions were reasonable.\n\nWhat the paper proposes and seems to be able to achieve empirically is very interesting. Combining generative models in the ways shown here could be an amazing multiplier of large pretrained models.\n\nWEAKNESSES:\nGenerally the paper is not making a good job of convincing the readers that the proposed method should work at the theory level, and that the effort of combining distributions by training a classifier is worth it (compared to retraining a generative model).\n\nQUESTIONS:\nl211 and around, I’m not sure I understand the move to approximate $p(y_i|s)$ with $p(y_i|x)$, even when $s$ and $x$ are related by $\\tau =(s_0,…,s,…x)$ a valid trajectory. How or when are these two quantities interchangeable?\nThe objective in 12 involves $w_i$, but the paragraph following (11) seems to suggest that $Q(y_2, .., y_n|x)$ can be replaced by the $w_i$s. So I’m confused whether (12) uses the loss described in (11) or a modified version of it. Another worrying aspect of eq (12) is that there’s an $O(nm)$ sum, which sounds like it can get expensive, and there’s a product of $(n-1)$ probabilities, which sounds like it can get awfully numerically unstable. I’m surprised the authors have been able to train models at all. Are there any tricks involved?\n\nThe appendix is incomplete. In fact, part of the proposed contribution of the paper is to provide a combination method for diffusion methods. This is never quite explained properly; readers are directed to appendix D which is empty. In addition, although the propositions and theorems are analogs of past work, it was quite surprising to find theorems with no proofs in a paper. Even if the proof is almost identical to prior work, reproducing it with proper credit seems like the least thing to do; in this case there are nuances with the GFlowNet framework that are left unexplained.\n\nThe authors already highlight this limitation, but this seems like something quite fundamental that is for some reason not reported: **How expensive is it to train the classifier**? If it’s just as expensive as training a new generative model, then there’s little incentive to use the proposed method. I’d like not to have to take the authors’ word for it and instead see empirical evidence, training curves, wall time, and so on.\n\nAnother missing result is some validation that the learned distributions are the expected ones (e.g. a plot showing that the JS divergence goes to 0 with more training/capacity) at least on a toy setting like the grid environment.\n\nI really appreciate what the paper is trying to accomplish, and the empirical results seem very nice (although lacking some crucial results). I’m not sure though that the proposed method is correct, and with no proofs or extra details of why this should work I’m really inclined to reject this paper. Happy to engage in conversation with the authors of course.\n\nLIMITATIONS:\nThe authors have addressed some of the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}