Mechanisms of Projective Composition of Diffusion Models
Authors: Arwen Bradley
Preetum Nakkiran
David Berthelot
James Thornton
Joshua M. Susskind
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
composition
theory
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We theoretically study compositions of diffusion models.

Abstract: We study the theoretical foundations of composition in diffusion models, with a particular focus on out-of-distribution extrapolation and length-generalization. Prior work has shown that composing distributions via linear score combination can achieve promising results, including length-generalization in some cases (Du et al., 2023; Liu et al., 2022). However, our theoretical understanding of how and why such compositions work remains incomplete. In fact, it is not even entirely clear what it means for composition to "work". This paper starts to address these fundamental gaps. We begin by precisely defining one possible desired result of composition, which we call *projective composition*. Then, we investigate: (1) when linear score combinations provably achieve projective composition, (2) whether reverse-diffusion sampling can generate the desired composition, and (3) the conditions under which composition fails. We connect our theoretical analysis to prior empirical observations where composition has either worked or failed, for reasons that were unclear at the time. Finally, we propose a simple heuristic to help predict the success or failure of new compositions.

Mechanisms of Projective Composition of Diffusion Models
Arwen Bradley"
Preetum Nakkiran" "1
David Berthelot
James Thornton
Joshua M. Susskind
Abstract
Photos your dog
Oil paintings
Oil paintings of your dog
We study the theoretical foundations of composi- tion in diffusion models, with particular focus out-of-distribution extrapolation and length- generalization. Prior work has shown that com- posing distributions via linear score combination can achieve promising results, including length- generalization in some cases (Du et al., 2023; Liu et al., 2022). However, our theoretical un- derstanding of how and why such compositions work remains incomplete. In fact, it is not even entirely clear what it means for composition to "work". This paper starts to address these fun- damental gaps. We begin by precisely defining one possible desired result of composition, which we call projective composition. Then, we inves- tigate: (1) when linear score combinations prov- ably achieve projective composition, (2) whether sampling generate the de- sired composition, and (3) the conditions under which composition fails. We connect our theo- retical analysis to prior empirical observations where composition has either worked or failed, for reasons that were unclear at the time. Finally, we propose a simple heuristic help predict the success or failure of new compositions.
+
Figure 1: Composing diffusion models via score combina- tion. Given two diffusion models, is sometimes possible to sample in way that composes content from one model (c.g. your dog) with style of another model (e.g. oil paintings). We aim theoretically understand this empirical behavior. Figure generated via score composition with SDXL fine- tuned the author's dog; details in Appendix C.
P2(x)
Pa(x)
Pola)
ElPoPil
TPPPPPP
Projective Composttion
P.(x)
Pala)
8 PPPPPPPPPP ElPu.Pr.PPPPoPPAl
diffisson trained conditional
Samples from compositions of 2, 3, models
Figure 2: Length-generalizatie another capability of com- position enabled by our framework. Diffusion models trained to generate a single object conditioned on location (left) can be composed at inference-time to generate images of multiple objects at specified locations (right). Notably, such images are strictly out-of-distributton for the individual models being composed. (Additional samples in Figure 9.)
1. Introduction
The possibility of composing different concepts represented by pretrained models has been of both theoretical and practi- cal interest for some time (Jacobs al., 1991; Hinton, 2002; Du Kaelbling. 2024), with diverse applications including image and video synthesis (Du al., 2023; 2020; Liu al., 2022; 2021; Nie et al., 2021; Yang al., 2023a; Wang et al., 2024), planning (Ajay et al., 2024; Janner et al., 2022), con-
straint satisfaction (Yang et al., 2023c), parameter-efficient training (Hu al., 2022; Ilharco et al., 2022), and many others (Wu al., 2024; Su et al., 2024; Urain al., 2023; Zhang et al., 2025). One central goal in this field is to build novel compositions inference time using only the outputs of pretrained models (either entirely separate models, or different conditionings of a single model), to create gener- ations that are potentially more complex than any model could produce individually. As a concrete example to keep
'Equal contribution 'Apple, Cupertino, CA, USA. Correspon- dence to: Arwen Bradlcy <arwen. bradley@appleeccom Preetum Nakkiran <p.nakkiran apple.coppp......e..oo
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mechanisms of Projective Composition
in mind, suppose we have two diffusion models, one trained on your personal photos of your dog and another trained collection of oil paintings, and we want to somehow com- bine these to generate oil paintings of your dog. Note that in order to achieve this goal, compositions must be able to gen- crate images that are out-of-distribution (OOD) with respect to each of the individual models, since for example, there was no oil painting of your dog in either model's training set. Prior empirical work has shown that this ambitious vision is at least partially achievable in practice. However, the theoretical foundations of how and why composition works in practice, as well as its limitations, are still incomplete.
sition be possible? Here is one illustrative toy example- consider the following construction, inspired by but slightly different from Du et al. (2023); Liu et al. (2022). Suppose ps is distribution of empty background images, and each pr a distribution of images with a single object at location i, on an otherwise empty background. Assume all locations we wish to compose are non-overlapping. Then, performing reverse- diffusion sampling using the following score-composition will work meaning will produce images with k objects at appropriate locations:
V logp)(z) +
Σ V log - V log (()))
(2)
The goal of this work is to advance our theoretical under- standing of composition- we will take a specific family of methods used for composing diffusion models, and we will analyze conditions under which this method provably generates the "correct" composition. Specifically, are there sufficient properties of the distributions we are composing that can guarantee that composition will work "correctly"" And what does correctness even mean, formally?
score dolta 6, E R
Above, the notation p denotes the distribution p; after time & in the forward diffusion process (see Appendix E). In- tuitively this works because during the reverse-diffusion process, the update performed by model i modifies only pixels in the vicinity of location i, and otherwise leaves them identical to the background. Thus the different mod- els do not interact, and the sampler acts as if each model individually "pastes" object onto an empty background. Formally, sampling works because the score delta vectors 8 are mutually orthogonal, and in fact have disjoint sup- ports. Notably, we can sample from this composition with a standard diffusion sampler, in contrast to Du et al. (2023)'s observations that more sophisticated samplers are necessary. This construction would not be guaranteed to work, however, if the "background" ph was chosen to be the unconditional distribution Pa. (as in Equation 1), common choice in many prior works (Du et al., 2023; Liu et al., 2022).
We focus our study on composing diffusion models by lin- early combining their scores, method introduced by Du et al. (2023); Liu et al. (2022) (though many other interest- ing constructions are possible, see Section 2). Concretely, suppose we have three separate diffusion models, one for the distribution of dog images Pdog. another for oil- ppinntiing Post. and another unconditional model for generic images Pu- Then, we can use the individual score estimates V, log p(x) given by the models to construct a composite score:
y log fog((x):= ==
Vz z log Pdog (x) + V. (o)) V. lx)).(x).
The remainder of this paper is devoted to trying to generalize this example as far as possible, and understand both its explanatory power and its limitations. It turns out the core mechanism can be generalized surprisingly far, and does not depend on "orthogonality" strongly as the above example may suggest. We will encounter some subtle aspects along the way, starting from formal definitions of what it means for composition to succeed - a definition that can capture both composing objects (as in Figure 2), and composing other attributes (such as style + content, in Figure 1).
This implicitly defines distribution which we will call "product composition": p(z) & Paog (z)Poa(z)/PP (x). Fi- nally, we can try to sample from ğ by using these scores with a generic acore-based sampler, or even reverse-diffusio.. This method of composition often achieves good results in practice, yielding e.g. oil paintings of dogs, but it is unclear why it works theoretically.
We are particularly interested in the OOD generalization capabilities of this style of composition. By this we mean the compositional method's ' ability to generate OOD with respect to each of the individual models being composed - which may be possible even if none of the individual mod- els are themselves capable of OOD generation. A specific desiderata is megtt-gennrraiiation. understood as the ability compose arbitrarily many concepts. For example, consider the CLEVR (Johnson et al., 2017) setting shown in Figure 2. Given conditional models trained on images each containing a single object and conditioned on its location, we want to generate images containing k > 1 objects composed in the same scene. How could such
1.1. Contributions and Organization
In this work we introduce a theoretical framework to help understand the empirical success of certain methods of com- posing diffusion models, with emphasis on understanding how compositions can sometimes length-generalize. We start by discussing the limitations of several prior defini- tions of composition in Section 3. In Section 4 we offer formal definition of "what we want composition to do". given precise information about which aspects we want to compose, which we call Projective Composition (Defini- tion 4.1.. (Note that there are many other valid notions
compo-
2

---

Mechanisms of Projective Composition
of composition; we are merely formalizing one particular goal.) Then, we study how projective composition can be achieved. In Section 5 we introduce formal criteria called Factorized Conditionals (Definition 5.2), which is type of independence criteria along both distributions and coor- dinates. We prove that when this criteria holds, projective composition can be achieved by linearly combining scores (as in Equation 2), and can be sampled via standard reverse- diffusion samplers. In Section 6 we show that parts of this result can be extended much further to apply even in nonlin- ear feature spaces; but interestingly, even when projective composition is achievable, it may be difficult to sample. We find that in many important cases existing constructions approximately satisfy our conditions, but the theory also helps characterize and explain certain limitations. Finally in Section we discuss how our results can help explain exist- ing experimental results in the literature where composition worked or failed, for reasons that were unclear at the time. We also suggest simple practical heuristic to help predict whether new sets of concepts will compose correctly.
3. Prior Definitions of Composition
In this section we will describe why two popular mathe- matical definitions of composition are insufficient for our purposes: the "simple product" definition, and the Bayes composition. Specifically, neither of these notions can de- scribe the outcome of the CLEVR ongghh-genrralization experiment from Figure 2. Our observations here will thus motivate us to propose a new definition of composition, in the following section. As a running example, we will consider a subset of the CLEVR experiment from Figure 2. Suppose we are trying to compose two distributions Pl. of images each containing a single object in an otherwise empty scene, where the object is in the lower-left corner under pr, and the upper-right corner under p2- We would like the composed distribution ý to place objects in at least the lower-left and upper-right, simultaneously.
3.1. The Simple Product
The simple product is perhaps the most familiar type of com- position:
Given two distributions pi and p2 over IR", the simple product is defined' as p(z) x p1 (z). The sim- ple product can represent some interesting types of compo- sition, but it has a key limitation: the composed distribution can never be truly "out-of-distribution w.r.t. Pl or p2, since p(x) = 0 whenever pT (x) or p2 (z) = 0. This presents problem for our CLEVR experiment. Using the simple product definition, we must have p(x) 0 for any image x with two objects, since neither P1 nor P2 was supported on images with two objects. Therefore, the simple product definition cannot represent our desired composition.
2. Related Work
Single vs. Multiple Model Composition. First, we distin- guish the kind of composition we study in this paper from approaches rely a single model but with OOD condition- ers; for example, passing OOD text prompts to text-to-image models (Nichol et al., 2021; Podell et al., 2023), or works like Okawa et al. (2024); Park et al. (2024). In contrast, we study compositions which recombine the outputs of multiple separate models inference time, where each model only sees in-distribution conditionings. Among compositions involving multiple models, many different variants have been explored. Some are inspired by logical operators like AND and OR, which are typically implemented as product po())pi (ar) and sum po (xr + pi (z) (Du et al., 2023; Du & Kaelbling, 2024: Liu et al.. 2022). Some composition meth- ods are based on diffusion models, while others use energy / density approximations (Du et al., 2020; 2023; Liu et al., 2021; Thornton et al., 2025; Skreta et al., 2024). In this work, we focus specifically on product-style compositions implemented with diffusion models via a linear combina- tions of scores as in Du et al. (2023); Liu et al. (2022). Our goal is not to propose a new method of composition but to improve theoretical understanding of existing methods.
3.2. The Bayes Composition
Another candidate definition for composition, which we will call the "Bayes composition". was introduced and studied by Du et al. (2023); Liu et al. (2022). The Bayes composition is theoretically justified when the desired composed distri- bution is formally a conditional distribution of the model's 's training distribution. However, it is not formally capable of generating truly -utto--distribution samples, as our example below will illustrate.
Let us attempt to apply the Bayes composition methodology to our CLEVR example. We interpret our two distributions Pl, P2 as conditional distributions, conditioned on an ob- ject appearing in the lower-left or upper-right, respectively. Thus we write p(r|c;) = p1 (a)) where e₁ is the event that an object appears in the lower-left of image x, and 02 the event an object appears in the upper-right. Now, since we want both objects simultancously, we define the composi- tion as p(z) := p(x|c1, e2). Because the two events and C2 are conditionally independent given x (since they are
Learning and Generalization. In this work we focus only on mathematical aspects of composition, and we do not consider any
aspects such as inductive bias or sample complexity. Our work is thus complementary to Kamb & Ganguli (2024), which studies how a type of compositional generalization can arise from inductive bias in the learning procedure. Additional related works are discussed in Appendix A.
(=)) is also often used; our discussion applies cqually to this as well.
'The geometric mean (21())p2((
3

---

Mechanisms of Projective Composition
P((3)
Py(k)
Py(3)
PyDa)
I1
Emery Pe singk object
(b) Rayes, single object
X
11g
( Bayes, cluttrrd
Figure 3: Attempted composition of 3 objects. (a) Compo- sition succeeds for single-object distributions using empty background (as in Figure 2). (b) Bayes composition fails for single-object distributions. (c) Bayes composition succeeds for 1-5 object distributions. (Additional samples in Fig- ure 10; further gengraaanalliiiiioonn explored in Figure 11; quantitative analysis in Table 1.
Figure 4: Distribution ý is a projective composition of pl and pa w.r.t. projection functions (II, II2), because á has the same marginals as P1 when both post-processed by 11, and analogously for p2-
4. Our Proposal: Projective-oompsttion
We now present our formal definition of what it means to "correctly compose" distributions. Our main insight here is, a realistic definition of composition should not purely be a function of distributions (Pı,P2, }, in the way the simple product p(x) = pi (2)()) is purely a function of pi P2. We must also somehow specify which aspects of each dis- tribution we care about preserving in the composition. For example, informally, we may want composition that mim- ies the style of pı and the content of p2. Our definition below of projective composition allows us this flexibility.
deterministic functions of x), we can compute ş terms of the individual conditionals:
(x) == p(aler, x p(ala))(aloa))/p(
(3)
Equivalently in terms of scores: Va (x)
Ve loloop(((cc)) + v log p(z(o)) - V. p(x). Line (3)
thus serves as our definition of the Bayes composition p. in terms of the conditional distributions p(ar(e)) and p(xle2).
and the unconditional p(z)).
Roughly speaking, our definition requires specifying a "fea- ture extractor" II; : R + RR associated with every distri- bution pt. These functions can be arbitrary, but we usually imagine them projections in some feature-space, e.g, II (I) may be transform of æ which extracts only its style, and (2)))) transform which extracts only its content. Then, projective composition is any distribution á which "looks like" distribution p; when both are viewed through II, (see Figure 4). Formally:
The definition of composition above seems natural: we want both objects to appear simultaneously, so let us simply condition on both these events. However, there is an obvious error in the conclusion: f(x)) must be whenever p(æ (c) or p(zlo2) is zero (by Line 3). Since neither conditional distribution have support on images with two objects, the composition p cannot contain images of two objects either.
Where did this go wrong? The issue is: p(r]e1, 02) is not well-defined in our case. We intuitively imagine some un- conditional distribution p(z) which allows both objects si- multaneously, but no such distribution has been defined, or encountered by the models during training. Thus, the defi- nition of ý in Line (3) does not actually correspond our intuitive notion of "conditioning on both objects at once." More generally, this example illustrates how the Bayes com- position cannot produce truly -uito--distribution samples, with respect to the distributions being composed. ? Fig- ure 3(b) shows that the Bayes composition does not always work experimentally either: for diffusion models trained in our CLEVR setting of Figure 2, the Bayes composition of three locations typically fails to produce three objects. The difficulties discussed lead us to propose precise definition of what we actually "want" composition to do in this case.
Definition 4.1 (Projective Composition). Given a collec- tion of distributions {p:} along with associated "projection" functions (II, : R" - IR** we call a distribution ş a projective composition if*
Vi:
11 = 11,1p;
(4)
That is, when P is projected by each II, it yields marginals identical to those of pt.
There are a few aspects of this definition worth empha- sizing, which are conceptually different from many prior notions of composition. First, our definition above does not construct a composed distribution; merely specifies what properties the composition must have. For given
We use the term "projection" " informally here, to convey intu- ition; these functions TI, are not necessarily coordinate projections, although this is important special case (Section 5). "The notation refers to push-forward of a prohability measure.
Athooghh Du et al. (2023) use the Bayes composition to achieve kind of length--eneralizzaion, our discussion shows that the Bayes justification does not explain the experimental results.
4

---

Mechanisms of Projective Composition
also be described projective composition in almost an identical way, by letting II; nd R be a restriction onto the set of pixels neighboring location i. We describe this explicitly later in Section 5.3.
BG
5. Simple Construction of Projective Compositions
It is not clear apriori that projective compositional distri- butions satisfying Definition 4. ever exist, much less that there is any straightforward way to sample from them. To explore this, we first restrict attention to perhaps the sim- plest setting, where the projection functions {II,} are just coordinate restrictions. This setting is meant to generalize the intuition we had in the CLEVR example of Figure 2, where different objects were composed in disjoint regions of the image. We first define the construction of the composed distribution, and then establish its theoretical properties.
R
G
B
Figure 5: Composing yellow objects with objects of other colors. Yellow objects successfully compose with blue, cyan and magenta objects but not with brown, gray, green, red objects. Per the histograms (left), in RGB RGB-colorspace yellow has R. G distributed like the background (gray) while B has distinct distribution peaked closer to zero. Taking Myellow 2 {B}, Theorem 5.3 predicts that standard dif- fusion can sample from compositions of yellow with any color where the channel is distributed like the background: namely, blue, cyan, magenta per the histograms. (Other col- may theoretically compose per Theorem 6.1, but be difficult to sample.) (Additional samples in Figure 12.)
5.1. Defining the Construction
Formally, suppose we have a set of distributions (p1, P2....,.. that we wish to compose; our running CLEVR example, each p; the distribution of images with single object at position i. Suppose also we have some reference distribution ps. which can be arbitrary, but should be thought of "common background" to the p,s. Then, one popular way to construct composed distribution is via the compositional operator defined below. (A special case of this construction is used in Du et al. (2023), for example).
set of (pr,, I1)), there may be many possible distributions I which are projective compositions; or in other cases, projective composition may not even exist. Separately, the definition of projective composition does not posit any sort of "true" underlying distribution, nor does it require that the distributions p: are conditionals of an underlying joint distribution. In particular, projective compositions can be truly "out of distribution" with respect to the p:: P can be supported on samples where none of the are supported.
Definition 5.1 (Composition Operator). Define the compo- sition operator acting on an arbitrary set of distributions (pb: PI. (n.P2...)) by
1
(s(a)) (o
Clpl = Clpo.P1.P2 P1. [(x) Po
(5)
where 7. is the appropriate normalization constant. We name CTp the composed distribution, and the score of Cipl the compositional score:
Examples. We have already discussed the style+content composition of Figure an instance of projective com- position. Another even simpler example keep in mind is the following coordinate-projedd take II; RR" + R to be the projection onto the i-th coordi- nate. Then, a projective composition of distributions {p.} with these associated functions {II,) } means: a distribution where the first coordinate is marginally distributed identi- cally to the first coordinate of P1, the second coordinate marginally distributed as P2, and so on. (Note, we do not require any independence between coordinates). This notion of composition would be meaningful if, for example, we are already working in some disentangled feature space, where the first coordinate controls the style of the image the second coordinate controls the texture, and so on. The CLEVR length- gengraeennealiztiion example from Figure 2 can
V, log (xp)
(6)
case. Suppose we
= V. loggoopp))) + Σ V logp.(z) - V log (x()).
Notice that pa is taken to be the unconditional distribution then this is exactly the coyypeccomooiiioon
5.2. When does the Composition Operator Work?
We can always apply the composition operator to any set of distributions, but when does this actually yield "correct" composition (according to Definition 4.1)? One special case is when each distribution p; is "active" on a different, non- overlapping set of coordinates. We formalize this property
5

---

Mechanisms of Projective Composition
below as Factorized Conditionals (Definition 5.2). The idea is, each distribution p; must have particular set of "mask" coordinates M, S [no] which it samples in a characteristic way, while independently sampling all other coordinates from a common background distribution. If a set of dis- tributions (pbx P1. P2........ has this Factorized Conditional structure, then the composition operator will produce a pro- jective composition (as we will prove below).
Unpacking this, Line 9 says that the final generated distribu- tion f(x) can be sampled by first sampling the coordinates M, according to ps (marginally), then independently sam- pling coordinates M; according to p; (marginally) for each i. Similarly, by assumption, px(x) can be sampled by first sampling the coordinates M, in some specific way, and then independently sampling the remaining coordinates accord- ing to ps- Therefore Theorem 5.3 says that p(z) samples the coordinates Mj exactly they would be sampled by pi- for each i we wish to compose.
We say a sel of distributions (po. P1, ...... Pk) over Rn are Factorized Conditionals if there exists a partition of coordinates [n] into disjoint subsets Mo, M2,... M2 such that:
Definition 5.2 (Factorized-Conditionnaa
Proof. (Sketch) Since & satisfies Definition 5.2, we have
My) are independent under Pi- 2. (x] zz) ala.......... M) are mutually indepen- dent under pb-
1. (xr) a. ()M)
Clp((a) == (b(2)) II
= (x())) II pb(x)
Pi(x)
po ( (p: pb (x) (16())))))
(x) II P((alam.) po(2)a,,) II (( == p(x). (x)la,),
3. p=()) = (o(2lar)).
The sampling guarantee follows from the commutativity of com- position with the diffusion noising process, i.c. C(D²) N₂ [C[]]] The complete proof is in Appendix H.
Equivalently, we have:
p((x) = pa ) (x)))))))))) and
pb(x)) (a(aaaas)) ar,
II pb(2))M)) (E
Remark 5.4. In fact, Theorem 5.3 still holds under any orthogonal transformation of the variables, because the dif- fusion noise process commutes with orthogonal transforms. We formalize this as Lemma J.
Equation (7) means that each p4 can be sampled by first sam- pling z Pb. and then overwriting the coordinates of M, according to some other distribution (which can be specific to distribution i). For instance, the experiment of Figure 2 intuitively satisfies this property, since each of the condi- tional distributions could essentially be sampled by first sampling an empty background image (pp), then "pasting" a random object in the appropriate location (corresponding to pixels M2). If a set of distributions obey this Factorized Conditional structure, then we can prove that the composi- tion operator C. yields correct projective composition, and reverse- diffu---o correctly samples from it. Below, let N₂ denote the noise operator of the diffusion process at time t.
Remark 5.5. Compositionalityy is often thought of in terms of orthogonality between scores. Definition 5.2 implies orthogonality between the score differences that appear in the composed score (6): V. log p5,,) Va log p (21), but the former condition is strictly stronger (c.f. Appendix F).
Remark 5.6. Notice that the composition operator C can be applied to a set of Factorized Conditional distributions with- out knowing the coordinate partition [Mr) That is, we can compose distributions and compute scores without knowing apriori exactly "how" these distributions are supposed to compose (i.e. which coordinates p: is active on).
Theorem 5.3 (Correctness of Composition). Suppose set of distributions (p6. Pr. P2.... .P)) satisfy Definition 5.2, with corresponding masks (Miji. Consider running the reverse- -rifrs-diifussin SDE using the following compositional scores at each time t:
5.3. Example: Factorized Conditionals in CLEVR.
Let us explicitly describe how our definition of Factorized Conditionals captures the CLEVR setting of Figure 2. Re- call, the background distribution pa over n pixels images of an empty scene with objects. For each i E 2,, 3, 4), define the set M, C [n] as the set of pixel indices surround- ing location i. (M; should be thought of as a "mask" that that masks out objects at location i). Let Mo = (U.M.) be the remaining pixels in the image. Then, we claim the distri- butions (pb, Pl. P2, Ps, P4) form approximately Factorized Conditionals, with corresponding coordinate partition {M,)-
se(xe)= == V. log Clrb, Ph. Pi. (12r);
where pt : Ne(pe] are the noisy distributions. Then, the distribution of the generated sample zo at time t = 0 is:
p(x) = po(al II (p())))
(9)
In particular, p(x)(s,) p((x)M,, for all i, and so p is a projective composition with respect to projections (II, := 2/M hi per Definition 4.1.
'Note, the conditions of Definition 5.2 do not exactly hold in the experiment of Figure 2- c.g., there is still some dependence between the masks M, since objects can cast shadows or even occlude each other. Empirically, these deviations have greater impact when composing many objects, as seen in Figure 1 (a).
*Our results are agnostic the specific diffusion noise- schedule and scaling used.
6

---

Mechanisms of Projective Composition
This is essentially because each distribution Pt matches the background ps on all pixels except those surrounding loca- tion i. See Appendix B.2 for more details.
painting" does correspond to partieular subset of variables. In this section, we extend our results to the case where the composition occurs in some conceptual feature space, and each distribution to be composed corresponds to some particular subset of features.
5.4. Discussion
Importance of Background. Our derivations highlight the crucial role of the background distribution ps for the composition operator (Definition 5.1). While prior works have taken ps to be an unconditional distribution and the p,s its associated conditionals, our results suggest this may not be the optimal choice - in particular, it may not satisfy Factorized Conditional structure (Definition 5.2). Fig- attempt
Our main result is a featurized analogue of Theorem 5.3: if there exists any invertible transform A mapping into fea- ture space where Definition 5.2 holds, then the composition operator (Definition 5.1) yields projective composition in this feature space. See Figure 14 in Appendix I.
Theorem 6.1 (Feature-spiee space Composition). Given distri- butions ~. := (pb. P1 P2.... Pk), suppose there exists a C² diffeomorphism A: R - R"" (that is, A and A should be differentiable) such that (Atpb: Atpl.... Abps) satisfy Definition 5.2, with corresponding partition M, S [m]. Then, the composition p == C Clp) satisfies:
ure 3(a.b) shows a CLEVR experiment where to compose three diffusion models, each trained to generate a single object conditioned on location (as in Figure 2), us- ing two different backgrounds. In (a) we choose an empty background, which (together with the conditionals) approxi- mately satisfies Definition 5.2 and thus yields a projective composition. In (b), we form a standard Bayes composi- tion by using the unconditional distribution (i.e., single object in an arbitrary location) as background, which does not satisfy Definition 5.2 in this case. The former succeeds experimentally while the latter fails.
R:
Atf(z) = (Attp(z)) M.
II (=)p: (10)
==1
Therefore, P is projective composition of w.r.t. projection functions {II, (z) == A(z)) J.
Bayes composition may be approximately projective. However, Bayes composition can also often succeed. For example, the CLEVR dataset Du et al. (2023); Liu et al. (2022) contains images with multiple objects and the lo- cation of one randomly-chosen object. We replicate this experiment in Figure 3(c) and verify that the Bayes compo- sition succeeds (and may even work better for composing many objects than single-ojecct-emge wwwte-mmccgrtooooeeoeeee background, as in Figure 11). The unconditional together with the conditionals can approximately act as Factorized Conditionals in "clut- tered" settings like this one. The intuition is that if the conditional distributions each contain one specific object plus many independently sampled random objects ("clut- ter"), then the unconditional distribution almost looks like independently sampled random objects, which together with the conditionals would satisfy Definition 5.2 (further dis- cussion in Appendix B.2 and G). This helps to explain the length-generalizaaion observed in Liu et al. (2022).
This theorem is remarkable because means we can com- pose distributions (pb, P1. P2....) ) in the base space, and this composition will "work correctly" in the feature space automatically (Equation 10), without us ever needing to compute or even know the feature transform A explicitly.
Theorem 6.1 1 may apriori seem too strong to be true, since it somehow holds for all feature spaces AA simultancously. The key observation underlying Theorem 6.1 is that the compo- sition operator C behaves well under reparameterization.
Lemma 6.2 (Reparameterization Equivariance). The com- position operator of Defimition 5.1 is reparaneterization equivariant. That is. for all diffeomorphisms A : RR" + R." and all tuples of distributions p
(pb. P1, . ...). Pk).
ClAtp| = -AiC
(11)
This lemma is potentially of independent interest: equiv- ariance distinguishes the composition operator from many other common operators (e.g. the simple product). Lemma 6.2 and Theorem 6.1 are proved in Appendix I.
6. Projective Composition in Feature Space
6.1. Sampling from Compositions.
So far we have focused on the setting where the projection functions II; are simply projections onto coordinate subsets M, in the native space (e.g- pixel space). This covers simple examples like Figure but does not include more realistic situations such as Figure 1, where the properties to be composed are more abstract. For example property like "oil painting" does not correspond to projection onto specific subset of pixels in an image. However, we may hope that there exists some conceptual feature space in which "oil
The feature-space Theorem 6.1 is weaker than Theorem 5.3 in one important way: does not provide a sampling algo- rithm. That is, Theorem 6.1 guarantees that ò == Clpl is projective composition, but does not guarantee that reverse- diffusion is valid sampling method. Part of this is inherent: in the feature-space space setting, the diffusion noise operator N, no longer commutes with the composition operator C, so scores of the noisy composed distribution N, [C(r]] cannot be
7

---

Mechanisms of Projective Composition
computed from scores of the noisy base distributions N- Nevertheless, one may hope to sample from the distribution P using other samplers besides diffusion, such as annealed Langevin Dynamics, Sequential Monte Carlo (Thornton et al., 2025), or Predictor-Correct 2020). We find that the situation is surprisingly subtle: com- position C produces distributions which are in some cases easy to sample (e.g. with diffusion), yet in other cases ap- parently hard to sample. For example, in the setting of Figure 5, our Theorem 6.1 implies that all pairs of colors should compose equally well at time 1 = O, since there exist diffeommophisms (indeed, linear transforms) between different colors. However, as we saw, the diffusion sam- pler fails to sample from compositions of non-orthogonal colors- and empirically, even more sophisticated samplers such as Predictor-Corrector also fail in this setting. At first glance, it may seem odd that composed distributions are so hard to sample, when their constituent distributions are relatively easy to sample. One possible reason for this below is that the composition operator has extremely poor Lipchitz constant, so it is possible for a set of distributions p to "vary smoothly" (e.g. diffusing over time) while their composition Clyj changes abruptly. We formalize this in Lemma 6.3 (further discussion and proof in Appendix K).
we formalize as Factorized Conditionals. While such con- ditional structure typically does not exist in pixel-space, it is plausible that factorized structure exists in an appropri- ate feature space, permitted by our theory (Section 6). In particular, a feature space and distribution with perfectly "disentangled" features (Chen et al., 2018; Kim & Mnih, 2018; Yang et al., 2023b; Locatello et al., 2019) would sat- isfy our assumptions. Conversely, if distributions are not appropriately disentangled, our theory predicts that linear score combinations will fail to compose correctly. This ef- fect is well-known; see Figure 6 for an example; similar failure cases are highlighted in Liu et al. (2022) as well (such as "A bird" failing to compose with "A flower"). Re- garding successful cases, style and content compositions consistently work well in practice, and are often taken be disentangled features (e.g. Karras et al. (2019); Kotovenko et al. (2019); Gatys et al. (2016); Zhu et al. (2017)).
methods (Song et al.,
Text conditioning with location information. Condition- ing on location is straightforward way to achieve factorized conditionals (provided the objects in different locations are approximately independent) since the required disjointness already holds in pixel-space. Many successful text-to-image compositions in Liu et al. (2022) use location information in the prompts, either explicitly (e.g. "A blue bird on tree" + "A red car behind the tree") or implicitly ("A horse" + "A yellow flower field"; since horses are typically in the foreground and fields in the background).
Lemma 6.3 (Composition Non-Smoothness). For any set of distributions {pb, (1. PI......... Pp),, and any noise scale : o, define the noisy distributions p. := Nelp:), and let o' denote the composed distribution at time t: q' == ClP"). Then, for any choice of t > 0, there exist distributions Pp, pp) over IK" such that
Unconditional Backgrounds. Most prior works on diffu- sion composition use the Bayes composition, with substan- tial practical success. As discussed in Section 5.4, Bayes composition may be approximately projective in "cluttered" settings, helping to explain its practical success in text-to- image settings, where images often contain many different possible objects and concepts.
1. For all i, the annealing path of pt is Oip)-iipshtzz Vt, t': WW2(p',p5) S ((1))t=t')
2. The annealing path of q has Lipshitz constant at least 1) 3t, t': W2(g', q²)) > [(-e")
Intuitively, this means that, even if projective composition is possible at / = O, reverse diffusion (or indeed any annealing method) may not be able to correctly sample from it.
7.2. A practical heuristic
We now discuss a simple heuristic to help predict whether Factorized Conditionals (FC) holds for a particular set of concepts. Although it is not sufficient to guarantee Projec- tive Composition, it is easy-to-apply apply in practice to get an initial "hint" about whether composition is likely to work.
7. Practical implications
We have presented a mathematical theory of composition. Although this theoretical model is a simplification of reality (we do not claim its assumptions hold exactly in practice) we believe the spirit of our results carries over to practical set- tings, and can help both understand empirical observations from prior work, and make predictions about the success or failure of new compositions.
We start with lemma showing that FC concepts satisfy a simple (necessary, but not sufficient) orthogonality condi- tion. The short proof is in Appendix F.
Lemma 7.1. Let (pb. Pi..... PPk be Factorized Condition- als. Let μt; Ep, [x]] for i 1....,K .k and f4b Ep, [x] denote the mean vectors. Then, for any # j:
7.1. Connections with prior work
-b) (p, -pb) 0.
(12)
Independence Assumptions and Disentangled Features. Our theory relies on a type of independence between dis- tributions, related to orthogonality between scores, which
Thus, orthogonality between the mean difference vectors {14 - 3 is a necessary (but not sufficient) condition for Factorized Conditionals.
8

---

Mechanisms of Projective Composition
"photo of dog"+ "photo of horse" "photo dog" "photo, with red hat"
Cortne Stuileriy of CLIP Image Embesddigg Costne Sindlortt of CLIP Emmheddings
10V
15
Arg
Holo
maberoie
u'v Figure 7: Cosine similarity TTillee| between mean dif- ferent vectors μ4 Mb and J4j 14b, for each pair of con- cepts i,j, with μ4 estimated either the average CLIP image embedding over several images representative of con- cept i (left) or the CLIP text embedding for a single text description of concept (right). Details in Appendix D. Note that the groups of concepts ("dog", "horse", "cat"), ("watercolor", holl-pannting";; {"hat", "sunglasses") have high intra- and low sioe- similarity. This suggests that concepts from different groups may successfully compose (such as "dog"""hat" or "dog"**"il-paintinn" cepts from the same group may not (such as "dog"+ "oors"",rs"""), consistent with the examples in Figures and
Figure 6: Composing Entangled Concepts. The left image composes the texl-conditions "photo dog" with "photo of a horse", which both control the subject of the image, and produces unexpected results. In contrast, the right im- age composes "photo of dog" with "photo, with red hat," which intuitively correspond to disentangled features. Both samples from SDXL using score-compor composiiion with an un- conditional background; details in Appendix C.
while con-
This lemma makes precise the common intuition that some type of approximate "concept-space orthogonality" is quired for successful composition in practice, such as LoRA task arithmetic (Zhang et al., 2023a; Ilbarco et al., 2022).
8. Conclusion
In this work, have developed theory of one possible mechanism of composition in diffusion models. We study how composition for it to be achieved. Our theory can help understand a range of diverse compositional phenomena in both synthetic and practical settings, and we hope it will inspire further work on foundations of composition.
We can apply the lemma to help predict whether a new com- position may be successful. Importantly, since we often do not expect FC to hold in pixel-space, we need to pro- pose some feature-space in which FC might hold, and verify Lemma 7.1 there. We choose the CLIP (Radford et al., 2021) feature-space the proposal since it is simple-to-use and is known to provide reasonably disentangled represen- tation (though other feature spaces could also be used). In order to test Lemma 7.1 in csace, the procedure is: for each concept distribution Pt, collect several representative images, compute their CLIP embeddings, and average them to obtain the mean 14:- Similarly, estimate the background mean 14 using arbitrary images (representing the uncon- ditional distribution). Then check whether Equation (12)
be defined, and sufficient conditions
A Acknowledgements
We thank Miguel Angel Bautista Mar- tin, Etai Littwin, Jason Ramapuram, and Luca Zappella for helpful discussions and feedback throughout this work, and Preetum's 's dog Papaya for his contributions to Figure I.
Acknowledgements:
Impact Statement
approximately holds.
CLIP's goal of heuristic-for for-thh--euurstic simply using the text-embedding for each concept. That is, could approximate μi as the CLIP eoxteembedding of text description of concept p:- Although CLIP has been shown suffer from "modality gap" (text and image embeddings are not perfectly aligned) (Liang et al., 2022), the orthogonality structure we care about related to angles between centered-concepls - may Figure we show a prelim- inary experiment with both the image and text heuristies.
suggest an even easier
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Ajay, A., Han, S., Du, Y., Li, S., Gupta, A., Jaakkola, T., Tenenbaum, J., Kaelbling, L., Srivastava, A., and Agrawal, Compositional foundation models for hi-
still be fairly well- sell-preserved.

---

Mechanisms of Projective Composition
erarchical planning. Advances in Neural Information Processing Systems, 36, 2024.
Jacobs, R. A., Jordan, M. Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Chen, R. T., Li, X., Grosse, R. B., and Duvenaud, D. K. Isolating sources of disentanglement in variational au- toencoders. Advances in neural information processing systems, 31, 2018.
Janner, M., Du, Y., Tenenbaum, B., and Levine, Plan- ning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999 2099999 2022.
Delon, J. and Desolneux, A. wasserstein-type distance in the space of gaussian mixture models. SIAM Journal on Imaging Sciences, 13(2)933-9770, 2020.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, I.., Lawrence Zitnick, C., and Girshick, Clevr: diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Du, Y. and Kaclbling, L. P. Position: Compositional gener- ative modeling: A single model is not all you need. In Forty-first International Conference on Machine Learn- ing, 2024.
Kamb, M. and Ganguli, S. An analytic theory of creativ- ity in convolutional diffusion models. arXiv preprint arXiv:2412.20299 2024.
Du, Y. Li, S., and Mordatch, I. Compositional visual gen- eration and inference with energy based models. arXiv preprint arXiv:2004.0660 2020.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Diele- man, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. S. Reduce, reuse, recycle: Composi- tional generation with energy-based diffusion models and meme. In International conference on machine learning, pp. 1489-8510, PMLR, 2023.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Gatys, A., Ecker, S., and Bethge, M. Image style trans- fer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern 2016.
Kim, H. and Mnih, A. Disentangling by factorising. In International conference on machine learning, pp. 2649- 2658. PMLR, 2018.
recognition, pp.
Gregor, K., Danihelka, I., Graves, A., Rezende, D., and Wierstra, D. Draw: A recurrent neural network for im- age generation. In International conference on machine learning, pp. 1462-1471. PMLR, 2015.
Kotovenko, D., Sanakoyeu, A., Lang, S., and Ommer, B. Content and style disentanglement for artistic style trans- fer. In Proceedings of the IEEE/CVF international con- ference on compuler vision, pp. 4422-4431, 2019,
Hinton, G. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771- 1800, 2002.
Liang, W., Zhang, Y., Kwon, Y., Yeung. S., and Zou, J. Mind the gap: Understanding the modality gap in multi- modal contrastive representation learning. In NeurlPS, 2022. URL https: wpw.revieeeeeeeee// net forum? i id=$7Evzt 9 9uit
Ho, J. and Salimans, T. Classifier-free free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Liu, N., Li, S., Du Y., Tenenbaum, J., and Torralba, A. Learning to compose visual relations. Advances in Neural Information Processing Systems, 14:23166-23111 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances neural information process- ing systems, 33:6840-6851, 2020.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable dif- fusion models. In European Conference on Computer Vision, pp. 423 423-439 439. Springer, 2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.. Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing mod- els with task arithmetic. arXiv preprint 2rx::2221..04089, 2022.
Liu, Y., Zhang, Y., Jaakkola, T., and Chang, S. Correcting diffusion generation through resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8713-8723, 2024.
10

---

Mechanisms of Projective Composition
Locatello, F,, Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124, PMLR, 2019.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. pmlr, 2015.
Song, J., Meng, C.,, and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https: /pppreevee..eeeeeeeeeee net http:/ooummiidstgiiiaeaee
McAllister, D., Tancik, M., Song, J., and Kanazawa, A. Decentralized diffusion models. arXiv preprinl arXiv:2501.05450, 2025.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXiv:2/12.10744 2021.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, Score-based generative mod- eling through stochastic differential equations. arXiv preprint arXiv:2011. 2020. URL https:// arxiv. www.v.org//oi.e 1tt.//211........ 13456.346.. 1...5..35........
Nie, W., Vahdat, A., and Anandkumar, A. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34:13497-13510, 2021.
Stracke, N., Baumann, S. A., Susskind, J., Bautista, M. A., and Ommer, B. Ctrloralter: Conditional loradapter for efficient O-shot control and altering of t2i models. In European Conference on Computer Vision, pp. 87-103. Springer, 2024.
Okawa, M., Lubana, E. S., Dick, R., and Tanaka, H. Com- positional abilities emerge multiplicatively: Exploring diffusion models on synthetic task. Advances in Neural Information Processing Systems, 36, 2024.
Parisi, G. Correlation functions and computer simulations. Nuclear Physics B, 180(3):378-384 1981.
Su, J., Liu, N., Wang, Y., Tenenbaum, J. B., and Du, Y. Com- positional image decomposition with diffusion models. arXiv preprint arXiv:2406. 19298, 2024.
Park, C. F. Okawa, M., Lee, A., Lubana, E. S., and Tanaka, H. Emergence of hidden capabilities: Exploring learning dynamics in concept space. Advances in Neural Informa- tion Processing Systems, 11844688-87799 2024.
Thornton, J., Béthune, L., ZHANG, R., Bradley, A., Nakki- ran, P., and Zhai, S. Controlled generation with distilled diffusion energy models and sequential monte carlo. In The 28th International Conference on Artificial Intelli- gence and Statistics, 2025.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Urain, J., Li, A., Liu, P., D'Eramo, C., and Peters, J. Com- posable energy policies for reactive motion generation and reinforcement learning. The International Journal of Robotics Research, 42(10))827-858, 2023.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp.
Wang. Y., Liu, L., and Dauwels, Slot-vae: Object-centric scene generation with slot attention. In International Con- ference on Machine Learning, pp. 36020-36033 PMLR, 2023.
PmLR, 2021.
Robert, C. P., Casella, G., and Casella, G. Monte Carlo statistical methods, volume 2. Springer, 1999.
Wang, Z., Gui, L., Negrea, J., and Veitch, V. Concept alge- bra for (score-based) text-controlled generative models. Advances in Neural Information Processing Systems, 36, 2024.
Roberts, G. O. and Tweedie, R. L. Exponential convergence of langevin distributions and their discrete approxima- tions, 1996.
Rossky, P. J., Doll, J. D., and Friedman, H. L. Brownian dynamics as smart monte carlo simulation. The Journal of Chemical Physics, 69(10) 4633, 1978.
Wiedemer, T., Mayilvahanan, P., Bethge, M., and Brendel, W. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2024.
Skreta, M., Atanackovic, L., Bose, A. J., Tong, A., and Neklyudov, K. The superposition of diffusion mod- els using the it' o density estimator. arXiv preprint 177722 2024.
Wu, T., Maruyama, T., Wei, L., Zhang, T., Du, Y., lacearino, G., and Leskovec, J. Compositional generative inverse design. arXiv preprint arXiv:2401.1317/ 2024.

---

Mechanisms of Projective Composition
Yang, M., Du, Y., Dai, B., Schuurmans, D., Tenenbaum, J. B., and Abbeel, P. Probabiliistic adaptation of text-to- video models. arXiv preprint arXiv:2306.0166 010000 2023a.
Yang. T. Wang, Y., Lv, Y., and Zheng, N. Disdiff: Unsuper- vised disentangiement of diffusion probabilistic models. arXiv preprini wxXv::3301.177.. 2023b.
Yang. Z., Mao, J., Du, Y., Wu, J., Tenenbaum, J. B., Lozano- Pérez, T., and Kaelbling, I., Compositional diffusion- based continuous constraint solvers. arXiv preprint arXiv:2309.00990 2023c.
Zhang, J., Liu, J., He, J., et al. Composing parameter- efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589- 12610, 2023a.
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023b.
Zhang, L.., Rao, A., and Agrawala, M. Scaling in-the-wild training for diffusion-basse illumination harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations, 2025.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cynssscon adver- sarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-22122 2233, 2017.
12

---

Mechanisms of Projective Composition
A. Additional Related Works
Structured compositional generative models. Structured generative models leverage architectural inductive biases in an encoder-decoder framework, such as recurrent attention mechanisms (Gregor et al., 2015) or slot-attention (Wang et al., 2023). These models decompose scenes into background and parts-based representations in an unsupervised manner guided by modeling priors. While these approaches can flexibly generate scenes with single or multiple objects, they are not explicitly controllable, and require specific model pre-training on datasets containing compositions of interest.
Controllable generation. Composition at inference-time is one potential mechanism for exerting control over the generation process. Another way to modify compositions of style and/or content attributes is through spatial conditioning a pre-traie tre-trained diffusion model on a structural attribute (e.g.. pose or depth) as in Zhang et al. (2023b), or on multiple attributes of style and/or content as in Stracke al. (2024). Another option is control through resampling, as in Liu al. (2024). These methods are complementary to single or multiple model conditioning mechanisms based on score composition that we study in the current work.
Single model conditioning. We distinguish the kind of composition we study in this paper from approaches that rely on a single model but use OOD conditioners to achieve novel combinations of concepts never seen together during training; for example, passing OOD text prompts to text-to- text-to-image models (Nichol et al., 2021; Podell et al., 2023), or works like Okawa et al. (2024); Park et al. (2024) where a single model conditions simultaneously on multiple attributes like shape and color, with some combinations held out during training. In contrast, the compositions we study recombine the outputs of multiple separate models inference time. Though less powerful, this can still be surprisingly effective, and is more amenable to theoretical study since it disentangles the potential role of conditional embeddings.
Multiple model composition. Among compositions involving multiple separate models, many different variants have been explored with different goals and applications. Some definitions of composition are inspired by logical operators like AND and OR, usually taken to mean that the composed distribution should have high probability under all of the conditional distributions to be composed, or at least one of them, respectively. Given two conditional probabilities po(x). Pi (z). AND typically implemented as the product Po(z) (z) and OR as sum Po(x) + P1 (z) (though these only loosely correspond to the logical operators and other implementations are also possible). Some composition methods are based on diffusion models and use the learned scores (mainly for product compositions), others use energy-based models (which allows for OR-inspired sum compositions, as well as more sophisticated samplers, in particular sampling at t = 0 (Du et al., 2020; 2023; Liu et al., 2021), and still others work directly with the densities (Skreta et al., 2024) (enabling an even greater variety of compositions, including a different style of AND, taken to mean (u()) = PL (z)). McAllister et al. (2025) explore another type of OR composition. (Wiedemer et al., 2024) take a different approach of taking the final rendered images generated by separate diffusion models and "adding them up" in pixel-space. as part of study on generalization of data-generating processes. Task-arithmetti (Zhang et al., 2023a; Ilharco al., 2022), often using LoRAs (Hu et al., 2022), is kind of composition in weight-space that has had significant practical impact.
Product compositions. In this work, we focus specifically on product compositions (broadly defined to allow for a "background" distribution, i.e. compositions of the form p(z) (= II, (((z) implemented with diffusion models, which allows the composition to be implemented via a linear combinations of scores as in Du et al. (2023); Liu al. (2022). Our goal is not propose wholly new method of composition but rather to improve theoretical understanding of existing methods.
(s(
Learning and Generalization. Recently, Kamb & Ganguli (2024) demonstrated how a type of compositional generalization arises from inductive bias in the learning procedure (equivariance and locality). Their findings are relevant to our broader motivation, but complementary to the focus of this work. Specifically, we focus only on mathematical aspects of defining and sampling from compositional distributions, and we do not consider any learning-theorets aspects such as inductive bias or sample complexity. This allows us to study the behavior of compositional sampling methods even assuming perfect knowledge of the underlying distributions.
13

---

Mechanisms of Projective Composition
B. CLEVR Experimental Details
All our CLEVR experiments use raw conditional diffusion scores, without applying any guidance/CFG (Ho Salimans, Details below.
2022).
B.1. Dataset, models, and training details
B.I.1. CLEVR DATASET
We used the CLEVR (Johnson et al., 2017) dataset generation procedure' to generate datasets customized to the needs of the present work. All default objects, shapes, sizes, colors were kept unchanged. Images were generated their original resolution of 320 240 and down-sampled to lower resolution of 128 x 128 to facilitate experimentation and to be more GPU resources friendly. The various datasets generated from this procedure include:
A background dataset (0 objects) with 50,000 samples
Single object dataset with 1,550,000 samples
A dataset having 1 to objects, with 500,000 samples for each object count, for total of 2,500,000 samples.
Our experiments cover two different conditioning setups. In Figures 2, 9, 11, we condition on the 2D location of the object (or the location of one randomly-choses object, for multi-object distributions). In Figures 5, 12, we condition on the color of object. In all experiments we condition only single attribute (either location or color) at time, with all other attributes sampled randomly and not conditioned on.
1.1.2. MODEL ARCHITECTURE
We used our own PyTorch re- ce-implementation of the EDM2 (Karras et al., 2024) U-net architecture. Our re-implementation is functionally equivalent, and only differs in optimizations introduced to save memory and GPU cycles. We used the smallest model architecture, e.g. edm2 - img 644 -xs from https: / / thub. com/NVl abs Blllabs/edie /edm2. This model has a base channel width of 128, resulting in a total 124M trainable weights. Two versions of this model were used:
An unmodified version for background and class-conditioned experiments.
A modified version for () conditioning in which we simply replaced Fourier embeddings for the class with concatenated Fourier embeddings for and y.
1.3. TRAINING AND INFERENCE
In all experiments, the model is trained with batch size of 2048 over 128 220 samples by looping over the dataset as often needed to reach that number. In practice, training takes around 16 hours complete 32 A100 GPUs. We used almost the same training procedure in EDM2 (Karras et al., 2024), which is basically standard training loop with gradient accumulation. The only difference is that we do weight renormalization after the weights are updated rather than before as the authors originally did.
For simplicity, we did not use posthoc-EMs EMA to obtain the final weights used in inference. Instead we took the average of weights over the last 4096 training updates. The denoising procedure for inference is exactly the same as in EDM2 (Karras al., 2024), e.g. 65 model calls using a 32-step Heun sampler.
B.2. Factorized Conditionals in CLEVR.
1.2.1. SINGLE OBJECT DISTRIBUTIONS WITH EMPTY BACKGROUND
Let us explicitly describe how our definition of Factorized Conditionals captures the CLEVR setting Figure 2. Recall, the background distribution ps over n pixels images of empty scene with no objects. For each i (1, 3, 4), define the set M, E [n] as the set of pixel indices surrounding location i. Each M, should be thought of as "mask" that that masks
thub. www..c.ooooo......ooo.ooo..o..oo com
14

---

Mechanisms of Projective Composition
out objects at location i. Then, let M, (U,M,)* be the remaining pixels in the image, excluding all the masks. Now claim the distributions (p6, P1: P2,P3, P4) approximately Factorized Conditionals, with corresponding coordinate partition (Ma, M1, M2, Ma, M4). We can confirm each criterion in Definition 5.2 individually:
1. In each distribution pt, the pixels inside the mask M, are approximately independent from the pixels outside the mask, since the outside pixels always describe an empty scene.
2. In the background pt. the of masks {M. specify approximately mutually-independen sets of pixels, since all pixels are roughly constant.
3. The distribution of p; and ps approximately agree along all pixels outside mask M,, since they both describe an empty scene outside this mask.
Thus, the set of distributions approximately form Factorized Conditionals. However the conditions of Definition 5.2 do not exactly hold, since objects can cast shadows on each other and may even occlude each other. Empirically, this can significantly affect the results when composing many objects, as explored in Figure (1))).
B.2.2. CLUTTERED DISTRIBUTED WITH UNCONDITIONAL BACKGROUND
Figure 8: Samples from unconditional model trained on images containing 1-5 objects. The sampled images sometimes contain 6 objects (circled in orange).
Next, we discuss the setting of Figure 3(c), which is Bayes composition based unconditional distribution where each scene contains 1-5 objects (with the number of objects sampled uniformly). The locations and all other attributes of the objects are sampled independently. The conditions label the location of one randomly-chosen object. Just as in the previous case, for each i E 21, 3, 4}. we define the set M, E [r]] as the set of pixel indices surrounding location i, and let M, := (U,M, be the remaining pixels in the image, excluding all the masks. Again, we claim that the distributions (pb- Pr, P2, Ps. Ps) are approximately Factorized Conditionals, with corresponding coordinate partition (Mr, MI, M2, Ma, M4). We examine the criteria in Definition 5.2:
1. In each distribution p.. the pixels inside the mask M, are approximately independent from the pixels outside the mask, since the outside pixels approximately describe a distribution containing 0-4 objects, and the locations and other attributes of all objects are independent.
2. In the unconditional background distribution pb. we argue that in practice, the set of masks {Mi} are approximately mutually-independeer By assumption, the locations and other attributes of all shapes are all independent, and the masks M, are chosen in these experiment to minimize interaction/overlap. The main difficulty the restriction to 1-5 total objects, which we discuss below.
3. The distribution of p3 and ps approximately agree along all pixels outside mask Mj. since pa My contains 0-4 objects, while polasę contains 0-5 objects (since one object could be 'hidden' in M,9..
There are, however, two important caveats the points above. First, overlap or other interaction (shadows, etc.) between objects can clearly violate all three criteria. In our experiment, this mitigated by the fact that the masks M, are chosen to minimize interaction/overllp (though interactions start to occur we compose more objects, leading some image degradation). Second, since the number of objects is sampled uniformly from 1-5, the presence of one object affects the probability that another will be present. Thus, the masks {Mr} are not perfectly independent under the background distribution, nor do p. and Ps perfectly agree on MC. Ideally, each p; would place an object in mask M, and independently follow ps on Mf, and ps would be such that the probability that an object appears in mask M, is independently Bernoulli fc.f. Appendix G.2). In particular, this would imply that the distribution of the total number of objects is Binomial (which
15

---

Mechanisms of Projective Composition
allows the total object-count range from zero the toal-mumber-of-loctioon, as well as placing specific probabilities each object-count), which clearly differs from the uniform distribution over 1-5 objects. However, few factors mitigate this discrepancy:
A Binomial with sufficiently small probability of- places very little probability large k. For example, under Binomial(9, 0.3), P(k 0 5) 0.04, 0.156, 0.27, 0.27, 0.17, 0.07 and Plk 5) 0.026.
Empirically, the leamnd unconditional distribution does not actually enforce hs < 5; we sometimes see samples with k: = 6 for example, as seen in Figure 8.
Intuitively, the train distribution is "close to Beroulli" and the learned distribution seems to be even closer.
With these considerations in mind, we see that the set of distributions approximately - though imperfectly - form Factorized single-object setting is that models can learn how multiple
Conditionals. One advantage of this setting compared objects should interact and even overlap correctly, potentially making easier compose nearby locations. We explore the length-generraizaaton of this composition empirically in Figure Il (note, however, that only compositions of more than objects are actually OOD w.r.t. the distributions p; in this case).
B.3. Additional CLEVR samples
In this section we provide additional non-cherrypicked samples of the experiments shown in the main text.
Table I: Quantitative analysis of different methods of composition of location-conditioned CLEVR distributions. generated 100 samples using each composition method, and manually counted (to avoid any potential error in using classifier) the objects in correct locations (i.e. locations corresponding the conditioners of the distributions being composed) in each generated image. The table shows the histogram of these manual counts, that is, each column lists the number of images that contained the given number of objects in correct locations. N denotes number of distributions being composed (hence the expected number of objects) - we test N = through N = 6. Single-object empty composes single- object object distributions with an empty background. Single-object Bayes composes single-object object distributions with an unconditional background. Bayes-cluttered composes 1-5 object distributions (with location label assigned to a single randomly-chon object) with an unconditional background.
Style Single-object empty
N
1
4
5
1
100
2
100
3
99
4
98
5
2
98
6
3
97
Single-object Bayes
1
100
2
10
67
32
3
36
62
4
23
5
66
32
6
34
6
3
Bayes-cluttered
1
100
2
100
3
100
4
100
5
2
98
6
2
16

---

Mechanisms of Projective Composition
Pi())
P2(x)
P2(x)
P4(x)
P₂(x)
8(pbpP]]
BlPp-Pi-Pal
PiPpPPPPPPPl
PPPPPPPPPPPPPlll
Figure 9; Additional non-cherrypicked samples for CLEVR experiment of Figure 2.
Empty background, single object
(b) Bayes, single object
pute)
Bayes,
Figure 10: Additional non-cherrypicked samples for CLEVR experiment of Figure
17

---

Mechanisms of Projective Composition
Experiment (A);
Experiment (B):
Experiment (C):
Empty background Po. single object
Uncomditional background single object
Multiple multiple object
Distribatiom
Bockground Pb
Pt
P2
Ps
P4
8lP-Pr-AAll
Suboess OOD
purtial suocess OOD
SUDDESS distribbttoon X fullure
Figure 11: Attempted compositional anntth-gnnrrliizain up to 9 objects, in the setting of Figure 3 (but with more-closely spaced objects). We attempt to compose via linear score combination the distributions P1 through py shown the far left, where each is conditioned on specific object location as described below. Settings (A) and (C) approximately satisfy the conditions of our theory of projective composition, and thus are expected to -ength-generalize at least somewhat; while setting (B) does not even approximately satisfy our conditions and indeed fails to taggth-generalize. Experiment (A): In this experiment, the distributions p; each contain single object at fixed location, and background ps is empty. this case any successful composition more than one object represents several objects, but then degrades number of objects increases (see Section 5.3 for details). Experiment (B): Here the distributions P; are identical to (A), but the background is chosen the unconditional distribution (i.e. single object at random location)- this the "Bayes composition" (Section 3). This composition entirely fails remarkably, trying to compose many objects often produces no objects! Experiment (C): Here each distribution p4 contains object at fixed location i, and 0 - 4 other objects (sampled uniformly) in random locations; see samples at far left. The background distribution ps a distribution
We find that composition succeeds up
5 objects (sampled uniformly) in random locations. this case brnggh--eneralization means composition of more than objects. This composition can length- generalize, but artifacts appear for large numbers of objects. See Appendix B. for full discussion.
18

---

Mechanisms of Projective Composition
Po
P1
P2
C Pp. Pil
P3
P4
Ps
P6
P7
8 81 1247/111 1- 21
21
Figure 12:
Additional non-cherrypicked samples for CLEVR experiment of Figure 5. Top left grid shows conditional
samples for each color. Top right grid shows compositions of red-colored objects (pa) with objects of other colors (8 samples of each), which only succeeds for cyan-colored objects. Bottom grid shows compositions of yellow-colored objects (p7) with objects of other colors (16 samples of each): these are additional samples of the exact experiment shown in Figure 5.
19

---

Mechanisms of Projective Composition
C. SDXL experimental details
C.1.
Figure!
The two models composed are
1. An SDXL model (Podell et al., 2023) fine-tuned on 30 personal photos of the author's dog (Papaya).
2. SDXL-base- (Podell al., 2023) conditioned on prompt "an oil painting in the style of van gogh."
The background score distribution is the unconditional background (i.c. SDXL conditioned on the empty prompt). We use the DDPM sampler (Ho al., 2020) with 30 steps, using the composed score, and CFG guidance weight of 2 Ho et al. (2020).
Note that using guidance weight 1 (i.c. no guidance) also performs reasonably in this case, but is lower quality.
C.2. Figure
Left: The score models composed are
1. SDXL-base- (Podell et al., 2023) conditioned on prompt "photo a dog"
2. SDXL-base-11. (Podell et al., 2023) conditioned on prompt "photo of a horse"
The background score distribution the unconditional background (i.e. SDXL conditioned on the empty prompt).
Predictor-Corrector method (Song et al., 2020) with the DDPM predictor and the Langevin dynamics corrector, both operating on the composed score. We use 100 predictor denoising steps, and 3 Langevin iterations per step. We do not use any guidance/CFG.
For improved sample quality,
Right: Identical setting as above, using prompts:
1. "photo of a dog"
2. "photo, with red hat"
Note that the DDPM sampler also performed reasonably this setting, but Predictor-Corrector methods improved quality.
D. CLIP experiment details
In the CLIP experiment of Figure 7, we used the following text prompts for each concept:
"dog": "a photograph of dog",
"horse": "a photograph of horse", "cat": "a photograph of cat", "watercolor": "a watercolor painting", "oil-painting": "an oil painting", "hat"; "wearing a hat", "sunglasses"; "wearing sungl asses", "uncond":
We did automated collection of 10 images for each concept initially and then manually filtered them to ensure that they actually representative of each concept, leaving with 10 for "dog". 2 for "horse", 9 for "cat", 3 for "watercolor", 4 for "oil-painting". 6 for "hat", 5 for "sunglasses"" 10 for "uncond" (the latter were arbitrary images searched with no specific keyword).
For each concept i, we ran the image experiment by computing the CLIP embedding of each representative image and averaging them to estimate the mean (4, Similarly, we estimated the background mean ls by using the arbitrary images
20

---

Mechanisms of Projective Composition
(representing the unconditional distribution). For the experiment simply estimated 14i Pb the CLIP embedding of the single representative text prompt (or empty prompt).
In either then computed the cosine similarity between the mean difference vectors 14 (4-1)) (n -1,) P4-M,lll --- to assess
whether the condition of Lemma 7.1, ie. μ4 -14) 1a) z 0 approximately holds.
E. Reverse Diffusion and other Samplers
E.1. Diffusion Samplers
DDPM (Ho et al., 2020) and DDIM (Song et al., 2021) are standard reverse diffusion samplers (Sohl-Dickstein et al., 2015; Song & Ermon, 2019) that correspond to discretizations of a reverse-SDE and reverse-ODE, respectively (so we will sometimes refer to the reverse-SDE DDPM and the reverse-ODE DDIM for short). The forward process. reverse-SDE, and equivalent reverse-ODE. (Song et al., 2020) for the variance-preserrvnng (VP) (Ho al., 2020) conditional diffusion
1
Forward SDE dz
2 Brodt + v Bidw.
(13)
DDPM SDE:
dx
2 Bex dt V. logp:(rlc)dt+ + BB,d0
(14)
1
DDIM ODE: dæ
Bix dt
3 V. logp(())))
(15)
2
2
E.2. Langevin Dynamies
Langevin dynamics (LD) (Rossky et al., 1978; Parisi, 1981) an MCMC method for sampling from desired distribution. given by the following SDE (Robert et al., 1999)
dæ
v log p(r)dt + Vedw,
(16)
2
which converges (under assumptions) p(x:) (Roberts & Tweedie, 1996). That is, letting Ps (xx) denote the solution of LD at time we have lim (a(x) p(x).
F. Factorized Conditionals vs. Orthogonality
Lemma 7.1 states that Factorized Conditionals (Definition 5.2) implies orthogonality between mean differences (providing a senesarr--bu--no-uuffficce condition to check for FC). The proof straightforward:
Proof. (Lemma 7.1)
Pa p)( by FC
#:: E(x], i I....,k. 146 = E [c]
(ui) (Hu)aag
Support(14: 1b)) C M,
(#t 10 (ng Hb) = 0. since M, n M, = 9
Similarly, Definition 5.2 also implies orthogonality between the score differences (recall that the score is related to the
21

---

Mechanisms of Projective Composition
conditional mean V logp'(x) J. Epl= x 24 12]. this is closely related to Lemma 7.1). To this:
(x) v (xx) - V log (2x,
V. log p((x) = V log p) (4)())))))
pp(2)
V log (x
p1(2|M,)
vk $ M;
(() = 0,
vi # j.
since M, n M, = =0,
where in the second-to-last line we used the fact that the gradient of function depending only on a subset of variables has zero entries in the coordinates outside that subset.
In fact, the same argument implies that (vf((r) E JR C M2; in other words, {x)(2) x RR and (vf(2) E R"} occupy mutually-orthogome subspaces. But even this latter condition does not imply the stronger condition of Definition 5.2. To find an equivalent definition in terms of scores we must also capture the independence of the subsets under ps. Specifically:
p((()) p ())
pb(TT) p(( [pb(
V log (x()) (ogp + V. (x)p)())))=))
v. logr((z) v log (og
Σ v logr((sl
Va logp/()) V logg/(x)) x log p M, pj (=|)),0))
V logr(())
v log pi m1)) + Σ Vn log (()l()))
So an definition equivalent to Definition 5.2 in terms of scores could be:
Definition F.1. The distributions (ph, P1, P2....) form factored conditionals if the score-deltas of == V, (o) v log pf (z) satisfy (x((2) x E R") My. where the Mj are mutually-ortogonad subsets, and furthermore the score of the background distribution decomposes over these subsets follows: V, log V. log p (x) +
Σ, V log (x)()M)).
(Note: this actually equivalent to a slightly more general version of Definition 5.2 that allows for orthogonal transformations, which is the most general assumption under diffusion sampling generates a projective composition, per Lemmas 6.1 and J.1.)
G. Connections with the Bayes composition
G.1. The Bayes composition and length-generalliation
We give counterexample for which the Bayes composition fails to length-generalize, while composition using an "empty background" succeeds. The example corresponds to the experiment shown in Figure 13 (left), Suppose we have conditional distributions p. that set a single index i to one and all other indices to zero, a zero-background distribution and an unconditional distribution formed from the conditionals by assuming p(c = i) is uniform. That is:
|| - eill
p((,,) = -N]x:N x exp
20
||Te|
px(x:) = Nixz; 0, 0, of) x exp
20
(x)) 1 Σ ((()))
(17)
n
22

---

Mechanisms of Projective Composition
Pg(z)
P,00
PaDt
P.00
8 Poy
G3 P,00
6=1.
<=3
624
626
c=
P.(0)
p,00)
Bayes: 8 Por Pa.P
P.(x
Bayos: 8 Par Pa.PPPPPPP
a cm6
e-0
e=0 cal. +
a c-5
c=1.
6=3
6m3.
Figure 13: Bayes composition vs. projective composition. All experiments use exact scores, which is possible since the diffusion-noised distributions are Gaussian mixtures. (Left) Distributions follow (17): each conditional p; activates index i only, unconditional Pu. averages over the Pi, and background ps is all-zeros. We attempt to compose the conditions Po. P2. Pt.Pa Ps and hope obtain the result [1, 0, 1, 0, 0j. This requires tionals Pt contains only single 1. The composition using the empty background ps (top) achieves this goal, while the Bayes composition using the unconditional Pu. (bottom) does not. Note that [ps, P1.P2:... satisfy Definition 5.2 while [Pu. PI: ... does not. (Right) Distributions follow (18), where each conditional p; activates index on an independently 'cluttered' background. In this case the unconditional is similar to the cluttered background. Again we attempt to compose Po, P2, Ps, Ps. and in this case we find that the composition using Ps works similarly well to pb-
since each of the condi-
Suppose we want to compose n distributions pt. that is, we want to activate indices. enough to consider æt the special form I; (a.... a) since there is no reason favor any condition over any another. Making this restriction,
(n (n= + (1 -ap)
-exp na² 22 + 1). Vi
(a. (a...., = px())) exp
20
20
na² - 2a + 1
pl,(x)) exx exp
207
na²
pb(x1) exp
20
Let us find the value of a that maximizes the probability under the Bayes composition of all condition:
(1xe)
x = (a,
p)(
(- na² 2a +1
(_n(a n(a 4)2 + const
= (x))
(xt) exp
exp
p (xt)
20
20
so the optimum is a* 1. That is, under the Bayes composition the most likely configuration places value 1 n at each index we wished to activate, rather than the desired value 1.
23

---

Mechanisms of Projective Composition
On the other hand, if we instead use Ps in the linear score combination and optimize, we find that:
(xre
2ax 20
2t = (a)...,
(xx) & exp
a (are)
na
ni(1 -
2ax) 2
nnnaa - -2a + 1)
p(()))
& exp
20 exp
x exp
(x1)
20
n(a
1)2
x exp
20
=1
so the optimum is o² = 1. That is, the most likely configuration places the desired value 1 at each index we wished to activate, achieving projective composition, and in particular, gegtt-ggnnralizing correctly.
G.2. Cluttered Distributions
In certain "cluttered" settings, the Bayes composition may be approximately projective. We explore this in the following
simplified setting, corresponding to the experiment in Figure 13 (right). Suppose that x is binary-valued, M; = {i}. Vi, the r; are independently Bernoulii with parameter under the background, and the projected conditional distribution p=(x)) just guarantees that 2;
p=(2]1=) ~ (2lss,, iid. Vi, (=(x))) Lr,=1,
(18)
The distributions (pb. Pl. PI.P2....) then clearly satisfy Definition 5.2 and hence guarantee projective composition. In this case, the unconditional distribution used in the Bayes composition is similar to the background distribution if number of conditions is large. Intuitively, each conditional looks very similar to the Bernoulli background except for single index that is guaranteed to be equal to 1, and the unconditional distribution is just weighted sum of conditionals. Therefore, we expect the Bayes composition to be approximately projective.
More precisely, we will show that the unconditional distribution converges to the background in the limit as n + co, where n is both the data dimension and number of conditions, the following sense:
Pu(x)) po(x)) (x(
27 + 0 as 12 + cc.
E
am-pb
We define the conditional and background distributions by:
E R
Mf {i}
(o(p))) ~ Bern, i.i.d. for i = 1,...,n
(+(k) = 1x/1=1, for all i I,...,
(6( q mmz(()) 9)" (=n=((x)
p.(x) = (x]::-)) (m===ge (1- -In-nnnns(alee
We construct the unconditional distribution with assuming uniform probabibility over all labels: p=(x) = -, The number-of-nonzeros (nnz) in all of these distributions follow Binomial distributions:
I ps po(nnz(x: = k) ~ Binom (k; n, 4)
x. ~ P; p.(nnz(x) k) = Pb (nnz (x/p=) = 1111) 1)
Binom 1:n 1.
if > else
Pu pa(nnz(x)= k) Σ (m)=(()) kb)
Binom 1;n-1.9h
if k: else
24

---

Mechanisms of Projective Composition
The basic intuition is that for large and Pb precisely, calculate:
Binom (k; n, q) and Pu
Binom(k: 1:n q) are similar. More
pu(x)) pb(2))
)
27
(ma(f)
B(k: 1;n - 1,9) k
E Pm
IE pp
-
since
(x)
qm
(kk; n,
gm.
( -1
1
E (inomm)))).))
E (nn-Binom(n.r)
[(t nn)
qn
(nq)²
(nq)² Var(k), ~ (nnom((..))
1-9 +0
(ng)² ng - -Q)
as
ng
H. Proof of Theorem 5.3 Proof. (Theorem 5.3) For any set of distributions q= (qb. 91 92....) satisfying Definition 5.2, we have
(x()) 96 (Z (x
9:(z))
(z] = qo (x)
II
= (x) II (x(Ma)))
(x) II (.(()))))
(19)
9b M,)
(where we used (7) in the second equality). Since (po, Pl, P2........ satisfy Definition 5.2 by assumption, applying (19) gives
( po( II (x) == p(x),
so the composition at = 0 is projective, desired. Now to show that reverse-difffuion sampling with the compositional scores generates Clp), we need to show that
- N.CC(l
where p' = [p denotes the t-noisy version of distribution under the forward diffusion process. First, notice that if P satisfies Definition 5.2, then p² does well. This is because the diffusion process adds Gaussian noise independently to each coordinate, and thus preserves independence between sets of coordinates. Therefore by (19), we have ((p"((x) = pf (x) II pl (#el IM.. Now we apply the same argument (that diffusion preserves independent sets of coordinates) once again, to see that Clp²] N₂ [[]]]. desired.
I. Parameterization--nneeeedd
Compositions and Proof of Lemma 6.1
The proof of Lemma 6.1 relies on certain general fact about irrammttizationnindeeeeeee of certain operators, which we develop here.
Suppose have operator that takes input two probability distributions (p, over the same space X' and outputs distribution over X. That is, F A(X) x (xx) ((x). We can think of such operators performing some kind of "composition" of p.
Certain operators are independent of parameterization, meaning for any reparameterizzation the base space A X:X' - V, we have
F(p.g) A= Azg))
equivalently:
F(Adp, Ady) A5F(p,
where f is the pushforward:
1 (Adp)(z) A P(A-
25

---

Mechanisms of Projective Composition
This means that reparameterization commutes with the operator: it does matter if we first reparameterize, then compose, first compose, then reparamterize. A few examples:
1. The pointwise-geometrie median, F(p, (2) p(z))(()), is independent of reparameterization:
2. Squaring a distribution,
p(x)², is NOT independent of reparameterization::
3. The "CFG composition" (Ho Salimans, 2022), (a) == p(x)" is independent of reparameterization:
We can analogously define of distributions p = (ps, .., our composition operator C of Definition 5.1, Clp] x po(ar) II, of parameterization.
for operators on more than 2 distributions. Notably, given tuple
Px(a) is independent
pula)
Lemma I.1 (Paramerriation-nddeeeddddde of 1-homogeneous operators). If an operator F is I-homogeneous, ie. F(tp, tq....) = tF and operates pointwise, then is independence of parametrization.
Proof.
F(AEp, Ady. F(Atp(())))))) Al(()))
pointwise
= [AA
VA 9(A (=))
|AA (AA))))(AA''))))).))(
1-homogeneous
= AFFF Ap.(p. (z)
Corollary 1.2 (Paranttiizationninaaaaccce of composition). The composition operator C given by Definition 5.1 is indepen- dent of parametrization.
Proof. The composition operator given by Definition 5.1 is I-homogencous:
C(tpo, tp1, tp....... tp: (z) II (x)
Pt(x) tC(pb. P1, PI..P....)l6 ps(x)
(2)(x) II
tp:(z)
and so the result follows from Lemma I.l. Alternatively, direct proof is:
(x(
P1, C(pb.PI.P2..... :- -po(x) II
pr()))
Atpi = (Adp)) (z) II (z)
1 VA
V Pb (A II
(x(A- Pb(A (2))
(2)
Theorem 6.1 follows from Corollary I.2:
Proof. (Theorem 6.1) Let (gb- 4ı ....... qk) (Atps, Atp ... .tpp)) for which Definition 5.2 holds by assumption.
Applying an intermediate result from the proof of Theorem 5.3 gives:
g: (=)
Cl (=(=) II
(=1xxx)) x II (=(lM,))
(b(=)
By Corollary I.2, C is independent of parametrization, hence
Atp = At(C(p)) C(Aip) == C(¹)
26

---

Mechanisms of Projective Composition
APP:
Adp
Alpz
compose
11pp22
feature space
pixel-space
Pi
C
P
compose
P2
Figure 14: Projective Composition in Feature Space. commutative diagram illustrating Theorem 6.1: Performing composition pixel space is equivalent to encoding into some feature space (A), composing there, and decoding back to pixel space (A-1). This holds for all feature spaces subject to smoothness conditions. Thus, if there exists some feature achieve
space where distributions pı, P2 projectively compose (e.g. due to orthogonality as illustrated here), then this same composition by simply operating in pixel space, without even needing to know the feature space.
J. Composition under Orthogonal Transformation
Lemma J.1 (Orthogonal transform enables diffusion sampling)- If the assumptions of Lemma 6.1 hold for A(x) Ax, where A is orthogonal matrix, then running reverse diffusion sampler with scores s, V log C(p' generates the composed distribution Clp) satisfying (10).
Figure 15 shows synthetic experiment illustrating the sampling guarantees of Lemma J.1 in contrast to the lack-of- guarantees in the non-orthogonal case.
The proof relies on the fact that diffusion noising commutes with orthogonal transformations, i.e. A [q] Ne[[Adg] if A orthogonal, since standard Gaussians are invariant under orthogonal transformation.
Proof. By assumption, (Atps, Atpı, Atps) satisfy Definition 5.2, where A(=) Az with A orthonormal matrix. By Lemma 6.1, P Clpl satisfies (10). To show that reverse-diffusic sampling with scores 8; V. log Clp*) generates the composed distribution Clpj we need to show that composition commutes with the forward diffusion process, i.e.
Clp² N.(C
Theorem 5.3 immediately gives us
[[N][ NcC[[Ep]).
Now we have to be careful with commuting operators. know that composition independent of parametrization, ie. ArC|ø clAbp). Diffusion noising N, commutes with orthogonal transformation, i.e. AtN. Nr[AAg] if A is orthogonal, because standard Gaussian multiplied by orthonormal matrix Q remains standard Gaussian: " (0 T) Qn ~ AO, QQT) = (O, 7) (this is false for non-orthogonal transforms, however). Therefore, in the orthogonal case, we can rewrite:
AFC[N AHNx[C(P).
which implies the desired result since A is invertible.
K. Proof and further discussion of Lemma 6.3
K.1. Benefits of sampling at
Interestingly, (Du et al., 2023) have observed that sophisticated samplers like Hamiltonian Monte Carlo (HMC) requiring energy-based formulations often outperform standard diffusion sampling for compositional quality. Lemmas 6.1 and 6.3
27

---

Mechanisms of Projective Composition
P Poe Pil
Pi
Exact
DDIM
LD
coordinate- uligned
orthogonal
transform
non-
orthogonal
transform
Figure 15: Synthetic composition experiment illustrating the sampling guarantees of Lemma J.1 in contrast to the lack- of-guarantees in the non-orthogonal case. We compare coordinate-aligned case (which satisfies Definition 5.2 in the native space) (top), orthogonal-tramsee transfform case (middle) (which satisfies the assumptions of Lemma J.1), and orthogonal-transfomr case (bottom) (which satisfies the assumptions of Theorem 6.1 but not of Lemma J.1). In the first two cases the correct composition can be sampled using either diffusion (DDIM) or Langevin dynamics (LD) at t = O, while in the final case DDIM sampling is unsuccessful although LD at & = 0 still works. The distributions are 4-dimensional and we show 8 samples (rows) for each. We show samples from the individual conditional distributions Po- P1 using DDIM, samples from the desired exact composition Clps- Po- Pi] at 0 (obtained by sampling from A#C with DDIM and transforming by 1) samples from the composition C(ps, Po- Pil using DDIM with exact scores, and samples from the composition Clpn, po: pi] using Langevin dynamics (LD) with exact scores at time (amin 0.02). The noiseless distributions Po and P1 are each 4-dimensional 2-cluster Gaussian mixtures with means as case, noted in the figure, equal weights, and standard deviation T 0.02. For example, in the non non-ortinn orthogonalltransien Po has means [1, 0, 0, 0] and [0, 0, 1, Oj. and P1 has means [1, 0, 0] and 00,0, 1,,1, 1], (which can be transformed to satisfy Definition 5.2 via a non-orthogonal linear transform).
0 in the diffusion schedule
help explain why this may be the case. In particular, HMC (or any variant of Langevin dynamics) can enable sampling pd time f 0, even when the path p' used for annealing does not necessarily represent a valid forward diffusion process starting from p" (as Du al. (2023) note, C[p')] may not be). Lemma 6.1 should gives us hope that approximately-proine projeccive composition may often be possible space (which need not be explicitly constructed). However, that does not mean that we actually sample from this projection at time t = 0. As Lemma 6.3 shows, C[p²1] is not necessarily valid diffusion path unless A is orthogonal, so standard diffusion sampling may not work. This is consistent with Du et al. (2023)'s observation that non-diffusion samplers that allow sampling at t = may be necessary. Interestingly, Lemma 6.3 further cautions that sometimes C[P")] may not even be an effective annealing path for any kind of sampler (which consistent with our own experiments but not reported by other works, our knowledge.)
0, since it allows any invertible transform. A to transform into factorized feature
K.2. Proof of Lemma 6.3
We will prove Lemma 6.3 using a counterexample, which is inspired by an experiment, shown in Figure 17 (left), where non oothortho conditions fail compose projectively.
The basic idea for the counterexample that given distribution p(x:) with two conditions, 0, such / 0,
1 - Se, (x) + (z). Pi (x
1
Po(x)=
Sueoter (I) 8 Suertes (z),
2
for some a S 1, so the conditional distributions do not satisfy the independence assumption of Definition 5.2, However,
28

---

Mechanisms of Projective Composition
Orthogonal conditioning
Nonorthogonal conditioning
Xi
red (3.0)
grean 00.105
bue 18AA
red (.A
yellow 0.100
p_(x)
PorPPo
22
%
15.20
0121
15.00
0.31
15.23
0.31
0121
15
15.05
0.31
Elp-PuPPll
Figure 16: (Left) A visualization of the intuition behind the proof Lemma 6.3, under a 2D projection. (Right) An experiment where the colors red, green, and blue all compose projectively, while the colors red and yellow do not. We trained Unet on images each containing a single square one of locations (selected randomly) and a certain color, conditioned on the color. We then generate composed distributions by running DDIM on the composed scores. The desired result of composing red and blue is image containing red and blue square, both with randomly-chose (so we occasionally get a purple square when the locations overlap). When we try to compose red and yellow, we only only ever obtain a single yellow qquare.Note that in pixel space, the colors are represented as red (1, 0, 0), green (0, 1, O), blue (0, 0, 1), yellow (1, 1), so that red, green and blue are all orthogonal and are expected to work by Lemma 5.3, while red and yellow are not orthogonal, and fail as allowed by Lemma 6.3. In fact this experiment closely related to the counterexample used to prove Lemma 6.3.
locations
p Po- P1]
81 Po- Po-P1l
Po
P1
Exact
DDIM
Po
Pi
Exact
DDIM
d=1
a
033
P 1-
Figure 17: Composition experiments for the setting in the proof of Lemma 6.3. Left pane shows 8 samples (rows) of each distribution in the native 4d representation; right pane shows 1000 samples under the 2D projection used in Figure 16. We show samples from the individual conditional distributions Po. Pi using DDIM, samples from the desired exact composition Clpb, Po: PI) at / 0 (obtained by sampling from AtC|p] with DDIM and transforming by A A-1), and samples from the composition C(ps. Po. Pı] using DDIM with exact scores. We take - 0.02, and set a = 0.02 in the diffusion schedule. In the top row we take a = 1 ("very non-orthogonall) as in the proof, and compare this to a = 0.3 ("mildly non-orthogonal") the bottom row. With a = 1, as in the proof we see that DDIM barely samples two of the clusters. With a = 0.3, DDIM still slightly undersamples the "hard" clusters but the effect much less pronounced.
29

---

Mechanisms of Projective Composition
there exists (linear, but not orthogonal) A such that the distribution of Axr is axis-aligned
(A)
5 e0 (x) 2
2 (x))
(A) 5 (2)) 2 ses (x))
and thus does satisfy Definition 5.2 at = O, which guarantees correct composition of p at f. = under Lemma 6.1. The correct composition should sample uniformly from {1++a)e++e) ep e0+
[[+a)eztea) J. What goes wrong is that as soon as we add Gaussian noise the distribution p(z) at time t of the diffusion forward process, Ax breaks and so we no longer guaranteed correct composition of (x)). In fact, the distribution still GMM but places nearly all its weight only two of the four clusters, namely: {(1 + a)cy + e1, (1 + ajez + es}. Intuitively, let us focus on the mode aeo + of and consider how it interacts with the two modes eo, e2 of po. at some time t > 0 when we have isotropic Gaussians centered at each mode. Since acg + ej is further away from e2 (distance va + 2) than it is from ég (distance va² 2a + 2). it is much less likely under ((2,, oe) than [eo,, oz), leading to a lower weight. This intuition is shown graphically in a 2D projection in Figure 16 (left).
020+te+ee)
the relationship
For the detailed proof, we actually want to ensure that p has full support even at so we add little bit noise to it, but choose the covariance such that Ar still holds f 0.
We begin by defining the distributions we will use for the counterexample.
Definition K.1. For any choice of T > define the following counterexample distributions:
1 Mxx;co, (AAAA) 1 e2,
pπ(x)) 2
1
pi()) 2
Vx; t A)
(A")) + N(x; a22 c3, (AT A)-
2
1
0
=)(=)= ((=0, where A =
0
0
0
0
0
1
so that in the transformed space
(A)p)(=) = 2)) == Ax
(=) -0(=;;,(;
(=) 2 60, (2),,,++++++0+1 / (2;e,,,))
(=) N(ziei, 73) 5 (2;e,,,)7))
The noised versions at time t >> are
(x)())))) (1 To, of)
pb(x)= N(rieo, (
p = 2 N(zz; + 0 (4 -)-") マ (2z + es. 0f + ?2(AT
Next, we state some intermediate results we will need for the proof.
Proposition K.2. A composition of two Gaussians with identical covariance (using Gaussian background with zero mean and the same covariance) is a scaled Gaussian with the following parameters
N(x; p1; (x; CN(xi + E), where C' = =xxxp(ux>" (22). N N(x; 0x
30

---

Mechanisms of Projective Composition
Proof.
M1 (1; N1x; Σ)
M2, _) =2x) - Σ
e
2 (0-11))
"0''0-"'--0 x
c -/2= --1x
1 (a m1) 21 M1) 1 -(1 3(2) 22)²2' (x
u2) + 1 2I=2 2
(2T)
|
exp
1 - Mn
1 Zuzz 42
(2T)
|
exp
2" I x + x - (M1 + 12)
1 2 (2 pn p2) ' Σ (I 1z 12)
C(2T)
Σ exp
CN;(1; + 12 2
C cxp
2 1
1
Σ k1
u/ Σ
u2 + 2 (uu x= + p22)
1
Proposition K.3. With (p6, ( P2) from Definition K.I, defining (x) = Clps, Clpss.b.pfi Pi). we have that ge (x)), 16 (x)) and (x) are all Gaussian mixtures (GMs) with identical means:
7= (11 + a)eo + er, en + a62 + e3. aeo t22 (1+ a)+ + eah,
and the following weights and covariances:
(x) weights: w : [1, 1, 1, 1,1], 1},
covariance: En :- (A A)-
pl (x): weights: w² = 1 8, E, 2 - E],
ao? 2 (a2 + 2) + o 71
6: - 5(-E), :
where 5(2) e
(logistic function),
covariance: E, :- : 07 7 A)
[x)
weights: w : [1, 1, 1, 1],
covariance: Σ, := 0/1 + (A A)-1
Proof. We apply Proposition K.2 to the distributions of Definition K. to analyze (x), (x(),, and (r) Proposi- tion K.2 gives that all three distributions are Gaussian mixtures with identical means F, and variances Eg. Σ,, and Σ,, respectively. The weights for (z) and (x) are uniform (w = 1[1, 1, 1, 1)). We just need to calculate the weights for (x)
31

---

Mechanisms of Projective Composition
First we compute the covariance and inverse covariance:
1
a
0
0
0
0
0
+
a
0
0
a
of + +2
-aT?
0
Σ -1
-07
(a²
1)r
o
0
(a2 r + o 7
0
0
of +
0
0
-0r
(a + 1)7
+
to,
After algebra (namely, computing C exp(uf E, U22 for each cluster), we find that
ao?
p((x): w (exp(E), 1, exp(E)),
{:= (a² + 2)0 + of + T
2 6, - -s],
e: 2
S(-E), where S(z) c +1
(logistic function).
The intuition for the proof of Lemma 6.3 will be that when is small, clusters (1,2) have much lower weight than clusters (0.3) in the GM. In that case, can lower-bound the W2 distance by noting that since w has almost mass the two of the clusters, we will need to move a little less than 1/4 probability over to those clusters. For example we need to move 1/4 probability onto cluster eg + ae2 + éz from either (1 + a)en + é1 (1.2 distance between means is /2a + 2) or (1 + a)e2 + ea (L2 distance 2). So overall we will have to move a bit less that 1/2 probability at least v2 distance.
We restate the following results from Delon & Desolneux (2020), which we will need help bound the W2 distance. Theorem K.4. (Mixture Wasserstein distance (Delon & Desolneux, 2020))
/ [110 - y1 d1 v1);
MW2(10,4g 91) =
inf (GGM)00)))))MMM.((0))
W10: cell(wo..r))
min
Σ
Ck W2 (40
(4o, d/) (Delon Prop. 4),
k.1
W2(go, q1) & MW2(go. q:) < W2(90 91)
Σ
Σ
u Tr(
(Delon Prop. 6),
1=0,1
where I1(go, q1) denotes the set of all joint distributions with marginals go and qı, and GMMa(oo) UK20GMMa(() denotes the set of all finite GMMs.
We will also need one more standard fact about the W2 distance between Gaussians.
Proposition K.5. (W2 distance between Gaussians: standard)
W (N Nx), Eg)) 1| - My 12 +r(E,, E,, 2(E)Σ,E)) 2 M -myl2
Proposition K.6. p is a Gaussian mixture distribution of the form
K
p(x) =
WiN(HE Ck), x DER"
then p is (nK)² -ipsschitz wat Wasserstein 2-distance:
Wa(p" p¹) S (nK) orl,
(that is, -iipshii,, where O only hides constants depending on ambient dimension and number-offclusserrs)
32

---

Mechanisms of Projective Composition
Proof.
K p(r) ==
WrNWHs Ck)
k=
(x())= Σ
w:NN (Jus, Ck + 01
k=1
W2(N(ux: Ex), ( En]) = M - Myll2 + Tr(Ex + My 2( E و )
= 11 - /2112 + || - ->31 if Egg Ey commute
(2' (k).
II(C + of (C++ +
II(A + - (A o
where C2 UAU is eigendecompositiin
< |1(0; - (( MI
(by concavity of square root and A = 0)
= -n(o;-ov) - 02)²
(2'0" .p') - (F .p')
Σ Ce, W2 (p' (M)))"(1)
min
Σ W2'0" [k).)'())) (since II(w, w))
S nK(o, x²) Wa(p", v²) s (nK)² lo, 0rl-
Thus p is w.r.t W2 distance.
Proposition K.7. With (pb. (p..P1.p2) P2) from Definition K.I and (1())= Clyy, pb: pi). the Wz-distance between ii and go bounded as follows
(1 4E) -0(74444+++ 2a²) + 20²) S W2("", +b) s ((1 (1-4e)(11 + 2a²) + +2o?))
Proof. Using Proposition K.5,
MW3 .²)
min cEIl( unf)
Σ
CK CK (F"
kdl
min cEll(w".," w(
Σ
Ck.l (IIu - 11l1 + Tr - -
kd
(1 = 1/4 -
2 < |11(0 3) 1111 or 11 2 +
0 < Tr(S0 SS, - -2 =
20²) + 40²
< MW2 (N.((" (( ())
+ 40,
(1 4 < MW N. 10 p)) < (1144)1++ 2n + 40
Above, we noted that any c E I[ w') has to move at least 1 probability each away from indices or 2 and onto indices either 0 or 3, and for any of these moves we can bound the squared L.2 distance the mass must move between 2 and 2(1 + a). We also used simple bounds on the trace term
33

---

Mechanisms of Projective Composition
We Theorem K.4 bound the W2 distance by the MW₂ distance: Using Theorem K.4 (Delon & Desolneux, 2020).
W2(0", . E MW2(1", .1')
- ((1- (1) 2a + +0,)
W2(0" 2 MW2(0 . - 2 Σ (A)TT))r(>>)) ( + w w (+)Tr(E)))
2 - -((r(2)) + Tr(E'))
2 (1 42) 2 + + 2a²)).
Proposition K.8. With (pb, P1, P2) from Definition K.I and (x) = Cipl, ph: pil. the Wz-distance between pe and N₂ [p") is bounded as follows
(1- 4(( 40,) S Wa( S (1- 43) (1)
Proof. Using Proposition K.5,
(N
min eETT(ur ann))
Cr W2 (N.
k,l
min eETT(W 2n
Ch, .m. ur
k.1
3) = 1/4-E -E
2 S || 3) u(1 or 2
1 4: s MW3 (Ne s (1- 4:)(1 +a)
(1 45) < MW2 S (1 (1
Using Theorem K.4 (Delon & Desolneux, 2020),
W2(N.1P")1 1') < (W
5 (1) + a)
2 2) (NW - > (w0 + [w= (()))(((('))
> (N - -4Tr(E')
2 (1 46) 4(403 + +
2a²))
Now we have all the pieces prove Lemma 6.3.
Proof. (Lemma 6.3)
We will show that the distributions (pb: p1, P2) of Definition K.1 satisfy Lemma 6.3. We make the choices a and oe = L for simplicity, and define &e (xx := Clgt pb: pi].
Lemma 6.1 applied to the distributions of Definition K.1 implies that at time t = 0,
f²(x) = Clrl C( p(())p(())
-8(01022)))((((13)) (f(x))
34

---

Mechanisms of Projective Composition
That is, we have a projective composition at time / 0. For Part 1 of Lemma 6.3, note that the ps are Gaussian mixtures, therefore Proposition K.6 gives that each ps is (1)-Liipshhiz w.r.t Wasserstein 2-distance:
vi: We(p', S v88t - t'
For Part 2, we need bound the Lipschitz constant of p' := C.'.
Proposition K.7 gives:
(1 - 42)+ -47 2a²) + 20 - W2(p", 2T) - ((1 (1) + a) + + 2a²) + 20?)) $
where & = 2 5( {){ ::
with S(z) == c +1
2
(a² + 2)0 72 0 7
Plugging in = 1,
(1 (1-433 41) - 247² 8 S W2(F", ,1) S (2(1 - 4s) + 1272 + +00))
of
where = = 2 5( S(-E), : 30/-2 + o 7
We will to show that After some algebra, we find that for any fixed t, the minimum and maximum of E are
3,,'
27 -- 1' "2(d',g") 2T 2 Elp.
1
ot - = - = - (max)
4
1
O = (-
= E ((-)(()) (min)
57"
2
a + 0 = (T) + x0
+
E + 0 (min)
Thus, taking oz (the minimizer of 2) and choosing any 7²
(somewhat arbitrarily, but small enough), gives
66
o = = (1 - --)= - 327 S W2(5", .p) - (2(1 - (e) + 3272)
1 S 57
where E =
1 122 - (1 4c) 21-22
72
= 10-6
66
0.3 - 337² < W2(1", (p) < (2 + 3272) < 2.
Thus, for any choice of 72
1. if we take o; := and = 0, we have as desired that 66
0.5r 0.5 < Wz (p 2D) < 27 2--1|1 1|1,.
that is,
3t, W2(5" "0) - = -(T -
We also prove another lemma using the same counterexample Lemma 6.3:
Lemma K.9. Let qf denote the composed distribution time t: g¹ := C(p*), and N, be the Gaussian-noising operator.
There exist distributions {ph, Pt..Pi..... p&} over DD and value of i such that Nelg") (the ideal diffusion path to qº) differs from q' (the path actually followed) by at least 9(1):
3t (2 2
35

---

Mechanisms of Projective Composition
Proof. We will show that the distributions of Definition K.1 satisfy Lemma K.9. We make the choices a 1 and or == L for simplicity.
By Proposition K.8,
(1 4z) - 4 - + S (N( 1.1') S (1 (1) (1+ a)
where e 1
aoz
2"² -5(
: (a² 2) 2 o 7
with S(z) e +1
Taking a = 1:
(1- (1- 24T² 1607 - W2(NN, (1)))()))))) (2(1 -=)
of
where := 2 {)-E) :: 3077² of +
For any fixed r, the minimum and maximum of e are
1
9 + = 5 = E
4 (max)
1
1 (-) (min)
= r + (-
572
E
2
0 - = (T) + 00
= E - 0 (min)
First we want to show that
1 3E: (2 2
Taking o; 7 (the minimizer of E) and choosing any 72 (somewhat arbitrarily, but small enough), gives
1 1
ot = = (1 11-48 46)² 407² S W2( (1')) S v2 -6) where = S(
2 5
1 = (1 45) 21
72
E 10- <<
82
4
0.5 - 417 < W2 1²) S 22.
Thus, fixing 72 1 and taking d; = T, we have desired that
82
W2(N.IP'), 1'p') 2 0.5.
The proof is now complete, but can make one more interesting observation. The bound above was obtained by choosing small value of r, but the diffusion path (specifically, for distributions of the form of Definition K.1) is much less problematic for larger T:
Vt: (2 = 7)
That is, even for our counterexample distributions, diffusion can still approximately work to sample from the composition po, if r is large enough. To see this, we note that
1L: W2(N.1F"))') < v2(1 1 48)
1 where = 8( 8(-E), ::= ::= 30? + o + 7
of
2
2 1
(-{(tt)) {(():= = 5
1
for any fixed T
V2
+ W2(N.(F9),') S v2(1 - 28 (-E <
VT.
36


Reviews:
[{"review_id": "XMWiU4r9FE", "reviewer": "Reviewer_cp6a", "summary": "This paper proposes a new theoretical framework for analyzing a special type of composition in diffusion models, and it specifically focuses on two previously discovered phenomena in diffusion model composition: out-of-distribution (OOD) extrapolation and length-generalization. The theoretical framework aims at the product-style compositions implemented with diffusion models via a linear combination of scores. Prior studies propose to describe the composed distribution as a simple product of two distributions, or the Bayes composition of them. Yet, the paper uses the CLEVR experiment as an intuitive illustration to show that these two definitions can not really cover OOD composition results and thus will fail to do length-generalization in the CLEVR experiments. Based on this, the paper defines a new form of distribution composition: **Projective Composition**. Intuitively, it requires the composed distribution to be the \"same\" as each single distribution when viewed from a projection defined for each single distribution. This **Projective Composition** can describe real OOD and length-generalization. The paper further defines a **Composition Operator** to compose a set of distributions, and a **Factorized-Conditionals** that defines specific features of a set of distributions and projections, such as the projections are disjoint masking of the coordinates. The paper further shows that when **Factorized-Conditionals** is satisfied, the reverse-diffusion SDE using compositional scores following **Composition Operator** will satisfy the desired **Projective Composition**. The paper then argues how the successful OOD settings in the CLEVR experiment approximately satisfy **Factorized-Conditionals**. Moreover, the paper discusses how similar analysis can be extended to feature space. In the feature space, they show under the constraint of **Factorized-Conditionals**, the **Composition Operator** also defines a **Projective Composition**; however, how to generate such Projective Composition via diffusion sampling is unknown within the theoretical framework. At last, the paper discusses how the proposed theoretical framework can help understand other empirical understandings during diffusion model composition.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support for our work and insightful questions, to which we respond individually below.\n\nWeaknesses:\n\nQ1: The main theoretical results only cover sampling in the pixel space, and a theoretically successful result is lacking in the feature space. Yet, feature space composition is an important application in diffusion decomposition.\n* This is certainly true. In fact, we show theoretically that in feature space, diffusion sampling may not work even when projective composition is possible at t=0 (Lemma 6.3). This is consistent with empirical findings in Du et al. (2023), who showed that HMC sampling (which in particular allows sampling directly at t=0) was necessarily to enable successful composition in many cases. Our theory helps to explain why HMC sampling worked for Du et al. when standard diffusion sampling did not. We discuss this further in Appendix J.1. This result contributes to our overall goal of understanding when composition will work — and just as importantly, when it may fail.\n\nQ2: [It is unclear how practical the definition of Factorized-Conditionals will be in real world composition.]\n* Please see response to Q4.\n\n\nQuestions:\n\nQ1: Can authors explain the intuitive meaning of Lemma 6.3 and discuss some remarks on it?\n* Lemma 6.3 intuitively says that, even if projective composition is possible at $t=0$, reverse diffusion (or indeed any annealing method) may not be able to correctly sample from it. Specifically, the lemma proves (using a counterexample) that it is possible for a set of distributions to all vary smoothly in time, while their composition changes extremely abruptly— making any annealing-based sampling method very challenging.\n\nQ2: In appendix B, training EDM2 on CLEVR, what is the conditional variable? Is the generation trained to be conditioned on the number of objects, shape of objects, color of objects, position of objects, or a mixture of them? \n * Appendix B includes two different conditioning setups. In Figures 7, 8, and 9 we condition on the 2d location of the object (or the location of one randomly-chosen object, for multi-object distributions). In Figure 10, we condition on the color of the object. In all experiments we condition only a single attribute (either location or color) at a time, with all other attributes sampled randomly and not conditioned on. Thanks for the question — we will clarify these points in the final draft!\n\nQ3: In appendix B.2, after training the diffusion model, how to use the trained diffusion model to get/define a generation model following certain $p_i$? \n* For the location-conditional models, the $p_i$‘s correspond to different location conditioners. Specifically, in these experiments, we choose a fixed set of locations $i$ that we wish to compose, and obtain the score of $p_i$ by forwarding our conditional diffusion model conditioned on location $i$.\n* Similarly, for the color-conditional models, the $p_i$‘s correspond to different color conditioners. There are only 8 colors so we assign a $p_i$ to every possible color, and we obtain the score of $p_i$ by forwarding our conditional diffusion model conditioned on color $i$.\n\nQ4: How practical will the definition of Factorized-Conditionals be in other real-world diffusion composition, and how far it can be generalized to other successful compositions, such as composing different text prompts with/without different region masks to generate an image?\n* This question is closely related to the existing literature on “disentangled feature representations.” For example, “style” and “content” features are typically believed to be disentangled in the existing literature, and thus we expect style & content to form Factorized Conditionals. Regarding composing different text-prompts, we expect similar intuitions about disentanglement to carry over --- see our Figure 6 for example, which composes using different text-prompts.\n* Regarding region masks, we believe that either explicit masks or simply text-conditioning that includes location information can indeed be very helpful for achieving Factorized Conditionals (please see Section 7 for further detail).\n\nWe thank the reviewer again, and hope these responses are helpful.", "claims_and_evidence": "The claims are clear and convincing.", "methods_and_evaluation_criteria": "The evaluation criteria make sense.", "theoretical_claims": "I have checked proofs of most of the theorems except for Lemma 6.3. I haven't fully got the intuitive meaning of Lemma 6.3 and would appreciate remarks from the authors.", "experimental_designs_or_analyses": "I have checked appendix B for the experiment details. \n1. There are questions related to the conditional variable of the trained EDM2 on CLEVR: what is the conditional variable - is the generation trained to be conditioned on the number of objects, shape of objects, color of objects, position of objects, or a mixture of them? This question arises since the paper seems to require distributions that can control position, or color, or number of projects.\n2. After training the diffusion model, in B.2, how to use the trained diffusion model to get/define a generation model following certain $p_i$?", "relation_to_broader_scientific_literature": "Maybe the paper can inspire future works to design a new sampling method for feature decomposition, or propose a theoretical framework to understand feature space decomposition.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow, with motivation, theorems, remarks, proof scratches, and empirical demonstrations.\n2. The paper studies a novel theoretical problem of understanding OOD and length-generalization phenomenon in diffusion model composition. The interesting projective composition is analyzed in both pixel space and feature space.\n\nWeakness:\n1. The main theoretical results only cover sampling in the pixel space, and a theoretically successful result is lacking in the feature space. Yet, feature space composition is an important application in diffusion decomposition.\n2. Although the paper discusses in B.2 how the CLEVR settings approximately satisfy Factorized-Conditionals, it is unknown how practical the definition of Factorized-Conditionals will be in other real world diffusion composition, and how far it can be generalized to other successful compositions, such as composing different text prompts with/without different region masks to generate an image.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Q1: Can authors explain the intuitive meaning of Lemma 6.3 and discuss some remarks on it?\n\nQ2: In appendix B, training EDM2 on CLEVR, what is the conditional variable? Is the generation trained to be conditioned on the number of objects, shape of objects, color of objects, position of objects, or a mixture of them? \n\nQ3: In appendix B.2, after training the diffusion model, how to use the trained diffusion model to get/define a generation model following certain $p_i$? \n\nQ4: How practical will the definition of Factorized-Conditionals be in other real-world diffusion composition, and how far it can be generalized to other successful compositions, such as composing different text prompts with/without different region masks to generate an image?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4indyGL9AM", "reviewer": "Reviewer_nnqo", "summary": "This paper gives a rigorous theoretical framework for understanding composition in diffusion models, with a focus on out‐of‐distribution extrapolation and length‐generalization. The authors introduce the notion of “projective composition,” which formalizes the idea that a composed distribution should, when viewed through specified projection functions, match the marginals of the component distributions. They derive conditions—most notably via the Factorized Conditionals assumption—under which linear score combination (and its feature-space analogue) yields a correct composition. This paper supports its theoretical results with experiments on synthetic CLEVR data, demonstrating instances of length-generalization and discussing practical sampling challenges.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support for our work and insightful questions, to which we respond individually below.\n\nWeaknesses\n\nQ1: The Factorized Conditional assumption, critical for the theoretical guarantees, may be too strong and not fully reflective of practical scenarios.\n\n* Theorem 6.1 shows that it is enough to satisfy Factorized Conditionals in *some* feature-space, even if the assumption is not satisfied in pixel-space: that is, as long as there exists some feature-map which “disentangles” features in the appropriate sense, then distributions will compose correctly. We further discuss the empirical evidence for such disentangled features spaces, as well as the robustness of the theory to approximate satisfaction of the conditions, in our answers to Q1 and Q2 below.\n\nQ2: Experimental validation is limited to synthetic datasets, leaving open questions about applicability in more complex, real-world settings.\n\n* Our goal is primarily to theoretically explain existing empirical evaluations in the literature. In text-to-image settings, the Bayes composition is often approximately projective, as discussed in Section 5.4. We therefore accept the existing experimental results of Du et al (2023) and others using the Bayes composition, and seek to understand/explain them (both successes and failures) through our theory. Of course, there is much more to study and explore empirically in text-to-image settings that we hope to explore in future work.\n\nQ3: Some proofs and technical derivations are only sketched, which could hinder reproducibility and complete understanding.\n\n* We provide complete proofs of all claims in the Appendix. In particular, Theorem 5.3 is sketched in the main text but proved formally in Appendix G. Theorem 6.1 and Lemma 6.2, 7.1, and 7.2 are proved in Appendices H, I, and J.\n\nQuestions\n\nQ1: Can you provide additional empirical evidence on real-world datasets to assess whether the Factorized Conditional assumption holds approximately in practice?\n\n* This question is closely related to the existing literature on “disentangled feature representations.” This long line of work (e.g. [1] on VAEs and [2] on GANs) implicitly assumes that such a factorized representation is (at least approximately) possible, and there is substantial empirical evidence supporting this for at least some concepts. For example, “style” and “content” features are typically believed to be disentangled in the existing literature, and thus we expect style & content to form Factorized Conditionals. We mention this connection in Section 7, but will elaborate on it in the revision.\n\nQ2: How robust are your theoretical results if the independence assumptions are only approximately satisfied? Could the framework be extended to account for partial dependencies?\n\n* Currently, the theory requires that the independence assumptions be satisfied exactly, but developing robust versions is an important direction for future work.\n* Empirically, we use the CLEVR experiments to probe the robustness of the theory. In the CLEVR setting, Factorized Conditionals holds only approximately, due to the possible occlusions and shadowing effects between different objects. Our experiments show that that projective composition is approximately, but not exactly, achieved. To push the limits of this robustness, in Figure 9 we attempt to length-generalize up to 9 objects (which works up to about ~6 objects and then degrades).\n\nQ3: Could you elaborate on potential strategies to mitigate the sampling challenges (as noted in Theorem 6.1 and Lemma 6.3) in practical implementations of your composition operator?\n\n* Yes! As you note, Lemma 6.3 tells us that even if projective composition is possible at t=0, reverse diffusion may not correctly sample from it. Practically, this suggests that non-diffusion sampling methods that enable sampling directly at t=0, such as variants of Langevin dynamics, may be necessary to achieve projective composition in practice (when it is possible at t=0). This is consistent with empirical findings in Du et al. (2023), who showed that HMC sampling was necessary to perform composition in many cases. We discuss this further in Appendix J.1. \n\nQ4: Are there any plans to integrate or test your framework with more complex, high-dimensional real-world image datasets to further validate its practical impact?\n\n* We agree this is an important area for future work; we consider the present work as the first step in this direction.\n\n\nReferences:\n\n[1]  Isolating Sources of Disentanglement in VAEs\nRTQ Chen, X Li, RB Grosse, DK Duvenaud. NeurIPS 2018.\n\n[2] A style-based generator architecture for generative adversarial networks\nT Karras, S Laine, T Aila. CVPR 2019.", "claims_and_evidence": "The main claims are that: first, prior definitions (simple product and Bayes composition) fail to capture the desired out-of-distribution behavior; second, projective composition, as defined via appropriate projection functions, can correctly compose diffusion models, and, third, under Factorized Conditional assumptions, the proposed composition operator yields a distribution with the intended marginals. These claims are supported by theoretical results (e.g., Theorem 5.3 and Theorem 6.1) and illustrated through synthetic experiments. Nevertheless, while the derivations are insightful, some proofs are only sketched and the reliance on strong assumptions (e.g., perfect factorization) may limit the generality of the evidence.", "methods_and_evaluation_criteria": "The paper combines theoretical analysis with experiments on a controlled synthetic dataset (CLEVR). The methods contain defining novel composition operators and establishing conditions for their correctness via rigorous proofs. The evaluation criteria are appropriate for a theory-focused work, though the empirical validation remains limited to synthetic settings. A broader set of experiments on more complex, real-world data would assist to corroborate the practical relevance of the theoretical findings.", "theoretical_claims": "The paper showss several non-trivial theoretical claims regarding the behavior of composition operators in diffusion models. The formal definition of projective composition (Definition 4.1) and subsequent results (e.g., Theorem 5.3 on the correctness of composition under Factorized Conditionals, and Theorem 6.1 in feature space) are substantial contributions. Nevertheless, the proofs are sometimes only outlined, and some underlying assumptions (such as exact independence across masked coordinates) might not hold in practice.", "experimental_designs_or_analyses": "Experiments on the CLEVR dataset illustrate key phenomena such as length-generalization and the sensitivity of composition to background choice. Although these experiments effectively demonstrate the theory in a controlled environment, the experimental section is relatively narrow in scope. Extending the experiments to more realistic datasets could help strengthen the overall impact.", "relation_to_broader_scientific_literature": "The work is well-situated within the literature on diffusion models, compositional generation, and generative modeling in general. It builds upon and extends prior methods such as those by Du et al. (2023) and Liu et al. (2022), offering a novel perspective by formally addressing the limitations of existing composition definitions. The paper, in addition, relates to literature on disentangled representations, which underpins its Factorized Conditional assumption.", "essential_references_not_discussed": "Despite the fact that the paper cites a wide range of related works, a deeper discussion of literature on disentangled feature learning and alternative composition strategies (especially in the context of real-world image synthesis) could further contextualize the contributions.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Introduces a novel and formal definition of composition (projective composition) that addresses clear limitations in prior work.\n2. Provides a set of theoretical results that illuminate when and why linear score combination can yield correct composition in diffusion models.\n3. Connects theoretical insights with empirical observations on synthetic data, offering useful perspectives on sampling challenges.\n\n\nWeaknesses:\n\n1. The Factorized Conditional assumption, critical for the theoretical guarantees, may be too strong and not fully reflective of practical scenarios.\n2. Experimental validation is limited to synthetic datasets, leaving open questions about applicability in more complex, real-world settings.\n3. Some proofs and technical derivations are only sketched, which could hinder reproducibility and complete understanding.", "other_comments_or_suggestions": "The paper could benefit from clearer exposition in some of the more technical sections, as well as from an expanded experimental section that explores the framework’s applicability beyond synthetic examples. Detailed discussion on potential methods to address the identified sampling challenges could also strengthen the work.", "questions_for_authors": "1. Can you provide additional empirical evidence on real-world datasets to assess whether the Factorized Conditional assumption holds approximately in practice?\n2. How robust are your theoretical results if the independence assumptions are only approximately satisfied? Could the framework be extended to account for partial dependencies?\n3. Could you elaborate on potential strategies to mitigate the sampling challenges (as noted in Theorem 6.1 and Lemma 6.3) in practical implementations of your composition operator?\n4. Are there any plans to integrate or test your framework with more complex, high-dimensional real-world image datasets to further validate its practical impact?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eShtUifGQd", "reviewer": "Reviewer_grsj", "summary": "The authors present a formalization of compositionality in diffusion models. Using diffusion models separate for particular objects and background can be joined together in various ways. The authors explore these different ways and point out the correct way of composing these. The authors suggest a particular way of composition if a collection of distributions satisfies a set of conditional independencies, and then continue to generalize that by using diffeomorphisms and showing if such a diffeomorphism exists, their composition stil holds. The authors show various illustrations in the form of the CLEVR dataset and an example from a text-conditional diffusion model.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time, insightful questions, and constructive critiques.\n\nOverall, we want to emphasize that the goal of this paper is to understand and predict when composition will work — and just as importantly, when it will fail. That is, we want to theoretically explain prior empirical observations about when composition worked or failed; we do not aim to introduce any new methods.\n\nWe paraphrase and respond to your specific questions below. \n\nQ1: In real-world settings, when we can expect the existence of a diffeomorphism that separates the conditions we want to compose?\n\n* First on a technical note, we actually only require “$C^1$ diffeomorphisms”, i.e. the feature-map and its inverse should be differentiable. We will clarify this in the revision. \n* Our “diffeomorphic” assumption is very closely related to existing assumptions in the literature on “disentangled feature representations.” For example, the long line of work on learning disentangled representations implicitly assumes that such a disentanglement is (at least approximately) possible. (e.g. [1] on VAEs and [2] on GANs). That is, if we are in a setting where we have a neural network that maps to and from a “disentangled” feature space (e.g. a VAE or a BiGAN), then this neural network defines our requisite $C^1$ diffeomorphism (technically this assumes the encoding and decoding networks are differentiable everywhere, which we can guarantee e.g. if if the network uses smooth activation functions).  \n* Finally, we do not believe that disentangled representations always exist for any distributions we might wish to compose--- and we are equally interested in understanding these failure cases. For example, “style” and “content” features are typically believed to be disentangled in the existing literature, and thus we expect style+content compositions to work. On the other hand, some concepts may be impossible to disentangle in any reasonable features space. In such cases, a diffeomorphism may not exist, and we do not expect these concepts to compose well (an example is the horse+dog composition in Figure 6).\n\nQ2: How robust is the theory? Is it really necessary to perfectly satisfy Factorized Conditionals?\n\n* Although our theory technically requires perfect independence, which is indeed a strong condition, our CLEVR experiments empirically study a case where the conditions hold only approximately, and explore both the robustness of the theory as well as its limits in this imperfect case  (please see response to reviewer nnqo for further detail).\n\nQ3: What are the practical implications/usefulness of the result in Lemma 6.3 which says that, even if projective composition is possible at $t=0$, reverse diffusion may not correctly sample from it?\n\n* The fact that reverse-diffusion sampling may not work even when composition is possible at $t=0$ explains the “negative-result” in Figure 5, and may help explain other failures of composition in the literature.\n* Most notably, this result helps explain empirical findings in Du et al. (2023), who showed that HMC sampling (which in particular allows sampling directly at $t=0$) was necessarily to enable successful composition in many cases. Our theory helps explain why HMC sampling worked when standard diffusion sampling did not. We discuss this further in Appendix J.1.\n\nQ4: Text-to-image evaluations?\n\n* Our goal is primarily to theoretically explain existing empirical evaluations in the literature. In text-to-image settings, the Bayes composition (used in Du et al. (2023) and other works) is often approximately projective, as discussed in Section 5.4. Therefore, existing empirical results in text-to-image settings are typically already constructed in a way that is compatible with our theory. We therefore accept the existing experimental results of Du et al (2023) and others and seek to understand/explain them (both successes and failures) through our theory. Of course, there is much more to study and explore empirically in text-to-image settings that we hope to explore in future work. \n\nQ5: Quantitative evaluation?\n\n* We performed some additional quantitative evaluations of our CLEVR experiments: please see the table of results and description of the experiments in our response to Reviewer DBEj. \n\n\nIf the reviewer’s concerns have been adequately addressed, we kindly ask they consider raising their score to support acceptance.\n\n\nReferences:\n\n[1]  Isolating Sources of Disentanglement in VAEs\nRTQ Chen, X Li, RB Grosse, DK Duvenaud. NeurIPS 2018.\n\n[2] A style-based generator architecture for generative adversarial networks\nT Karras, S Laine, T Aila. CVPR 2019.", "claims_and_evidence": ".", "methods_and_evaluation_criteria": "There are no quantitative results in the paper.", "theoretical_claims": "I can not find any particular issues with the theorems and proofs in the work, but I find the requirement of the existence of a diffeomrophism in 6.2 strong enough that I question the usefulness of the result.", "experimental_designs_or_analyses": "See also strenghts and weaknesses:\nThe authors mostly focus on the CLEVR dataset for examples and all experimental examples are qualitative.", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "- The idea of formalizing compositionality of diffusion models is relevant and an interesting topic\n- The authors mostly focus on the CLEVR dataset for examples and all experimental examples are qualitative. While the field is perhaps not developed to a point where there is a set benchmark, the work of Du et al (2023) serves as major inspiration for the authors, and it would certainly make sense to run the same experiments reported in that paper for a quantitative comparison.\n- While theorem 5.3 is interesting, it is of limited use other than for datasets such as clevr, where objects are easily separated spatially. The authors do present theorem 6.2, and while I can not find any issues with the proof perse, its use is extremely limited. The assumption of the existence of a diffeomorphism that perfectly separates all variables is very strong. Effectively the work has now moved to the assumption, and the authors make no effort to investigate when this assumption is valid. There is moreover the issue that there is no guarantee the reverse process is correct. Altogether, this brings into question of how useful the result in 6.2 is.\n- In this situation, it would be good to have a strong, quantitative, experimental evaluation to demonstrate the use of such a theorem, but as mentioned earlier, particular on evaluation on natural image datasets is lacking.", "other_comments_or_suggestions": ".", "questions_for_authors": "See strenghts and weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6exiLFKOQC", "reviewer": "Reviewer_DBEj", "summary": "The paper proposes a theory for understanding composition in diffusion models and how it can produce samples that are out of distribution for each of the constituent models. Their key insight is that composition of distributions is ill specified unless tied to a projection that specifies which attribute we would like to compose. This leads to the idea of projective composition, which can be realized with the composition operator given sufficient conditions, which are related to the factorability of the distributions we would want to compose. An explicit construction of this composition is provided, which is similar to Bayesian composition except for the replacement of the unconditional score with a background score. It is shown that it is also sufficient for this factorability to be present in some feature space, greatly generalizing the theory, although this result does not provide an explicit construction for sampling from the composed distribution.\n\n## update after rebuttal\nThe empirical evidence provided in the rebuttal addresses my concerns surrounding the empirical evidence. I have increased my score from 3 to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support for our paper and helpful suggestions. \n\n* The reviewer suggests a quantitative analysis of our CLEVR experiments. We agree that this is an excellent idea and have performed the analysis. The results are shown in the table below and will be included in the camera-ready version should the paper be accepted.\n* To produce the table below, we generated 100 samples using each composition method, and manually counted (to avoid any potential error in using a classifier) the objects in correct locations (i.e. locations corresponding to the conditioners of the distributions being composed) in each generated image. In the table below we record the histogram of object counts in correct locations.\n* Regarding the reproducibility of Figures 3 and 5, we provide additional samples in Figures 8 and 10, respectively, in the appendix. Further length-generalization is also explored in Figure 9. In addition, the new table below quantitatively confirms the reproducibility of the results of Figure 3 (when attempting to compose 3 single-object distributions as in Figure 3, the empty-background (projective) composition correctly produced images containing 3 objects in 99/100 trials, while the Bayes composition never produced an image containing 3 objects in 100 trials).\n\nComposition of location-conditioned CLEVR distributions\n* N denotes number of distributions being composed (hence the expected number of objects) -- we test N=1 through N=6\n* \"Single-object empty\" composes single-object object distributions with an empty background\n* \"Single-object Bayes\" composes single-object object distributions with an unconditional background\n* \"Bayes-cluttered\" composes 1-5 object distributions (with location label assigned to a single randomly-chosen object) with an unconditional background\n* The table shows the histogram of manual counts, that is, each column lists the number of images that contained the given number of objects in correct locations.\n\n| Style | N | | | | | | | | |\n|---|----|---|---|---|---|---|---|---|---|\n| Single-object empty | | | 0 | 1 | 2 | 3 | 4 | 5 | 6 |\n| | 1 | | | 100 | | | | | |\n| | 2 | | | | 100 | | | | |\n| | 3 | | | | 1 | 99 | | | |\n| | 4 | | | | | 2 | 98 | | |\n| | 5 | | | | | | 2 | 98 | |\n| | 6 | | | | | | | 3 | 97 |\n| Single-object Bayes  | | | 0 | 1 | 2 | 3 | 4 | 5 | 6 |\n| | 1 || | 100 | | | | | |\n| | 2 ||  10 | 67 | 32 | | | | |\n| | 3 || 36 | 62 | 2 | | | | |\n| | 4 || 77 | 23 | | | | | |\n| | 5 || 66 | 32 | 2 | | | | |\n| | 6 || | 34 | 6 | 3 | | | |\n| Bayes-cluttered   | | | 0 | 1 | 2 | 3 | 4 | 5 | 6 |\n| | 1 || | 100 | | | | | |\n| | 2 || | | 100 | | | | |\n| | 3 || | | | 100 | | | |\n| | 4 || | | | | 100 | | |\n| | 5 || | | | | 2 | 98 | |\n| | 6 || | | | | |  2 | 98 |", "claims_and_evidence": "The theoretical results are well supported with proofs. Some empirical results are given, but appear anecdotal and lacks statistical analysis.", "methods_and_evaluation_criteria": "The main contribution of this work is theoretical, so the limited empirical results do make sense modulo the issues described above.", "theoretical_claims": "The theorems that appear in the main body of the work all make intuitive sense. I have not thoroughly checked the proofs.", "experimental_designs_or_analyses": "As mentioned above, I think the empirical results are more anecdotal than scientific as is. While not the main contribution, some basic analysis could be performed (e.g. something like “out of 100 identical trials, composition with an empty background was successful 100 times and bayesian composition was successful 12 times, which is significantly improved with a p-value of …”).", "relation_to_broader_scientific_literature": "This work provides theoretical foundations for understanding the the composition of diffusion models. While composition has been shown to be possible in prior work, the aim of this work is to provide additional theoretical understanding for why and under what conditions it is expected to work.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "See my suggestions under \"Experimental Designs Or Analyses\".", "questions_for_authors": "Please see my comments under “Experimental Design or Analyses” - I am curious about how consistent the results provided in Figures 3 and 5 are.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a theoretical study on the possibility of composing the score function of a series of diffusion models for generating new data with mixed properties. The authors proposed a projective composition condition and factorized conditionals to allow the composition of score functions. Meanwhile, there are fruitful discussions on the fail of composition and relation to independent coupling, as well as Bayes composition.

The strength of the paper is clearly appreciated by reviewers: 1) clear presentation with extensive discussions; 2) focused on a relevant and interesting topic; 3) correct and sound theoretical results. Therefore, a positive recommendation is formed. However, there are also weaknesses prevent a direct strong accept recommendation: 1) the assumptions are vaguely validated on a particular data set, limiting the broader practicality in the current scope; 2) the assumptions are relatively strong and in the current submission, no relaxation is immediately possible; 3) there are no numerical results demonstrating theory-motivated methods.
Decision: Accept (poster)