CoInD: Enabling Logical Compositions in Diffusion Models
Authors: Sachit Gaudi
Gautam Sreekumar
Vishnu Boddeti
Venue: ICLR 2025 Poster (2025)
Keywords: logical compositionality
generative models
diffusion models
causality
Primary area: generative models
TL;DR: Diffusion models violate the conditional independence required for compositionality. We introduce a training objective, CoInD, to mitigate this violation and significantly enhance compositionality.

Abstract: How can we learn generative models to sample data with arbitrary logical compositions of statistically independent attributes? The prevailing solution is to sample from distributions expressed as a composition of attributes' conditional marginal distributions under the assumption that they are statistically independent. This paper shows that standard conditional diffusion models violate this assumption, even when all attribute compositions are observed during training. And, this violation is significantly more severe when only a subset of the compositions is observed. We propose CoInD to address this problem. It explicitly enforces statistical independence between the conditional marginal distributions by minimizing Fisher’s divergence between the joint and marginal distributions. The theoretical advantages of CoInD are reflected in both qualitative and quantitative experiments, demonstrating a significantly more faithful and controlled generation of samples for arbitrary logical compositions of attributes. The benefit is more pronounced for scenarios that current solutions relying on the assumption of conditionally independent marginals struggle with, namely, logical compositions involving the NOT operation and when only a subset of compositions are observed during training.

Published as a conference paper at ICLR 2025
COIND: ENABLING LOGICAL COMPOSITIONS IN DIFFUSION MODELS
Sachit Gaudi
Gautam Sreekumar
Vishnu Naresh Boddeti
Michigan State University
(gaudi sac, rreekuml, vi visu) @msu. edu
ABSTRACT
How can we learn generative models to sample data with arbitrary logical compo- sitions of statistically independent attributes? The prevailing solution sample from distributions expressed as a composition of attributes' conditional marginal distributions under the assumption that they are statistically independent. This pa- per shows that standard conditional diffusion models violate this assumption, even when all attribute compositions are observed during training. And, this violation is significantly more severe when only a subset of the compositions is observed. We propose COIND to address this problem. It explicitly enforces statistical inde- pendence between the conditional marginal distributions by minimizing Fisher's s divergence between the joint and marginal distributions. The theoretical advan- tages of CoIND are reflected in both qualitative and quantitative experiments, demonstrating a significantly more faithful and controlled generation of samples for arbitrary logical compositions of attributes. The benefit is pronounced for scenarios that current solutions relying on the assumption of conditionally inde- pendent marginals struggle with, namely, logical compositions involving the NOT operation and when only subset of compositions are observed during training. Our code is available at http://hwwbb..oomcccccctt////ooo
INTRODUCTION
Suppel
Uniisen
Nex-anttore
Pertal
4
4
4
4
4
4
(C3)
Color (oo)
Coleri22
7/1
4
4
4
4
4 7
4
4
8
8
H
Dight (C))
Digs
(CC)
Digi (C)
(a) Uniform
(b) Non-uniform
(c) Partial support
(d) Generated samples
Figure 1: Generative Modeling of Logical Compositions. (a-c) Consider the task of generating
MNIST samples for any logical composition of digits and colors by learning observational data of
different supports. (d) Standard diffusion models fail to generate data with arbitrary logical compo- sitions of attributes. We generate data from simple unseen compositions (row 2), and more complex logical compositions (rows 3,4) through CoIND, even under non- non-uniform and partial support.
Many applications of generative models, including image editing (Kim et al. 2022 Brooks et al., 2023), desire explicit and independent control over statistically independent attributes. For example, in face generation, one might want to control the amount of hair, smile, etc., independently. This challenge relates to the broader task of logical compositionality in generative models, where the goal is to combine attributes according logical relations. Consider the illustrative task in Fig, []] of generating realistic samples colored handwritten digits with explicit and independent control over the composition of color and digit. For example, "generate image of digit 4 while excluding the colors green and pink". This composition can be logically expressed as "4 A -(Green Pink]". where A, V. and represent the three primitive logical operators AND, OR, and NOT, respectively.
Existing solutions (Liu et al. 2022; Du et al. 2020 Nie et al. 2021) realize this goal by mapping the logical expressions into a probability distribution involving the conditional marginal distributions

---

Published as a conference paper at ICLR 2025
p(image digit 4), p(image color # Green), and p(image color * Pink), and sampling from it. These marginal distributions are obtained either by learning separate svergy-based models for each compositional attribute (Du et al., 2020; Nie et al.) 202T by factorizing the attributes' learned joint distribution Liu et al. (2022). Both approaches. however, are predicated on the critical assumption that the conditional marginal distributions are statistically independent of each other.
Employing the approaches mentioned above, for instance Liu et al. (2022). to our illustrative ex- ample, we observe that when the conditional diffusion model learned on data with non-uniform (Fig. lb) or partial (Fig. [c] support of the compositional attributes, the models fail to generate re- alistic samples (columns and 5 of row 2 in Fig. [d] or generate realistic samples with logically inaccurate compositions (columns 3 and 5 of rows 3 and 4 in Fig. [Td). This is true even for simple unseen logical compositions of attributes (AND row of Fig. Icl or for complex logical com- positions (rows 3 and 4 of Fig. 1d involving a NOT operation). Such failure under partial support was also observed by Du et al. (2020). Surprisingly, note that even when all compositions of the attributes are observed, the model fails to generate realistic samples (column of row 2 in Fig. ld]
These observations naturally raise the following research questions that this paper seeks to answer:
-RQQ)) Why do standard classifter-free conditional diffusion models fail to generate data with arbitrary logical compositions of attributes? We hypothesize that violating the assumption that the conditional marginal distributions are statistically independent of each other will result in poor image quality, diminished control over the generated image attributes, and, ultimately, failure to adhere to the desired logical composition. We verify and confirm hypothesis through case study in $ $
(RQ2) How can we explicitly enable conditional diffusion models to generate data with arbitrary logical compositions of attributes? We adopt the principle of independent causal mechanisms (Pe- ters et al. 2017) to express the conditional data likelihood in terms of the constituent conditional marginal distributions to ensure that the model does not learn non-existent statistical dependencies.
Summary of contributions.
1. In Section we show that conditional diffusion models trained to maximize the likelihood of the observed data do not learn independent conditional marginal distributions, even when all compositions of the attributes are uniformly (Fig. Ta observed. Furthermore, this problem is exacerbated in more practical scenarios where we learn from non-uniform (Fig. Ib) or partial (Fig. Icl support of the compositional attributes. Instead, the models learn non- existent statistical dependencies induced by unknown confounding factors.
2. Through causal modeling, we derive a training objective, CoIND, comprising the standard score-matching loss and conditional independence violation loss required to enforce the COnditional INdependence relations necessary for enabling logical compositions in condi- tional Diffusion models.
3. Strong inductive biases, in the form of the conditional independence relations in COIND, en- able arbitrary logical compositionality in conditional diffusion models with fine mne-graieed con- trol over conditioned attributes and diversity for unconditioned attributes. CoIND achieves these goals while being monolithic and scalable with the number of attributes.
LOGICAL COMPOSITIONALITY IN DIFFUSION MODELS
We study the problem of generating data with attributes that satisfy a given logical relation between them. We consider the case where the attributes statistically independent of each other. However, not all attribute compositions may be observed during training. To study this problem, we first model the underlying data-generation process using suitable causal model that relates data and their independently varying attributes.
Notations. We use bold lowercase and uppercase characters denote vectors (e.g., a) and matri- ces (c.g., A) respectively. Random variables are denoted by uppercase Latin characters (c.g.. X). The distribution of random variable X is denoted as p(X), or as pe (X) ) if the distribution param- eterized by a vector O. We adopt non-standard terminology where marginals denote the conditionals p(X C₂) rather than integrated marginals, p(C.) emphasizing their functional role as modular com- ponents in our compositional framework. Correspondingly, joint joint refers to p(X C), acknowledging this deliberate departure from probabilistic conventions due to a lack of better terminology.
2

---

Published as a conference paper at ICLR 2025
Ulsbserveed Confouding
Data Generation Process. The data generation process con- sists of observed data X (c.g. images) and its attribute vari- ables C1, C2...., Cn (e.g., color, digit, etc.). To have explicit control over these attributes during generation, they should vary independently of each other. In this work, we limit our study to only those causal graphs in which the attributes are not causally related and can hence vary independently, as shown in Fig. 2a Each C; assumes values from a set C and their Cartesian product C = C1 .... C, is referred to as the at- tribule spuce. Each attribute C; generates its own observed component Xc; = =fc. (Ct), which together with unobserved exogenous variables Ux form the composite observed data x= f(Xc,.... ,c,, Ux) (see Fig. 2a). We do not restrict much except that it should not obfuscate individual observed components in X (Wiedemer et al. 2024). A simple example of is the concatenation function. We also assume that all fc. invertible and therefore it is possible to estimate Ci.... C. from X. These assumptions together ensure that Ct,... Cn are mutually independent given X despite being seemingly d-connected.
(a) True underly- (b) Causal model ing causal model during training Figure 2: (a) C2, Co...... Cn vary freely and independently in the un- derlying causal graph. (b) How- ever, they become dependent dur- ing training due to unknown and unobserved confounding factors.
Problem Statement. When the training data is sampled according to the causal graph in Fig. 2a
all attribute compositions are equally likely to be observed. We refer this scenario as uniform support (illustrated in Fig. [a). However, real-world datasets often deviate from the independence due to unobserved confounders such sample selection bias (Storkey 2008). inducing an attribute shift. As shown in Fig. 2b] this shift modifies the causal structure during training through unobserved confounding relationships, resulting in non-unifon non-uniform support (Fig. [bb) where attribute compositions exhibit unequal occurrence probabilities. In extreme cases, this dependence could lead to the training samples consisting of only a subset of all attribute compositions (Fig. [c], i.e., Cuain C C. We refer to this scenario as partial support. We aim to learn conditional diffusion models under these scenarios to generate samples with attributes that satisfy a given logical compositional relation between them.
The attribute space in our problem statement has the following properties. (1) The attribute space observed during training Crain covers C in the following sense:
Definition 1 (Support Cover). Let C = C1 X ... x Cn. be the Cartesian product of n finite sets C,,..., C.. Consider a subset Cirain C C, where [Craiul = m. Let Cirain = {(c15.-.... Cog) : Caj E C,1 S S n, S S m] and C, = {Cig : 1 S j S m) for sism n. The Cartesian product of these sets is Ciain = C₁ x ..... x C.. We say Cruin covers C iff C = Crain.
Informally, this assumption implies that every possible value that C; can assume is present in the training set, and open-set attribute compositions do not fall under this definition. For instance, in the Colored MNIST example in Fig. [ we are not interested in generating digit with an unobserved 11 color. (2) For every ordered tuple c E Cuain. there is another c' E Crain such that e and c' differ on only one attribute. Similar assumptions were discussed in (Wiedemer et al. 2024
Preliminaries on Score-based Models. In this work, we train conditional score-based mod- els (Song et al.] (0211b using classifier-free guidance (Ho & Salimans (022) to generate data corresponding to a given logical attribute composition. Score-based models learn the score of the observed data distributions Purain( (X) and Purin [ C') through score matching (Hyvärinen) 2005). Once the score of a distribution is learned, samples can be generated using Langevin dynamics. For logical attribute compositional generation, the given attribute composition is decomposed in terms of two primitive logical compositions: (1) AND operation (e.g., C1 = CI A C2 = C2 gener- ates data where attributes C1 and C2 takes values e₁ and C2 respectively). and (2) NOT operation (e.g., C1 = -C generates data where the attribute C1 takes any value except e1). Liu et al. (2022) proposed the following modifications during sampling to enable AND and NOT logical operations between the attributes, assuming that the diffusion model learns the conditional independence rela- tions from the underlying data-generation process, i.c., p(C),...., Cr XX) = II (().
Since Pe A C2 X) (C | X (C2 X) samples are generated for the logical composition C₁ A C2 by sampling from the following score:
Logical AND (A) operation:
V x log X
C1 A C2)
Vx log po X C1) V log X C2) Vx log (1)
3

---

Published as a conference paper at ICLR 2025
Logical NOT (-) operation: Following the approximation po po X) po(Cs|/jj, the score to sample data for the logical composition C₁ -C2 can be expressed as,
Vx ( CI A C2) = V log pe X) + Vx log Cii Vx logpo C2)
(2)
To achieve precise control over attribute composition, the hyperparameter y is used to modulate the relative intensity of attribute C2 with respect to Cj. We sample from the distribution, Vx logpe CLA C2), expressed as
Precise Control:
(1) + V ( X C2) - >V log (X) Vx Po ( X
(3)
Logical OR (V) operation: From the rules of Boolean algebra, Ci V C2 operation can be expressed in terms of A and - as --- A -C2). Following the approximation for - from above, it follows that p(-(-(( A -C2)) = ((C))
For example, to generate colored handwritten digits with the "4 A - -[Green V Pink]" logical compo- sition, the score of the logical composition can be decomposed into its constituent logical primitive operations and further in terms of the score of marginals, which can be obtained from the trained diffusion models. Therefore, Vx log po X
4 A
PJ) is given by:
Vx Po X C 4 C2 -G) Vx log Pa X C AAC2 = -P) Vx (X()
x log X C1 = Vx lo C2 = G) Vx logps X C2 = + Vx log po (X)
Note that the scores to sample from these primitive logical compositions involve conditional marginal likelihood terms X C.. Therefore, to perform logical composition, it is critical to accu-
rately learn the conditional marginals of the attributes.
Evaluation We evaluate the distributions learned by the model based on their accuracy in gen- erating images with attributes that align with the desired compositions for a logical relation. For example, to evaluate AND (A) composition, consider sampling an arbitrary digit and color, repre- sented as C' = (4, Cyan). We generate images X by sampling from Eq. (. and subsequently infer attributes, (éi, 02) (Øc, (X), &c, (8)). We then verify if (21,02) {4) {Cyan), and this process is averaged over all combinations in C to obtain CS.
Conformity Score (CS) To formally define CS: For a logical relation, R. This relation is defined as boolean function over the attribute space C, such that R : C + {0, 1}. This induces a constrained attribute space given by R ((c,,...,C.) R(cI,...,, Cn) 1) S C. The CS is defined as: (go CS 0) Ec-noy [1r (10c, (R(C),
(4)
where R(C') can represent various logical operations such as A, -, and V on the attribute space C. Here, ge ge( (R(C), U) denotes generative model parameterized by 0, which samples according to the logical relations specified above. The variable U represents exogenous noise in the diffusion model.
The functions Φc, are attribute-specific classifiers that infer attributes from the generated images. The term RC is an indicator function, equals if the inferred attributes (øc, (go (R(C), U))) S
Re. Further details regarding the Conformity Score can be found in App. D.6
3
WHY DO CONDITIONAL DIFFUSION MODELS FAIL TO GENERATE DATA WITH ARBITRARY LOGICAL COMPOSITIONS OF ATTRIBUTES?
To address (RQ1), we utilize the task of generating synthetic images from the Colored MNIST dataset for any given combination of color and digit, as introduced in [ To study the effect of data
support, we consider the three training distributions of attribute compositions defined in $ (1) uni-
form support, where every ordered pair in C' has an equal chance of being observed (Fig. Lap. (2) non-
uniform support, where every ordered pair in appears but with unequal probabilities (Fig lb). and
(3) partial support, where only subset of ordered pairs, Cuain C are observed (Fig. (c).
For each support, we train a diffusion model and evaluate the conditional joint, Po C) and marginal, (X Ct) distributions. During inference, the images are separately sampled from the joint distribution, x log | C), and from the product of the learned marginals as shown in Eq. (T) We refer the former method as joint sampling and the latter as marginal sampling. To
4

---

Published as a conference paper at ICLR 2025
Conformity Soore
Conformity Scores and Jensen-Shannon divergence for samples generated from joint and marginal distributions learned by models under vari- ous support settings for the Colored MNIST dataset.
Table 1:
Support
JSD 4
Joiat
Marginal 1
Uniform 99.98
98.15
0.16
Non Uniform
99.98
86.10
0.30
Partial 33.14
7.40
2.75
measure the accuracy of the attributes in the generated image in accordance to the desired attributes, we use conformity score (CS) defined in 2 Tab.[commaree the joint and the marginal distributions learned by models trained under various training scenarios. We draw the following conclusions.
Diffusion models struggle to generate unseen attribute compositions. From the conformity scores of images sampled from the joint distribution, we conclude that while the models trained with uniform and non non-uniform support generate images with accurate attribute compositions, those trained with partial support struggle to generate images for unseen attribute compositions. The standard training objective of diffusion models is to maximize the likelihood of conditional gen- eration, for every observed attribute composition, the model accurately learns Prein (X C), i.e., P C) R Ptrain (X C)) However, with partial support, the model does not observe samples for every attribute composition from Purai
C). Therefore, the model does not accurately learn the
density of the unobserved support region.
Diffusion models violate underlying Conditional Independence relations. Although the dif- fusion model is trained on all marginals (X C.), per the support cover assumption, marginals samples performs inferior to that of sampling from the joint distribution. This further drop in con- formity score when sampled from the product of marginals ( (Eq. ( for the models trained under non-uniform and partial support settings is due to the disparity between the joint distribution and the product of marginals, which points the violation of independence relations from the underlying data-generation process in the learned model. Refer to App. [ for a detailed proof. To further strengthen the claim, we measure this violation as the disparity between the conditional joint distri- bution pe(Co (C') X) and the product of conditional marginal distributions II." Pe(C; | X) learned by the guidance term model using Sonsen-Shannor divergence (ISD):
JSD Ec,x Dis PC X)
II po (C; xx)
(5)
where Dis is the densen-Shannon divergence and following (Li et al. 2023) po is obtained by eval- uating the implicit classifier learned by the diffusion model. More details can be found in App. D.7
positive JSD value suggests that the model fails to adhere to the independence relations present in the underlying causal model. Our findings (Tab. [ indicate that as the training distribution of attribute compositions diverges from the true underlying distribution where attributes vary inde- pendently the trained models increasingly violate independence relations, reflected by the JSD. These findings demonstrate diffusion models lack inherent compositional bias, instead propagate dependencies as present in their training data.
Training objective of the diffusion models is not suitable for logical compositionality. The objective of the diffusion models trained with classifier-fre free guidance is to maximize the condi- tional likelihood of power-set of attributes. However, due to confounding induced by the train- ing support (Fig. 2bb. the attributes become dependent during training, i.e.. Prrain (C1, 1, ... C.,) K Il== Phrain (C.). As result, the conditional distribution of marginals does not match its true under- lying distribution. i. (X C.)) Purain (X
C+) # Pdata (X C.) Refer to App. B.2 for formal proof. Therefore, any method (Nic et al. 2021) that relies on training on these incorrect marginals or relies on conditional independence (Liu al., 2022) bound fail. Moreover, even when realistic samples of unseen composition are successfully generated, it is by accident rather than design.
Failure of Logical Compositionality
Standard conditional diffusion models trained with classifier-fre guidance struggle to gen- erate data with arbitrary logical compositions of attributes because they violate the indepen- dence relations inherent in the causal data-generation process.
Based on these observations. propose CoIND to train diffusion models that explicitly enforce the conditional independence dictated by the underlying causal data data-generation process to encourage the model to learn accurate marginal distributions of the attributes.
5

---

Published as a conference paper at ICLR 2025
4 COIND: ENFORCING CONDITIONALLY INDEPENDENT MARGINAL TO ENABLE LOGICAL COMPOSITIONALITY
In this section, we propose CoIND to answer (RQ2) posed in $ [ How can explicitly enable conditional difflusion models to generate data with arbitrary logical compositions of altribute??
In the previous section, we observed that diffusion models do not obey the underlying causal rela- tions, learning incorrect attribute marginals, and hence struggling to demonstrate logical composi- tionally we showed in Fig. To remedy this, COIND uses training objective that explicitly enforces the causal factorization to ensure that the trained diffusion models obey the underlying causal relations. From the causal graph Fig.
2a along with the assumption of C1 11 1L C. X
mentioned in $ 2 we have p p
()
P II:
p(x(C))p(Ct) Note that the invariant p(X C)
PIC
p(X)
now expressed as the product marginals employed for sampling. Therefore, training the diffu- sion model by maximizing this conditional likelihood is naturally more suited for learning accurate
marginals for the attributes. We minimize the distance between the true conditional likelihood and the learned conditional likelihood as,
Pe (C)
Pe X (C,)
Leump W/
P(X C),
(6)
po (X)
where W2 is 2- Wasserstein distance. Applying the triangle inequality to Eq. (6) we have,
PD (X)
Po X (o:)o(C()
Ceomp
W2 (p(X X
Co (X C))
W22
po (X
C):
po
po (X)
Distribution matching
Conditional Independence
Kwon et al. by the square root of the score-matching objective.
2022 showed that the Wasserstein distance between (X) qo () is upper bounded
W2 (po (X), (x)) s K (x) [II x log po (X) V log g0 (X)
Distribution matching: Following this result, the first term in Eq. is upper bounded by the standard sctre-maa objective of diffusion models (Song et al. 2021b),
Licore = Pxxx ()
| Vx log pe X C) - Vx log p(x C)
(8)
Conditional Independence: Similarly, the second term in Eq. is upper bounded by score- matching between the joint and product of marginals
Lai = E| V log
X
C)
V (X)
V X
C.) - Vx po (X)
(9)
Substituting Eq.
Eq.
in Eq. will result in final learning objective
Commp I K1 / Cycure K2 Lei
(10)
where K1, K2 are positive constants, i.e., the conditional independence objective Lai is incorporated alongside the existing score-man more-matching loss Eycore-
Lei, the Fisher divergence between the joint and the product of marginals. From the prop-
erties of Fisher's divergence SSánchez-Moreno et al. (2012). Lei 0 iff (X C)
xo(X) Po(C)
II (x((), Detailed derivation of the upper bound can be found App.
B.3
(a(X)
computational burden presented by Ler in Eq. (9p is that the re- quired number of model evaluations increases linearly with the number of attributes. To mitigate this burden, we approximate the mutual conditional independence with pairwise conditional inde- pendence (Hammond & Sun 2006). Thus, the modified Lai becomes,
Practical Implementation.
fa - Ep(x, c. & X log Po (X Cy, C+) Vx log Po X |C,) Vx log C+) V (X)
The weighted sum of the square of the terms in Eq. (1O) has shown stability. Therefore, CoIND's training objective:
Limal = Lucare + XLe
(11)
where À the syper-parameter that controls the strength of conditional independence. The reduction to the practical version of the upper bound (Eq. (IO)) is discussed in extensively in App. C For
guidance on selecting syper-parameters proposed approach can be implemented with just few lines of code, outlined in Algorithm
principled manner, please refer to App. C.3 Finally,

---

Published as a conference paper at ICLR 2025
5 DOES COIND IMPROVE THE LOGICAL COMPOSITIONALITY?
COIND encourages diffusion models to learn conditionally independent marginals of attributes, and thereby improve their logical compositionality capabilities. In this section, we design experiments to evaluate CoIND on two questions: (1) Does CoINi) effectively train diffussin models that obey the underlying causal model?, and (2) Does COIND improve the logical compositionality of these models? We measure the JSD of the trained models answer the first question. To answer the second question, we use two primitive logical compositional tasks: (a) A (AND) composition and (b) (NOT) composition. In each case, the generative model is provided with a logical relation between the attributes, and the task is to generate images with attributes that satisfy this logical rela- tion. more detailed description of task construction can be found in App. D.2. Beyond improved logical compositionality, we ask: Does learning conditionally independent marginals lead to greater diversity in uncontrolled attributes and enhanced controllability of attributes?
Datasets. We use the following image datasets with labeled attributes for our experiments: (1) Colored MNIST dataset described in [], where the attributes of interest are digit and color, (2) Shapes3d dataset (Kim & Mnih 2018) containing images of 3D objects in various environ- ments where each image is labeled with six attributes of interest. (3) CelebA with gender and smile attributes demonstrates effectiveness of CoIND on real-world datasets. Refer to App. D.5
Observed training distributions. We evaluate COIND on four scenarios where we observe different distributions of attribute compositions during training: (1) Uniform support, (2) Non-uniform support (3) Diagonal par- tial support, defined in $2 (4) Orthogonal partial support includes only the attribute compositions along the axes originating from corner of the hy- percube C. following (Wiedemer et al. 2024) (Fig. 3 For Colored MNIST experiments, we evaluate with uniform, non uni-urnifom. and diagonal partial support. For Shapes3d experiments, we evaluate with uniform and orthogo- nal partial support, following the compositional setup in (Schott et al. 2020)). We evaluate CelebA on orthogonal partial support, where all compositions except unseen male smiling celebrities are observed.
Figure 3: Orthogo-
nal partial support
Baselines. LACE (Nie et al. 202I) and Composed GLIDE (Liu et al. 2022) are our primary baselines. LACE trains distinct energy-based models (EBMs) for each attribute and combines them following the compositional logic deseribed in $ 2 during sampling. A similar approach was pro- posed by (Du al. 2020). However, in our experimental evaluation for LACE, we train distinct score-based models instead of EBMs. In contrast, Composed GLIDE samples from score-based models by factorizing the joint distribution into marginals, assuming these models had implicitly learned conditionally independent marginals of attributes. Additional details about the baselines are delegated to App. D.3
Metrics. We assess how accurately the models have captured the underlying data generation pro- cess using the JSD, defined in 3 To measure the accuracy of the attributes in the generated image w.r.t. the input logical composition, we use conformity score (CS) from $ 2 As a reminder, CS mea- sures the accuracy with which the model adds the desired attributes the generated image using attribute-specifi classifiers. In addition to the conformity score, since the Shapes3d dataset contains unique ground truth images corresponding to the input logical relation, we directly compare gen- erated samples with reference images at the pixel level using the variance-weighted coefficient of determination, R²², Additionally, for CelebA, we measure FID (Setzerr 2020). We evaluate uniform and non-uniform support the generations for the input logical relations correspond to attribute compositions that span the attribute space C. In other cases, we evaluate models ability to generate input logical relations corresponding to the unseen compositional support, i.e., C Cirin-
Learning Independent Marginals Enables Logical Compositionality
Fig. 4a compares CoIND with baselines A and composition tasks. The "- Color" task gener- ates images with the negation applied on color attribute, while"- Digit" applies the negation to the digit attribute. From these results, we make the following observations:
Conditional diffusion models do not learn accurate marginals even when all attribute compo- sitions are observed during training with equal probability. This is evident from the positive JSD
7

---

Published as a conference paper at ICLR 2025
Support
Method
JSD
A (CS) f
Color (CS)
Digit (CS) f
100
LACE
96.40
92,56
83.67
Composed GLIDE
0.16
99.30
Uniform
COIND (A 0.2)
0.14
99.73
99.32
84.94
75-
CoIND (A 1.0)
0.10
99.99
99.33
89.60
50
LACE
82.61
65,16
Non-uniform
Composed GLIDE
0.30
86,10
81,61
70.44
CoIND (A 1.0)
0.15
99.95
92.41
84,98
25
LACE
10.85
9,03
28.24
Partial
Composed GLIDE
2.75
7,40
5.09
33.86
10
CuIND (A 1.0)
1.17
52.38
53,28
52.59
JSD
(a) Results on Colored MNIST Dataset
(b) JSD vs CS
Figure 4: Results on Colored MNIST dataset. (a) We compare ISD and CS of CoIND against
baselines trained under various settings and on different compositional tasks. (b) Plotting CS against JSD in the log scale of the models trained under different settings reveals a negative correlation.
of the methods trained with uniform support. Furthermore, the conformity score (CS) lower when JSD is higher. This observation has significant ramifications for compositional generative models.
This result contradicts the intuitive expectation that uniformly observing the whole compositional support during training is sufficient to generate arbitrary logical compositions of attributes. And, suggests that even in this ideal yet impractical case, the current objectives for training diffusion models are insufficient for controllable and accurate closed-set, let alone open-set, compositional generation. As such, we conjecture that scaling the datasets without inductive biases (conditional independence of marginals in this case) insufficient for arbitrary logical compositional generation.
Even methods like LACE that train separate models for each attribute fail on composition tasks. This suggests that softer inductive biases, such learning separate marginals for each attribute with- out paying heed to the desired independence relations, are insufficient for logical compositionality.
In the more practical scenarios of non- non-uniform and partial support, JSD increases with non- non-uniform support and worsens further with partial support due to incorrect marginals as discussed in $ This result suggests that current -late-of-the-ee non-uniform or partial support scenario and thus may fail to generate accurate and realistic data for arbitrary logical compositions of attributes.
models learned on finite datasets likely operate in the
Logical AND (A) and NOT (-) compositionality deteriorates with increasing dependence between the marginals. The negative correlation between JSD and CS was noted in and can observed in Fig. 4b, which shows JSD-vs-CS for A compositions across different methods, and under different settings for observed support. This negative correlation strongly suggests that violation of condi- tional independence plays major role in the diminished logical compositionality demonstrated by standard diffusion models.
By enforcing conditional independence between the attributes during training, CoIND achieves lower JSD and improves both A and - compositionality in non-uniform and partial support. Even when trained on non-uniform support, CoIND matches compositionality with the uniform support in terms of compositional score. Under partial support setting, COIND achieves R 2 - 10x fold improvement over the baselines A and compositions. These results demonstrate that enforcing conditional independence between the marginals is vital for enabling arbitrary logical compo- sitions in conditional diffusion models.
4
(a) LACE; H 1.82
(b) Composed GLIDE; H = 1.71
(c) COIND; H = 2.63
Figure 5: Images generated by CoIND for the logical composition digit 4 under non-uniform scenario are significantly diverse compared to the baselines. / is the Shannon entropy.
s COIND generates diverse samples. It is desirable that any attribute not part of the logical composition for generation assumes diverse values in the generated samples to avoid harmful gen- erated content-ing includng stereotypes (Dehdashtian al. (2025) and biases (Luccioni et al., 2024). In Fig. 5 we observe that although ColND does not explicitly optimize for diversity, the samples generated by CoIND for the logical relation digit 4 are significantly more diverse compared
8

---

Published as a conference paper at ICLR 2025
to the baselines. We quantitatively measure the diversity of these images using the Shannon en- tropy H of the color attributes in the generated images. Higher Shannon entropy indicates more diversity. Entropy is maximum for uniform distribution with H (uniform)
(ogg(10) = 3.32,
since there are 10 colors. We observe that H(COIND) = 2.63, while H(LACE) = 1.82, H (Composed GIJDE) 1.71. Although COIND does not explicitly seek diversity, breaking the dependence induced by unknown confounders exhibits diversity in attributes.
Expected
CoIND
Dampesed GLIDE
LACE
Composition
Composition
Support
Method
JSD J
Re 1
CS
R²t
CS²
Unitonn A compp
t
LACE
0.97
91.19
0.85
50.00
Uniform
Composed GLIDE CoIND (A 1.0)
0.302
0.94
83.75
0.91
48.43
Portill
0.215
0.98
95.31
0.92
55.46
comp.
LACE
0.88
62.07
0.70
30.10
Orthogonal
Composed GLIDE CoIND (A= 1.0)
0.503
0.86
51.56
0.61
34.63
Portal
0.287
0,97
91.10
0.92
53,90
comp
(a) Quantitative Results on S Dataset
(b) Visual comparison of samples
Figure 6: Results Shapes3d dataset. (a) We compare JSD, R?, and CS of CoIND against the baselines trained with uniform and partial support on the Shapes3d dataset for A and - composition tasks. (b) Samples generated by CoIND match the expected image in all cases.
COIND is scalable with attributes. We use the Shapes3d dataset to evaluate the scalability of COIND w.r.t. the number of attributes. As reminder, every image in the Shapes3d dataset is labeled with six attributes of interest. For the negation composition task, the - operator is applied to the shape attribute such that the attribute composition satisfying this logical relation is unique. Detailed descriptions of the composition tasks are provided in App. D.2 Fig. 66a compares COIND against the baselines for the uniform and orthogonal partial support scenarios. CoIND leads to a significant decrease in JSD and, consequently. significant increase in the composition score. When trained with orthogonal support, the performance (CS) of both LACE and Composed GLIDE suffers significantly while CoIND matches its performance when trained on uniform support. In conclusion, COIND affords superior logical compositionality from a single monolithic model in a sample-efficient manner even as the number of attributes increases.
CoIND generates unseen compositions of real-world face images
We evaluate ability of CoIND to generate un- seen "smiling male celebrities". Diffusion model is trained on all compositions of the CelebA dataset (Liu et al. 2015) except gender "male" and smiling "true". This is equiva- lent to the orthogonal support scenario shown in Fig. 3 During inference, the model is asked to generate images with the unseen attribute com- bination gender "male" and smiling- "true" through joint sampling and A composition.
"smiling male"
"smiling"/ "A"male"
Method
JSD
CS1
CS
FID
LACE
24.20
80.40
Composed GLIDE COIND (A= 100)
2.44
2.51
61.21
10.55
95.41
1.82
8.63
43,97
8.79
43.76
Table 2: Results on CelebA dataset. CoIND out- performs the baselines on both CS and FID across various compositionality tasks.
Tab. [] compares CoIND against baselines in terms of CS and FID. (1) COIND outperforms the baseline by > 4x in joint. (2) COIND generates realistic faces, closer to smiling male celebrities in the held out set, as measured by FID and displayed in Fig. 7c y 1). In App. E.3 we show that CoIND extends to Text-to- Texa-to-Image models by fine-tuning Stable Diffusion (Rombach et al. 2022).
COIND provides fine-grained control over attributes. In addition to merely generating sam- ples with conditioned attributes, CoIND can also control the amount of attributes in the sample. For example, in the task of generating face images of smiling male celebrities, we may wish adjust the amount of smiling without affecting gender-specific attributes. To achieve this, we sample from Eq. 3 where 7 controls the strength of smile. Fig. 7 shows the result of increasing ? to increase the amount of smiling in the generated image. The subjects in the face images generated by COIND attribute. For instance, the images for y = 1 show soft smile while the subjects in the images for y 6 show teeth. However, those generated by baselines contain gender-specifie attributes such as long hair and carrings. These distinctions are quantified in Figs. 7a and 7b Refer App. E.2 for more analysis.
smile more as 7 increases without any changes to any gender-specing

---

Published as a conference paper at ICLR 2025
Methods
CoIND
Composed GI IDE
LACE
LACE
Composed GLIDE
ColnD
The ofaammottoaaaiiii Fommle vineresies
40-
100
0
30-
75
20
50-
10
7
(c) Samples with Y
(a) FID with 7
(b) CS with Y
Figure 7: Effect of 7 on FID and CS: Varying the amount of smile in generated image through y does not affect the FID of COIND. However, the smiles in the generated images become more apparent, leading to easier detection by the smile classifier and improved CS.
6 RELATED WORK
Our work concerns compositional generalization in generative models, where the goal is to gener- ate data with unseen attribute compositions expressed through logical relations between attributes. One class of approaches achieves logical compositionality by combining distinct models trained for each attribute (Du et al. 2020) Liu et al. 202T, Nie al. 202T] Du et al. (023). In contrast, we are interested in monolithic diffusion models that learn logical compositionality. Besides being ex- pensive and scaling linearly with the number of attributes, these models fail under practical partial support scenarios. [Lu et al. (2022) studied logical compositionality broadly without differentiat- ing between attribute supports and proposed methods to represent logical compositions terms of marginal probabilities obtained through factorization of the joint distribution. However, these fac- torized sampling methods fail since the underlying generative model learns inaccurate marginals. In comparison, CoIND is trained to obey the independence relations from the underlying causal graph. Also, (Cho et al., 2024) note that diffusion models lack the conditional independence needed for controllability and address this with a hyperparameter during sampling. We argue that, even with disentangled features, learning accurate marginals tackles the root cause more effectively than such post-hoc adjustments. Encouragingly. Okawa et al. (2023) shows that compositional abilities emerge multiplicatively. Liang et al. (2024) highlights factorization in diffusion models, suggesting they naturally exhibit compositional capabilities. However, these studies focus on generating from the joint distribution-a special case of logical compositionality-
are limited to binary attributes. Our work extends these ideas to more general compositions. Lastly, (Wiedemer al., 2024) studies compositional generalization for supervised learning and provides sufficient conditions for compo- sitionality. Our empirical observations in generative models are consistent with their theoretical results, suggesting that their findings could perhaps be extended to conditional diffusion models.
7 CONCLUSION
Conditional diffusion models struggle to generate data for arbitrary attribute compositions, even when all attribute compositions are observed during training. Existing methods represent logical relations in terms of the learned marginal distributions, assuming that the diffusion model learns the underlying conditional independence relations. We showed that this assumption does not hold in practice and worsens when only subset of these attribute compositions are observed during train- ing. To mitigate this problem, we proposed CoIND to train diffusion models by maximizing con- ditional data likelihood in terms of the marginal distributions that are obtained from the underlying causal graph. Our causal modeling provides COIND a natural advantage in logical compositional- ity by ensuring it learns accurate marginals. Our experiments on synthetic and real image datasets highlight the theoretical benefits of CoIND. Unlike existing methods, CoIND is monolithic, easy to implement, and demonstrates superior logical compositionality. CoIND shows that adequate induc- tive biases such as conditional independence between marginals are necessary for effective logical compositionality. Refer to Apps. | Fndd for more discussions, analysis and limitations of COIND.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements: This work was supported by the Office of Naval Research (award #N00014- 23-1-2417). Any opinions, findings, and conclusions or recommendations expressed this material those of the authors and do not necessarily reflect the views of ONR. We also thank Mashrur Morshed for his insights on improving diffusion model training and for sharing the bare-bones code, Lan Wang for providing the code for fine-tuning Stable Diffusion, and Ramin Akbari for his assis- lance with the proof's presented in Apps. B and
REFERENCES
Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to Follow Image Editing Instructions. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Wonwoong Cho, Hareesh Ravi, Midhun Harikumar, Vinh Khuc, Krishna Kumar Singh, Jingwan Lu, David Inouye, and Ajinkya Kale. Enhanced controllability of diffusion models via feature disentanglement and realism-enhanced sampling methods. In European Conference on Computer Vision, 2024.
Sepehr Dehdashtian, Gautam Sreekumar, and Vishnu Naresh Boddeti. OASIS Uncovers: High- Quality T21 Models, Same Old Stereotypes. In International Conference on Learning Represen- fations, 2025.
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs Image Synthesis. In Advances in Neural Information Processing Systems, 2021.
Yilun Du and Leslie Kaclbling. Position: Compositional Generative Modeling: A Single Model is Not All You Need. In International Conference Machine Learning. 2024.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional Visual Generation with Energy Based Models. In Advances in Neural Information Processing Systems, 2020.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, Reuse, Recycle: Compositional Generation with Energy-Base Based Diffusion Models and MCMC. In International Conference Machine Learning, 2023.
Peter J Hammond and Yeneng Sun. The essential equivalence of pairwise and mutual conditional independence. Probability Theory and Related Fields, 135(3):415-277, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance, 2022.
URL https:
wtaxxiv.org//bb//2222 12598
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, 2020.
Aapo Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 6:695-709, 2005.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. DiffusionCLIP: Text-Guided Diffusion Mod- els for Robust Image Manipulation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising. In International Conference on Machine Learning, 2018
Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based Generative Modeling Secretly Mini- mizes the Wasserstein Distance. In Advances in Neural Information Processing Systems, 2022.
Tuomas Kynkäänniemi, Miia Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models. In Advances in Newral Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2025
Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your Dif- fusion Model is Secretly a Zero-Shot Classifier. In /EEE/CVF International Conference on Com- puter Vision, 2023.
Qiyao Liang, Ziming Liu, Mitchell Ostrow, and Ila R Fiete. How Diffusion Models Learn Fac- torize and Compose. In Advances in Neural Information Processing Systems, 2024.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli
and Itai Gat. Flow Matching Guide and Code, 2024.
wttp://pppp.a.a./e 06264
Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to Compose Visual Relations. In Advances in Neural Information Processing Systems, 2021.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. In European Conference on Computer Vision, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In IEEE/CVF International Conference on Computer Vision, 2015.
Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jemnite. Stable Bias: Evaluating Societal Representations in Diffusion Models. In Advances in Neural Information Processing Systems, 2024.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and Compositional Generation with Latent-Space Energy-Basd -Based Models, 2021. URL
10873
Maya Okawa, Ekdeep Lubana, Robert Dick, and Hidenori Tanaka. Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task. In Advances in Neural Information Processing Systems, 2023.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Founda- tions and Learning Algorithms. The MIT Press, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis With Latent Diffusion Models. In IEEE/CVF Conference Com- puter Vision and Pattern Recognition, 2022.
Pablo Sánchez-Moreno Alejandro Zarzo, and Jesús S Dehesa. Jensen divergence based on Fisher's information. Journal of Physics A: Mathematical and Theoretical, 45(12):125305, 2012.
Lukas Schott, Julius Von Kügelgen, Frederik Träuble, Peter Vincent Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual Representation Learning Does Not Generalize Strongly Within the Same Domain. In International Conference on Learning Representations, 2020.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. pytorch- fid August 2020. Version 0.3.0.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusson Implicit Models. In Inter- national Conference Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion. In Advances in Neural Information Processing Systems, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In Interna- tional Conference on Learning Representations, 202
Amos Storkey. When Training and Test Sets Are Different: Characterizing Learning Transfer. In
Dataset Shift in Machine Learning. The MIT Press, 2008.
12

---

Published as a conference paper at ICLR 2025
Max Welling and Yee W Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference on Machine Learning, 2011.
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Composi- tional generalization from first principles. In Advances in Neural Information Processing Systems, 2024.
13

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Preliminaries Score-based Models
Prools for Claims
B.1
Proof for the case study $
B.
2.2
Standard dilfusion model objective suitable for logical compositionality
16
B.3
Step derivation of CoIND in $4
17
C
Practical Considerations
18
C.1
Scalability of La
18
C.2
Simplification of Theoretical Loss
19
C.3
Choice of Hyperparameter X
20
D
Experiment Details
20
D.1
CoIND Algorithm
20
D.2
Details Logical Compositionality Task
................................
21
D.3
Training details, Architecture, and Sampling
................................
22
D.4
Analytical Forms Support Settings
23
D.5
Datasets. ................................
23
D.6
Conformity Score (CS)
23
D.7
Computing ISD
24
D.8
Measuring Diversity in Attributes
24
E
ColND for Face Image Generation
25
CoIND can successfully generate real-world face images
25
H.2
ColND provides fine- graingr control over attributes
25
E.3
Finetuning T21 models with CoIND improves logical compositionality
27
Discussion on ColND
28
F.1
Connection compositional generation from first principles
28
F2
2D Gaussian: Closed- form Analysis of ColND
28
F3
Extension to Gaussian source flow models
30
F4
Compositional Monolithic models
30
Limitations
30
G
Additional Results and Discussion on COIND
31
G.I
Learning under non-uniform p(C:)
31
G.2
Failure examples of COIND)
................................
31
G.3
Conformity score for each attribute combination
................................
31
G.4
CoIND also improves conditional generation
................................
32
G.5
ColND interpolate between discrete altributes
33
14

---

Published as a conference paper at ICLR 2025
A PRELIMINARIES OF SCORE-BASED MODELS
Score-based models Score-based models (Song et al. 202 lb) learn the score of the observed data
distribution, Purin ()) through score matching (Hyvärinen, 2005). The score function se sx(x) Vx log (x) is learned by a neural network parameterized by 0.
Lecore Ex--een
[|s0(x)
Vx log Puain [x)
(12)
During inference, sampling is performed using Langevin dynamies:
X x + ! Vx (x. + Viees, 6, N(
(13)
2
where 77 > 0 is the step size. As 1 + 0 and T + oc, the samples x converge to (xX under certain regularity conditions (Welling & Teh 201T
Diffusion models [Song & Ermon proposed a scalable variant that involves adding noise to the data. Ho al. has shown its equivalence Diffusion models. Diffusion models are trained by adding noise to the image x according to a noise schedule, and then neural network, is used to predict the noise from the noisy image, Xt- The training objective of the diffusion models is given by:
Lscrr Excepui [e |le Eo (x),
(14)
Here, the perturbed data x, is expressed as: X₁ = vā,x + -I- āje where āe = II 0g for pre-specified noise schedule az, The score can be obtained using,
Ex
sp (x(x,, t)
(15)
V1 -
Langevin dynamics can used to sample from the so (xx, () to generate samples from p(X). The conditional score (Dhariwal & Nichol 202T) is used to obtain samples from the conditional distri- bution (X C)) as:
Vx, (X, C*= Vx, Po (X,) V log (C X,)
Unconditional score
noisy classifier
where y is the classifier strength. Instead of training separate noisy classifier, Ho & Salimans have extended to conditional generation by training V x, log tog (X C* so Xx, I, C). The sampling can be performed using the following equation:
Vx, log p(X, C*) = v) x, log 7 V Vx log (X, C)
(16)
However, the sampling needs access to unconditional scores as well. Instead of modelling Xx, log (X,). V Vx, log po (X.(C) as different models Ho & Salimans have amortize train- ing a separate classifier training a conditional model sø sx(x,, c) jointly with unconditional model trained by setting Ø
In the general case of classifier-free guidance, a single model can be effectively trained to accom- modate all subsets of attribute distributions. During the training phase, each attribute e is randomly set to Ø with probability Puncond- This approach ensures that the model learns to match all possible subsets of attribute distributions. Essentially, through this formulation, we use the same network to model all the possible subsets of conditional probability.
Once trained, the model can generate samples conditioned on specific attributes, such as ca and Cj. by setting all other conditions to Ø. The conditional score is then computed as, Vx, log (X( = Xt, chd), where eij represents the condition vector with all values other than and j set to Ø. This method allows for flexible and efficient sampling across various attribute combinations.
Estimating Guidance Once the diffusion model trained, we investigate the implicit classifier, (e X), learned by the model. This will give us insights into the learning process of the diffusion models. (Li et al. 2023) have shown way to calculate po(C; = ca X = x), borrowing equation (5), (6) from (Li et al. 2023).
p(c:) px(x cc))
Pa (C.
xx)
D(cu)) px cu)
15

---

Published as a conference paper at ICLR 2025
exp( (x [/|e - (X,, t, c
Po(C; = x)
(17)
Bc. lexpl Rr,e (x, t, c* ||]
Likewise, extend it to joint distribution by
exp(-Er,d E, tx,, t,, po = c. = 1 = Ec.,c, lexpl - E1,-E,,lllee (x,,,, )
(18)
The authors Li al. have showed many axproximations compute Br... However, we use a different approximation inspired by Kynkäänniemi et al. (2024), where we sample 5 time-steps between [300,600] instead of these time-steps spread over the [0. T).
Practical Implementation
B PROOFS FOR CLAIMS
In this section, detail the mathematical derivations for case study from $ in App. relate the origin of the conditional independence violation to the unsuitable loss function of vanilla diffusion models in App. B.2 and then derive the final loss function of CoIND in App. B.3
B.1 PROOF FOR THE CASE STUDY IN $
In this section, we prove that failure of compositionality in diffusion models is due to the violation of conditional independence.
Following conditional independence relation:
p(C X) II P(C; XX)
(CI relation)
This CI relation used by several works (Liu al. 2022 Nie et al. 2021). including ours, derive the expression for the joint distribution p(X C') in terms of the marginals p(X C+) for logi-
cal compositionality. As reminder, logical compositionality is preferred over simple conditional generation as it (1) provides fine-grained control over the attributes, (2) facilitates NOT relations on
attributes, and (3) is more interpretable. The joint likelihood is written terms of the marginals
using the CI relation and the causal factorization as,
p(X) P(C)
p(X C.)((C.) x(X)
CX
II
(JM relation)
Note that CI relation crucial for JM relation to hold. We sample from joint likelihood using the score of LHS of JM relation, referred to joint sampling in $ 3 Similarly, score of RHS of JM relation, referred to as marginal sampling in $ 3 If the learned generative model satisfies the JM relation, then there should not be any difference in the CS between joint sampling and marginal sampling. However, in Tab. [ we see a drop in CS, implying JM relation is satisfied in the learned model.
sample using the
JM relation must hold in the learned generative model if CI relation is true in the learned generative model. Therefore, we check if the CI relation holds in the generative model by measuring JSD between LHS and RHS of CI relation as shown in Eq. (5) in the main paper. The results Tab. [ confirm that the CI relation does not hold in the learned model. This is significant finding since existing works (Liu et al., 2022 Nie et al. 2021) blindly trust the model to satisfy CI relation, leading to severe performance drop when the training support non-un non-uniform or partial.
The CI relation is violated in the learned model because the standard training objective is not suitable for compositionality, as does not account for the incorrect Ptrain (X C.). The proof is detailed in the next section App. B.2 Therefore, we proposed CoIND to ensure the JM relation was satisfied by explicitly learning the marginal likelihood according to the causal factorization.
B.2 STANDARD DIFFUSION MODEL OBJECTIVE IS NOT SUITABLE FOR LOGICAL COMPOSITIONALITY
This section proves that the violation conditional independence in diffusion models is due to learning incorrect marginals, Purain
C2) under C2 V.C. Cj- We leverage the causal invariance
16

---

Published as a conference paper at ICLR 2025
property: Puain (X () Pirue [X] C), where Pusin is the training distribution and Pire is the true
underlying distribution.
Consider the training objective of the score-based models in classifier free formulation Eq. 12 For the classifier-free free guidance, a single model (x, C') effectively trained to match the score of all subsets of attribute distributions. Therefore, the effective formulation for classifier-free guidance can be written as,
Lecore
Es
I po cs) Vx log Prrain (x cs)l
(19)
where S is the power set of attributes.
From the properties Fisher divergence. Lecore iff po (X cg) Purin (X cs), AS. In the case of marginals, Pe(X C.) i.. S = {Ci} for some 1 S n,
(e C₁) Purin (X C2)
Perain X
Cj P-sii (
Σ Puue C₁ C-i) (C Ci)
4 Pinue [X Ci P (C Pirne (X Ci)
C-1
pa X C:) + Pime C.)
(20)
Where C_, m Cj, which every attribute except C, Therefore, the objective of the score- 374
based models is maximize the likelihood of the marginals of training data and not the true marginal distribution, which different from the training distribution when C. M. Cj-
B.3 STEP-BY-STEP DERIVATION OF COIND IN S
The objective is to train the model by explicitly modeling the joint likelihood following the causal factorization from Eq. (M relation). The minimization for this objective can be written
Po
Po C.)po
Leomp = W/2 p X
C)2
(21)
Po
Pe (x(
X)
where W2 2- 2-Wasserstein distance. Applying the triangle inequality to Eq.
21 we have,
Coump W2 (p(X Co, C)) W/ (X C)2 Po (X) (C)
II Po X (C))o(C;)
(22)
(X
Distribution matching
Conditional Independence
Kwon et al. 2022 showed that under some conditions, the Wasserstein distance between Po( (X) ), go (X) is upper bounded by the square root of the score-matching objective. Rewriting
Equation 16 from (Kwon et al. 2022)
W/ (Po(X), (x))) s K KV (x) [|| log Po (X)
V log XX)
(23)
Distribution matching Following Eq. 23 result, the first term in Eq. 22 replacing po as p and Yo as Pe will result in
W2 (p(X Co,po(X (X (C) S Ki (x)( [||Vx logp(X C* Vx log (X X)
K1 v Lxcur
(24)
17

---

Published as a conference paper at ICLR 2025
Conditional Independence as pe and (x( as PC(C)
Following Eq.
23 po( (p
result, the second term in Eq. 22
replacing Po
(x(
II.
xx
Po [
pa [ (C)
W22 po C)2
Co(C)
po(X)
Po X)
II Po X
Ci) (C;; (x
E Vx log Po (X CC Vx log
po (C)
Further simplifying and incorporating Vx po Pm (C) (C;) = 0 and Vx log po (C) = 0 will result
pol X [C),
n
po X
(C:)
W2
Po
po(C) C)
Px(X)
I 12
V log p X |C) Vx Oogpo(X) Σ
Vx lo Cc) Vx log
Cat
= K2V CCI
(25)
Substituting Eq.
24
Eq.
(5
Eq. [22) will result in our final learning objective
Compp I K1 KIv Lucore K2v Car
(26)
where K1, K2 positive constants, i.e., the conditional independence objective Lai is incorporated alongside the existing score- matching loss Lecere-
Note that Eq. (25 is the Fisher divergence between the joint pe (X C*) and the causal factorization
Po(X]
II:
pa C. (C)pa((.) from Eq. (JM relationo. From the properties Fisher divergence (aánchez-
Pa
()
Moreno et al.
2012 3 La = 0 iff po
(x
C)
=
pa (X) CC
II
Po(XIC.)poCC) and further implying,
X)
11, PPl(C X Puran (C X)
When Leomp = 0: PX C) = Purin (X C) P C). and II, Po X)= = Prain (C X). This implies that the learned marginals obey the causal independence relations from the data- generation process, leading to more accurate marginals.
C PRACTICAL CONSIDERATIONS
To facilitate scalability and numerical stability for optimization, we introduce two approximations to the upper bound of our objective function Eq. (10).
C.1 SCALABILITY OF Lei
key computational challenge posed by Eq. linearly with the number of attributes. The Eq. (9) derived from conditional independence formu-
9 is that the number of model evaluations grows
lation as follows:
(C X) II (C X).
(27)
By applying Bayes' theorem to all terms, obtain,
po X C))o(C) = II (x CC))o(Ca)
(28)
(X)
(X
Note that this formulation is equal the causal factorization. From this, by applying logarithm and differentiating w.r.t. X, we derive the score formulation.
V log (X CC Vx log Epo(X C.) Vx (X)
(29)
18

---

Published as a conference paper at ICLR 2025
The Ly norm of the difference between LHS and RHS of the objective in Eq. forms our La objective.
29 is given by. which
La
VV log
|C) V log Σ po x C.) - Vx log po (og) (x)
(30)
Due to the Eg in the equation, the number of model evaluations grows linearly with the number of attributes (n). This (n) computational complexity hinders the approach's 's applicability at scale.
To address this, we leverage the results of (Hammond & Sun) 2006), which shows conditional in- dependence is equivalent to pairwise independence under large to reduce the complexity to (1)) in expectation. This allows for a significant improvement in scalability while maintaining computa- tional efficiency. Using this result, modify Eq. (27) to:
po CC(C Cj X) = Pe(C; ; X))a(C) X). Vi,j
Accordingly, we can simplify the loss function for conditional independence as follows:
La Ex B,, Vx Poogoooo Cj. Ck) log po (X X Cý) log po (X)Ck) Nogro(X))]i
(31)
In score-based models, which are typically neural networks, the final objective is given as:
La = Ee
Cj, Cx) - se ( C,) (X( - sg [X 2)
(32)
where s-(-) = Vx (.() is the score the distribution modeled by the neural network. We leverage classifier-free guidance to train the conditional score (X, C.) by setting Cs Ø for all + i, and likewise for so (X, (xC,, Cj), we set C = Ø for all k {1,5)-
C.2 SIMPLIFICATION OF THEORETICAL LOsS
In Eq. [10,, we showed that the 2- 2-sassertein distance between the true joint distribution p(X (C) and the causal factorization in terms of the marginals p(X C:) is upper bounded by the weighted sum of the square roots of Lucure and Lai as Leump
V + Kav In practice, however, we minimized simple weighted sum of Lacone and LC1, given by Limal Lw + XLeı as shown Eq. (IT) instead Eq. (10p.
used Eq. (II) to avoid the instability caused by larger gradi- ent magnitudes (due to the square root). Eq. (1] also provided the following practical advantages: (1) the simplicity of the loss function that made hyperparameter tuning easier, and (2) the similarity of Eq. (IT) to the loss functions of pre-trained diffusion models allowing us reuse existing hyper- parameter settings from these models. We did not observe any significant difference in conclusion between the models trained Eq. (T0) and Eq. (11) shown in Tabs. and Both approaches significantly outperformed the baselines.
Support
Method
JSD L
A (CS) 1
Color (CS)f
Digit (CS)
LACE
96.40
92.56
83.67
Composed GLIDE
0.16
98.15
99.30
81.64
Uniform
Theoretical ColND Eq. 100
0.12
98.44
100.00
81.25
CoIND (A = 0.2)
0.14
99.73
99.32
84.94
CoIND 1.0)
0.10
99.99
99.33
89.60
LACE
82.61
65.16
69.51
Composed GLIDE
0.30
86.10
81.61
70.44
Non-uniform
Theoretical CoIND Eq. 10
0.17
96.88
93.75
72.66
CoIND (A = 1.0)
0.15
99.95
92.41
84.98
LACE
10.85
9.03
28.24
Composed GLIDE
2.75
7.40
5.09
33.86
Partial
Theoretical ColND Eq.
10
1.11
23.44
64.84
53.12
CoIND (A = 1.0)
1.17
52.38
53.28
52.59
Table 3: Results on Colored MNIST to directly minimize the upper bound (K1 1, K₂ 0.1)
19

---

available code and checkpoints at Published as a conference paper at ICLR 2025
Support
Method
JSD 4
Composition
Composition
R² f
CS f
R²4
CSt
LACE Composed GLIDE
0.97
91.19
0.85
50.00
0.302
0.94
83.75
0.91
48.43
Uniform
Theoretical CoIND Eq. 10
0.270
0.98
92.19
0.92
64.06
COIND (A = 1.0)
0.215
0.98
95.31
0.92
55.46
LACE
0.88
62.07
0.70
30.10
Composed GLIDE
0.503
0.86
51.56
0.61
34.63
Partial
Theoretical COIND Eq. 10
0.450
0.93
78.13
0.88
51.56
ColND (A 1.0)
0.287
0.97
91.10
0.92
53.90
Table 4: Results on Shapes3D with the objective of directly minimizing the upper bound Eq. (00
(K1 1, K2 0.1)
C.3 CHOICE OF HYPERPARAMETER À
Effeet of À on the Learned Conditional Independence. CoIND enforces conditional inde- pendence between the marginals of the attributes learned by the model by minimizing Lei defined Eq. (32). Here, we investigate the effect of LC on the effectiveness of logical composition- ality by varying its strength through À in Eq. I Figure 8 plots JSD and CS (A) as functions of Å for models trained on the Colored MNIST dataset under the diagonal partial support setting. When À O, training relies solely the score matching loss, re- sulting higher conditional dependence between C₁ X. As A increases, CS improves since ensuring conditional independence between the marginals also encourages more accurate learning of the true marginals. However, when takes large values, the model learns truly independent conditional distribution C X but effec- tively ignores the input compositions and generates samples based solely on the prior distribution pe X). As result, CS drops.
2.05
1.25
18.52
10
The value for the hyperparameter ents from the matching objective Lycore and the conditional independence objective L.r are balanced magnitude. One way to choose À is by training a vanilla diffusion model and setting À = Lycore We used two values for À in our experiments and noticed
chosen such that the gradi-
Figure 8: Effect of X logi- cal compositionality under agonal partial support the Colored MNIST dataset.
that they gave similar results, indicating that the approach was stable for various values of X.
D EXPERIMENT DETAILS
In this section, we outline the high-level design choices of our approach. We provide full implementation details in our publicly
D.1 COIND ALGORITHM
To compute pairwise independence in scalable fashion, we randomly select two attributes, and j. for sample in the batch and enforce independence between them. As the score in Eq. [55 is given (x(x)))
by VI-a. The final equation for enforcing La will be:
1
|xo t, ct) cx c²) co(x,, (.)) cg(x,, t, c ||
Lei 1 Ci
We follow Ho et al. (2000) to weight the term by 1 ā, This results in an algorithm for COIND, requiring only few modifications of lines from (Ho & Salimans, 2022), highlighted below.
20

---

Published as a conference paper at ICLR 2025
Algorithm ColND Training
1: repeat
2: (c, ~ Puain (c,in
3: Ct < Ø with probability Pancond
D Set element of index, i.e, ce to 2 with PuncondVk N]
probability
4: Uniform((0. ... N)).j Uniform (0,...., Un.form(O....... N) (i)
Select random attrihute indices
Uniform ,T)....T))).....
I)..
7: I, voxxo v1 EGE
&:
c',c', c' tc.
c.d + {ar = Ø kk {i.j)). c² < 10: Lci |x,, )²) + (x.x,,,,))) (t, ct.) eo(x,, 112
9:
c' (ck = il.
11: Take gradient descent step one
Vellie - (x,,,, c)ll +\bei
12: until converged
Practical Implementation In our experiments, we have used Puncond = 0.2 and for Shapes31)
instead of enforcing C₁ 11. Cj X, for all i.j enforcing C, IL. C-i X for all have led slightly better results.
D.2 DETAILS OF LOGICAL COMPOSITIONALIITY TASK
We designed the following task to evaluate two primitive logical compositions. (1) AND Composi- tion A, (2) NOT Composition
AND Composition To evaluate the A composition, we apply the A operation over all the attributes to generate respective image. Consider an image from the Shapes3D dataset (see Figure Fig. 99) The image is generated by some function, ʃ. with the input e following image can be queried using the logical expression C1 6 ... Ca 11. We follow
4 6 2
11 The
Equation Eq. (T) to sample from the above logical composition. To reiterate, for the A composition
task on Shapes3D, the sampling equation given by Xxpe X C1 = 6 -A...A Co 11):
Vx log Po (X) +
Σ
V log po X C.) - Vx logro(X))
(33)
Similarly, to evaluate the AND composition for the Colored MNIST dataset, we perform the ^ operation over digit C₁ and color C2.
To evaluate the compositions, the image is queried as an AND on all the attributes except the object attribute, which is queried by its negation. For example, consider the same im- age from Figure Fig. g where the object sphere (C; = 2) can be ex- pressed Cs 00 v v 3]. because the object class can only take four possible values. Therefore, the same image can be described as G A A Cs -10 v 1 3.... Cs 11. The only possi- ble generation that meets these criteria is the image (Fig. [9) displayed as expected.
NOT Composition
The sampling equation for a test image with attributes C1, C2, C3, C4, Cs Cs be written C1 A C2 A C2 A = C5 = -10 v V 3] A C. = 11. Following Eq. 2 the
Figure 9: Image from
Shapes3d with attributes
sampling equation is written as follows:
c = [6, 4,6,,,,,,,,11
Vx log Po C 6) V log (X C2 8) + Vx log (
Vx log X C4 = 6) Vx log x Co 11) - Vx log X = 0)
x log X C3 = 1) - Vx log ( =) - Vx log po (X)
Similarly, for Colored MNIST, we perform two kinds of negation operations: one on digit and another color. In Section have shown negation color A -(Green Pink), along with
21

---

Published as a conference paper at ICLR 2025
its sampling equation. A similar logic can be followed for negation on color; example of negation on digit is 13 4] A Pink.
For A and -, evaluations support for Shapes3D and under diagonal partial support for Colored MNIST. This approach allows us explore how effectively the model handles logical operations through unseen image generation. Additionally, we evaluate compositions observed during training with less frequency under non- uniform support.
strictly restricted to unseen compositions under orthogonal partial
D.3 TRAINING DETAILS, ARCHITECTURE, AND SAMPLING
Training Composed GLIDE & COIND We train the diffusion model using the DDPM noise scheduler. The model architecture and hyperparameters used for all experiments are detailed in Tab.
The LACE method involves training multiple energy-based -based models for each attribute and sampling according to logical compositional equations. However, we use score-base based models instead. We follow the architecture outlined in Tab. 5 for each attribute to train multiple score-based models. For Colored MNIST, which has two attributes, we create two models one for each attribute-use using the same architecture as other methods, effectively doubling the model size. Similarly, for Shapes3D with six attributes, Channels for each attribute model to fit these into memory while keeping all other hyperparameters consistent. Since train single model per attribute, do not match the joint distribution, preventing us from evaluating and measuring the JSD.
Training LACE
develop six models. We reduce the Block Out
Sampling To generate samples for given logical composition, we sample from equations from App. D.2 using DDIM (Song et al. 2021a) with 100 steps.
Colored MNIST
Shapes3D
Hyperparameter
CoIND & Composed GLIDE
LACE
COIND & Composed GLIDE
LACE
Optimizer
AdamW
AdamW
AdamW
AdamW
Learning Rate
2.0 x 10
2.0 10
2.0 x 10
2.0 10
Num Truining Steps
50000
100000
100000
100000
Train Noise Scheduler
DDPM
DDPM
DDPM
DDPM
Train Noise Schedule
Linear
Linear
Linear
Linear
Train Noise Steps
1000
1000
1000
1000
Sampling Noise Schedule
DDIM
DDIM
DDIM
DDIM
Sampling Steps
150
100
Model
U-Net 2 Lincar
U-Net
U-Net 2 Lincar
U-Net 2
Layers per block
Beta Schedule
Linear
Lincar
Sumple Size
28x3x3
28x3x3
64x3x3
64x3x3
Block Out Channels
12,
156.112,168
15..112.166.224
156,112,168]
Dropout Rate
0.1
0.1
0.1
0.1
Attention Head Dimension
8
8
8
Norm Num Groups
8
Number of Parameters
8.2M
8.2M
17.2M
8.2M 6
Table 5: Hyperparameters for Colored MNIST and Shapes3D used by CoIND, Composed GLIDE, and LACE
CelebA To generate CelebA images, we scale the image size to 128 x 128. We use the latent encoder of Stable Diffusion 3 (SD3) to encode the images to latent space and perform diffusion in the latent space. The architecture similar to the Colored MNIST and Shapes3D, except that Block out Channels are scaled as [224, 448, 672, 896]. We use learning rate of 1.0 10-4 and train the model for 500,000 steps on one A6000 GPU.
FID Measure To evaluate both the generation quality and how well the generated samples align with the natural distribution of 'smiling male celebrities', we use the FID metric (Seitzer) 2020). Notably, calculate the FID score specifically the subset of 'smiling male celebrities, primary objective to assess the model's ability to generate these unseen compositions. We generate 10000 samples to evaluate FID.
22

---

Published as a conference paper at ICLR 2025
T21: Finetuning SDv1.5 We finetune SDv1.5 with the data constructed from CelebA, where the labels are converted to text. For example, a label of (male=1, smiling=1) 1) is converted to a "photo of smiling male celebrity."
D.4 ANALYTICAL FORMS OF SUPPORT SETTINGS
Below are the analytical expressions for the densities under the various support settings that considered in the paper. Let n; be the number categories for the attribute C.. For non-uniform
and diagonal partial support settings, assume that nı n = n. Vi,j, # j.
Uniform setting: p(C, = ci) + and p(C; = c1, C, = = = p(C; = c1 cI)p(CC = =2) n,yy"
c 0 or C2
Orthogonal support setting: p(C; = c1, Cj = 02) =
14 n
0,
otherwise
a
02 S S C2 +1
Non-uniform setting: p(C, = C1, Cj = 02)
where 1 abbsa
b,
otherwise
5 02
Diagonal partial support setting: p(C; Cj = C2) =
In-I 0,
otherwise
D.5 DATASETS
Colored MNIST Dataset In Section we introduced the Colored MNIST dataset. Here, we will detail the dataset generation process. We selected 10 visually distinct colors taking the value C2 E [0. 9]. The dataset is constructed by coloring the grayscale images from MNIST by converting them into three channels and applying one of the ten colors to non-zero grayscale values.
The training data is composed of three types of support:
Uniform Support: A digit and a color are randomly selected to create an image.
Diagonal Partial Support: A digit is selected, and during training, it is only assigned one of two colors, C2 {d, + 13, except for which only takes one color. This creates dataset where compositions observed during training are along the diagonal of the C space, meaning each digit seen only with its corresponding colors.
Non-uniform Support: All compositions are observed, but combining digit and its cor- responding colors occurs with higher probability (0.5). The remaining color space is distributed evenly among other colors, resulting approximately a 0.25 probability for each corresponding color and 0.0625 probability for each remaining color.
Shapes3D Full support for Shapes3ID consists of all samples from the dataset. For orthogonal support, we use the composition split of Shapes3I as described by [Schott al.), whose code publicly available
CelebA consists of 40 attributes, from which we select the "smiling" and "male" at- tributes. We train generative models on all combinations of these attributes except (smiling=l, male=1), resulting in an orthogonal partial support.
CelebA
D.6 CONFORMITY SCORE (CS)
In Section we described the Conformity Score (CS) to quantify the accuracy the generation per the prompt. To measure the CS, train single ResNet-18 (He al. (016)) classifier with multiple classification heads, one corresponding to each attribute, and trained on the full support. This classifier estimates the attributes in the generated image, x, and extracts these attributes as
httsps://mokole..com/p/s wwwp://m//plett.e.eeeee html
23

---

Published as a conference paper at ICLR 2025
G.]. (xx) = (c,..., These attributes matched against the input prompt that generated the image to obtain accuracy.
To explain further, for example, if the prompt is generate "4 A -[Green V PinkJ", the generated sample will have CS of if &j = 4 and 62 & {Green, Pink]. We average this across all the prompts in the test set, which determines the CS for given task.
The effectiveness of the classifier in predicting the attributes is reported Table I0
Feature
Attributes
Possible Values
Accuracy
Feature
Attributes
Possible Values
Accuracy
C1
Digit
0-9
98.93 100
C1
Gender
(0.1)
98.2
C2
color
10 values
C2
Smile
[0.1)
92.1
(a) Colored MNIST Dataset
(b) CelebA Dataset
Feature
Attributes
Possible Values
Accuracy
C
floor hue
10 values in [O, 11
C2
wall hue
10 values in [0. 1]
100
C3
object hue
10 values in [O, 11
100
C4
scale
8 values in [0. 1]
100
C.
shape
4 values in [0-3]
100
C.
orientation
15 values in [-30, 30]
100
(c) Shapes3D Dataset
Figure 10: Independent attribute, their possible values, and the classifier accuracy in estimating them for different datasets
D.7 COMPUTING JSD
We are interested in understanding the causal structure learned by diffusion models. Specifically, we aim to determine whether the learned model captures the conditional independence between attributes, allowing them vary independently. This raises the question: Do diffusion models learn the conditional independence between attributes? The conditional independence is defined by:
(C C2 x) = pe X p)pe(C X)
(34)
We aim to measure the violation of this equality using the Jensen-Shamin divergence (JSD) to quantify the divergence between two probability distributions:
JSD Epan [Dis (pe (C X) |l po (C) X))
(35)
The joint distribution, (a Cy X). and the marginal distributions, (C X) and PC X), evaluated all possible values that C₁ and Cj take obtain the probability mass function (pmf). The probability for each value is calculated using Equation Eq. (18) for the joint distribution and Equation Eq. (7) for the marginals.
Practical Implementation For the diffusion model with multiple attributes, the violation in con- ditional mutual independence should be calculated using all subset distributions. However, we focus on pairwise independence. We further approximate this in experiments by computing JSD be- tween the first two attributes, C1 and C2. We have observed that computing JSD between any attribute pair does not change examples' conclusion.
D.8 MEASURING DIVERSITY IN ATTRIBUTES
To achieve explicit control over certain attributes during the generation process, these attributes must vary independently. Therefore, an ideal generative model must able to produce samples where all except the controlled attributes take diverse values. This diversity can be measured by the en- tropy of the uncontrolled attributes in the generated samples, where higher entropy suggests greater diversity. Therefore, the accurate generation of controlled and diverse uncontrolled attributes for the given the underlying data distribution has uniform attributes indicates that the model has suc- cessfully learned when the underlying attribute distribution is uniform. In contrast, for non-uniform
24

---

Published as a conference paper at ICLR 2025
distributions such as the Gaussian example discussed in App. G. no longer applies, and minimum KL divergence between the model and the true distribution be- comes the appropriate measure. Under uniform attribute assumption, however, the KL divergence essentially reduces to maximum entropy.
simple diversity argument
For example, consider the generation of colored MNIST digits. In this case, controllability means that that the model has learned that digit and color attributes are independent. When prompted to generate specific digit (controlled attribute), the model should generate this digit in all possible colors (uncontrolled attribute) with equal likelihood, implying maximum entropy for the color attribute and diverse generation. We measure this entropy by generating samples x' po c1 = 4) and passing them through a near pear-peerect classifier to obtain the color predictions p( = p()2(x")) The diversity is then quantified as: H ( [log2 p())]
Ensuring diversity through explicit control has applications in bias detection and mitigation in gen- crative models. For example, a biased model may generate images of predominantly male doctors when asked to generate images of "doctors". Ensuring diversity in uncontrolled attributes like gen- der or race can limit such biases.
E COIND FOR FACE IMAGE GENERATION
In $ we demonstrated that COIND outperforms haseline methods on the unseen logical com- positionality task using synthetic datasets. In App. E. we showcase the success of CoIND in generating face images from the CelebA dataset (Liu et al. 2015), where CoIND demonstrates su- perior control over attributes compared to the baseline. CoIND also allows us to adjust the strength of various attributes and thus provides more fine-grained control over the compositional attributes, shown in App. E.2 Finally, in App. E. we extend CoIND to text-to-image (T2I) models widely used in practice to generate face images by providing the desired attributes as logical expressions of text prompts.
Problem Setup We choose the CelebA dataset to evaluate COIND's ability to generate real-world images. We choose the binary attributes "smiling" and "gender" the attributes wish control. During training, all combinations of these attributes except gender = "male" and smiling "true" are observed, similar to the orthogonal support shown in Fig. 3 During inference, the model is tasked to generate images with the attribute combination gender "male" and smiling "true", which was not observed during training.
Metries Similar to the experiments on the synthetic image datasets in $ we assess the accuracy of the generation w.r.. the input desired attribute combination CS (conformity score). We also measure the violation of the learned conditional independence using JSD. In addition to CS, we compute FID (Fréchet inception distance) between the generated images and the real samples in the CelebA dataset where gender "male" and smiling "true". A lower FID implies that the distribution of generated samples is closer to the real distribution of the images in the validation dataset.
E COIND CAN SUCCESSFULLY GENERATE REAL- WORLD FACE IMAGES
Tab. [ shows the quantitative results of COIND and Composed GLIDE trained from scratch in the tasks of joint sampling and A composition. Similar to our observations from previous experiments, COIND achieves better CS in both tasks by learning accurate marginals as demonstrated by lower JSD. When sampled from the joint likelihood, CoIND achieves nearly 4x improvement in CS over the baseline.
E.2 CoIND PROVIDES FINE-GRAINED CONTROL OVER ATTRIBUTES
So far, studied the capabilities of COIND to dictate the presence and absence of attributes in the task of controllable image generation. However, there are applications where we desire fine- ane-ggained control over the attributes. Specifically, generated sample. We can mathematically formulate this task by revisiting the formulation of logical expressions of attributes in terms of the score functions of marginal likelihood. As example, the
may want control the amount of each attribute in the
25

---

Published as a conference paper at ICLR 2025
CoIND
Composed GLIDE
LACE
The am of inereases y increases
Figure 11: By adjusting 7. COIND allows us the vary the amount of "smile" in the generated images. However, Composed GLIDE associates the smile attribute with the gender attribute due to their association in the training data. Hence, the images generated by Composed GLIDE contain gender-specific attributes such as long hair and earrings.
A operation can be written as,
Vx log pe X C1 C2) = Vx lo (1) + Vx log C2) Vx (X)
Here, to adjust the amount of attribute added to the generated sample, we can weigh the score functions using some scalar 7, as follows,
Vx po X C1) - V lo C2) - Vx (x)
(36)
where y controls for the amount of C2 attribute.
Methods
LACE
Composed GLIDE
CoInD
40
100
30
20
10
2
2
4
6
7
(a) Variation of FID with
(b) Variation CS with y
Figure 12: Effect of FID and CS: Varying the amount of smile in generated image through does not affect the FID of CoIND. However, the smiles the generated images become more apparent, leading easier detection by the smile classifier and improved CS.
Fig. I1 shows the effect of increasing 7 to adjust the amount of smiling the generated image. Ide- ally. we expect increasing to increase the amount of smiling without affecting the gender attribute. When 7 (top row), both CoIND and Composed GLIDE generate images of men who are not smiling. As y increases, we notice that the samples generated by CoIND show an increase in the amount smiling, going from a short smile to wider smile to one where teeth are visible. Note that the training dataset did not include any images of smiling men or fine-grained annotations for the amount of smiling in each image. This conclusion strengthened by Fig. 12b that shows an increase in CS when increases. CS increases when it is easier for the smile classifier to detect the smile. CoIND provides this fine- fine-grained control over the smiling attribute without any effect on the realism of the images, as shown by the minimal changes in FID in Fig. 12a
26

---

Published as a conference paper at ICLR 2025
In contrast, the images generated by Composed GLIDE show increase in the amount of smiling while adding gender-specife attributes such as long hair and makeup. We conclude that, by strictly enforcing conditional independence loss between the attributes, CoIND provides fine-grained control over the attributes, allowing us to adjust the intensity of the attribute the image without additional training. As shown in Tab. _ CoIND) outperforms the baselines for generating unseen compositions. Tuning further improves the generation.
E.3 FINETUNING T2I MODELS WITH CoIND IMPROVES LOGICAL COMPOSITIONALITY
"smiling male"
"smiling" AND "male"
"smiling"" NOT "fomale"
Composed GLIDE
COIND
Samples generated after fine-tuning SDv1.5 on CelebA. The first row shows images gen- erated by SDv1.5 fine-tuned on CelebA, while the second row shows images generated by SDv1.5 fine-tuned with COIND. Columns indicate samples generated from the respective prompts indicated above.
Figure 13:
We proposed CoIND to improve control over the attributes an image through logical expressions of these attributes. Since larger pre-trained diffusion models such as Stable Diffusion (Rombachi jet al. 2022) have become more accessible, we seek to incorporate the benefits of CoIND these models. This section shows that text-to-image (T2I) models
be fine-tuned to generate images using logical expressions of text prompts. Specifically, we use Stable Diffusion v1.5 (SDv1.5) to generate face images from the CelebA dataset where smiling and gender attributes can be controlled. We consider both joint and marginal sampling, similar to our case study in 3 For joint sampling, we provide SDv1.5 with the prompt "Photo of smiling male celebrity". In the marginal sampling, we provide the values for smiling and gender attributes using separate prompts "Photo of smiling celebrity" A "Photo of male celebrity". Then, we sample from these marginal likelihoods resulting from these prompts following Eq. (I. To evaluate capabilities, we use the prompts "Photo of smiling celebrity" "Photo female celebrity" and follow Eq.
Joint
A Composition
Composition
Support
Method
JSD
CS
FID 4
CS f
FID
CS f
FID +
Composed GLIDE CoIND (A 1.0)
0.57
56.57
58.31
14.19
73.53
11.02 18.80
115.95
Orthogonal
0.37
58.57
58.19
49.15
61.16
86.31
Table 6: Results on SDv1.5 fine-tuning, CoIND outperforms the baseline on all the metrics.
Discussion
1. In Tab. 6 COIND improves performance across all metrics achieving 3.46x and 2x im- provement in CS over Composed GLIDE in A and - composition tasks. The images generated by COIND have better FID than those from the baseline.
2. Visual inspection of the generated samples for the same random seed provides insights into how Composed GLIDE and ColND perceive the prompts. Images in columns 1. 3, and 5 of Fig. [ewree generated with the same random seed. Similarly, those in columns 2 and 4 share the random seed. We note the following observations:
27

---

Published as a conference paper at ICLR 2025
Both Composed GLIDE and CoIND generated images with the desired attributes when sampled from the joint likelihood using "photo of a smiling male celebrity". The images generated by these models from the same random seed were also visually similar. This shows that both models can aptly set attributes the generated images and have identical stochastic profiles, leading unspecified attributes that assume similar values.
When the attributes were passed as the expression "smiling" "male", COIND gen- erated images that were visually similar to those with matching random seeds generated from joint sampling. This implies that CoIND learned accurate marginals that help it to correctly model the joint likelihood.
When tasked with generating images for "smiling" A "male", Composed GLIDE gener- ated images of smiling persons with gender-specific attributes such as thinner eyebrows, commonly seen in photos of female celebrities. These gender-specific features increase when the task is to generate images of "smiling" - "female". In contrast, COIND gen- erates images of smiling celebrities while adding attributes such beard. Thus, conclude that CoIND offers better control over the desired attributes without affecting correlated attributes.
F DISCUSSION ON COIND
F.1 CONNECTION TO COMPOSITIONAL. GENERATION FROM FIRST PRINCIPLES
Compositional generation from first principles Wiedemer et al. (2024) have shown that restricting the function certain compositional form will perform better than a single large model. In this section, we show that, by enforcing conditional independence, we restrict the function to encourage compositionality.
Let C1, C2...., Cn be independent components such that C1 C...... On. E R. Consider an injective function ʃ IR" Red defined by [cc) æ. If the components, e conditionally independent given x the cumulative functions, F must satisfy the following constraint:
(G..,)..... = II (c))
(37)
Fc,.c,,....xxx.(" (x) = inf(es, Cn...., en
F(ch, Cn) 2 2), where F-1
X= X== is
Ca
generalized inverse distribution function.
((.,,,....., = (So F-1 e. G.I=x Fa(x==((ii)
(f F-1
S...,....II... oe) Σ log Fc.1x=x(())
=( Σ di(())
Therefore, we are restricting f take a certain functional form. However, it is difficult show that the data generating process, f. meets the rank condition on the Jacobian for the sufficient sup- port assumption Wiedemer et al. (2024), which is also the limitation discussed their approach. Therefore, we cannot provide guarantees. However, this section provides a functional perspective of CoIND.
F.2 2D GAUSSIAN: CLOSED-FORM ANALYSIS OF CoIND
In this section, we derive closed-form expressions for the score functions underlying method and demonstrate how COIND leverages conditional independence constraints to generate the true data distribution.
Data Generation Process
We consider data generated from two independent attributes, C1 and C2, which are binary variables taking values in {-1, +1}. The observed variable X is defined as:
X f(G) f(C2),
(38)
28

---

Published as a conference paper at ICLR 2025
True underlying data distribution
Training data Orthog- onal Support
Conditional distribu-
Conditional distri- learned by
(a)
(b)
(c) tion learned by vamilla
(d) bution CoIND
diffusion objective
Figure 14: CoIND respects underlying independence conditions thereby generating true data distri- bution (d).
where
f(c) = = de, N(0,1,
Thus, f(Ci) produces Gaussian mixture along the r-axis with means -1 and +1, and similarly f(C2) produces a mixture along the y-axis with means at -1 and +1 (see blue plot on the axis of Figure 14ap. The combination yields a two-dimensional Gaussian mixture (Figure [4a).
Training Setup and Orthogonal Support For training, we assume orthogonal support where
only the following combinations of C2) are observed: ((-1,-1), (-1,+1), (+], The model is then tasked with generating samples from the unseen composition +1). Recall that our assumptions (see Section 2) are satisfied: C1 and C2 independently generate X, and possible values for each attribute are observed at least once during training.
Score Function Decomposition Let (+),+1(x) denote the score corresponding to p(x | C1 +1, C2 = +1), and let $1.0 (x: denote the marginal score p(z C1 +1) (with a similar definition for 1 (r))). Leveraging Eq. 8 541,+1 (x) is decomposed as follows:
$+1,+1(2) $1,2 (2,1 $2 (z).
(39)
where $2,2 (z) is the score of the training data and not full data.
For example, when training on the observed combination (-1.-1) the score function of the $1,0, $2,1, is Gaussian, and written in closed form as
141.-1
$1,2(x)
x²
(0.1 = 1+11-11
(40)
g²
In contrast, the score of (z), is the mixture (over the three training components) given as:
E:N(x: pli (2
80 (x)
(41)
E,N(1:14,
However, when using Langevin dynamics for sampling (see Eq. 13) ) the naive combination in
Eq. (39) produces an incorrect conditional distribution (Figure 14ch Specifically, the generated dis-
tribution shows spurious red blob between the (+1,-1) and I-I, +1) modes rather than proper Gaussian centered at (+1,+1) This shows that Diffusion models interpolate between the modes, rather than following underlying conditional independence and generalizing to unseen modes.
Correcting with Conditional Independence Constraints Instead of modeling $1,2 (x) directly, COIND learns the joint scores for the three observed combinations:
841, 141,-1 1-1,-
8-1,+1
29

---

Published as a conference paper at ICLR 2025
These are then combined under the assumption of pairwise conditional independence to infer the score for the unseen composition:
s 5-1,-1(2) $+1,0 (2) $z)+1 $2,2(0),
$41,-1(2) = S41,0 + $2,0(2))
8-1,+( (2) = ( (1)+ s0.0(2)
(42)
which leads to the following expression for the unseen (+1++1) composition:
(+)++1(2) = (+),0( 50 (x) (
= -841,-1 (2) + --1,+1 (r) - 8 8-1,-1 (z)
|+1,-1 1,+1 11-1.11
(43)
The derivation above shows that CoIND effectively enforces conditional independence constraints to generate the unseen data distribution. This analysis underscores the necessity of incorporating conditional independence constraints into diffusion models to faithfully reproduce the target distri- bution, particularly when extrapolating to unseen compositions.
F.3 EXTENSION TO GAUSSIAN SOURCE FLOW MODEL.
Diffusion models can be viewed specific case of flow-based models where: (I) the source dis- tribution is Gaussian, and (2) the forward process follows predetermined noise schedule (Lipman et al. 2024). Can we reformulate CoIND in terms of velocity rather than score, thereby generaliz- ing to accommodate arbitrary source distributions and schedules? When the source distribution gaussian, score and velocity related by affine transformation as detailed in Tab. I of (Lipman ct al. 2024).
(x, C1, C2) axx br, C1,
(44)
replacing (-) into Eq. 332
La = Bxx Bullls Ck) - s, Cj) m (ar Ck) (x)))
Exx. [6 , Cj. Ck) - $ (z. Cj) (x, Ct) + (z))
However we can ignore b7. weighting for the time step t.
Ca = =Eppxx.c),.-co.0111... [Il Cj, Ck) w (x, C2) (z) C*) + uz)
(45)
Therefore, if the source distribution is gaussian and for any arbitrary noise schedule, constraint in score translates directly to velocity constraint as given as Eq. (45).
F.4 COMPOSITIONAL VS MONOLITHIC MODELS
Our findings echo the prior observations (Du & Kaelbling, 2024 that composite models consisting of separate diffusion models trained on individual factors (e.g., LACE) demonstrate better A compo- sitionality under partial support than sampling from factorized distributions learned by monolithic models (e.g., Composed GLIDE), However, we found that monolithic models can be significantly improved by enforeing the conditional independence constraints necessary for enabling logical com- positionality. For instance, CoIND achieved a 2.4x better CS on Colored MNIST with diagonal partial support and a 1.4x improvement on orthogonal partial support on Shapes3D compared to LACE.
F.5 LIMITATIONS
This paper considered compositions of closed set of attributes. As such, CoIND requires pre- defined attributes and access to data labeled with the corresponding attributes. Moreover, CoIND must be enforced during training, which requires retraining the model whenever the attribute space changes to include additional values. Instead, state-of-the-art generative models seek to operate without pre-defined attributes or labeled data and generate open-set compositions. Despite the seem- ingly restricted setting of our work, findings provide valuable insights into critical limitation of current generative models, namely their failure to generalize for unseen compositions, by identifying the source of this limitation and proposing effective solution to mitigate it.
30

---

Published as a conference paper at ICLR 2025
G ADDITIONAL RESULTS AND DISCUSSION ON COIND
G.1 LEARNING UNDER NON-UNIFORM p(C))
Coldd
Color CC
A.CS))
(CS))
Digit (CS)
LACE
$0.23
98.90
57.51
Composed
0.77
CA
Digx (CS)
(a) Gaussian support
(b) Quantitative results for Gaussian support
C2 = 4) vs pC (c) p(22
C1
Figure 15: Results on Gaussian support: When the independent attributes have non-uniform cat- egorical distributions, the joint distribution of attribute combinations is not uniform. Even in this case, COIND learns (C C₃) accurately.
In our experiments, we considered the uniform support setting as an example where the attribute variables are independent of each other in the training data, i.e., C1 11 C2 X during training. However, uniform support is not the only scenario that can arise from independent attribute variables. In this section, we show that COIND can learn accurate marginals irrespective of the distribution of C.
We designed experiment using the Colored MNIST images where the attributes C1 and C2 sume values from a non-uniform categorical distribution that resembles a discrete Gaussian distri- bution. The resulting joint distribution of the attributes, which we refer as Gaussian support, is illustrated in Fig. [5a We trained CoIND and baselines on this dataset and evaluated on A and compositionality tasks. Apart from comparing the CS of baselines and ColND on these com- positionality tasks, we also evaluate if CoIND accurately learns p( (C;)) by comparing the learned (o(C. I C₁) against the true p(C, C₃). Intuitively, this verifies if COIND generates images with uncontrolled attributes matching their distribution in the training dataset.
Fig. quantittttty compares COIND against Composed GLIDE on CS in both and compo- sitionality tasks. Like previous experiments, CoIND outperforms Composed GLIDE w.r.t. CS in all tasks. In Fig. 15c, we verify if COIND has learned po C1) accurately by comparing against the true distribution p(C² C1). Po C1 c*) Po(Cy X))o(X C1 c*) is ob- tained as the histogram density of the attributes that appear in the generated images when C₁ = c"
We observe that the learned distribution PC C 4) is close to the true distribution, forming bell shape.
G.2 FAILURE EXAMPLES OF COIND
Here, we examine some samples generated by COIND where failed to include the desired at- tributes. We show these failure cases from each dataset, i.e., Colored MNIST, Shapes3d, and CelebA datasets. Samples from Colored MNIST and Shapes3d datasets are taken from the par- tial support setting, while the ones from the CelebA dataset are taken from the orthogonal support setting. Fig. 66a shows some failure samples from the Colored MNIST dataset. The images in the first row contain digits with colors leaking from the nearby seen attribute combination. Those in the second row correspond to approximation and have wrong attributes due to the approximation in the probabilistic formulation in Eq. ( Some images, like those in the third row, are unrealistic, although they may contain the desired attributes. We observe similar failures in Shapes3d samples shown in Fig. [6b] where the CoIND deviates from the desired compositions (column 1). Some failed samples from the CelebA dataset are shown in Fig. [6c] The samples correspond to the task of "smile" A "male". In the top image, it is hard to distinguish if the subject is smiling or laughing. In samples, observed only weak soft smile. This could be because smile is difficult to control due to its limited spatial presence in an image.
G.3 CONFORMITY SCORE FOR EACH ATTRIBUTE COMBINATION
31

---

Published as a conference paper at ICLR 2025
Green
Red (I v
Real
CoLND
"Male" A "Smiling"
Sinittr color
Single Singlesttruug misoaais
Smile laugh
Attrihun mismatch Compoxttion
compostion
3
A.A. Cream
Bluc)
Hard quantify
Utrealiaie
Unvealitic
(a) Failure samples from Colored (b)
Failure samples from (c) Faiture samples from Shapes3d dataset
MNIST dataset
CelebA dataset
Figure 16: Some samples generated by CoIND where it could not enforce the desired attributes.
In all experiments, report CS the primary metric to evaluate if the generative model produced images with accurate attributes. However, CS is the average accuracy across all unseen attribute combinations. Not all attribute combinations may be generated with equal accuracy.
100)
37
155
QO
89
27
77
94
100
81
50
32
50
89
99
55
5P
90
90
100
1:0
47
84
100
For instance, Fig. 17 shows the CS for each attribute combination in the A compositionality task in Colored MNIST image generation with partial support setting (row Fig. 4a). As reminder, CoIND achieved 52.38% CS unseen attribute combinations in this task.
Coler
77
100
99
100
100
99
1:0
97
80
97
100
100
98
36
20
We can see that CoIND can successfully generate all seen attribute combinations that appear the diagonal. Some unseen attribute combinations achieve > 90% CS, while others have nearly 0% CS. We do not observe the model struggling to generate images with any specific attribute digit, although some colors have generally lower CS than others. For example, colors 2 and have zero CS with more digits than others. On the other hand, colors 4, 5, and have high CS with all digits. We hypothesize that this disparity in CS could depend on the nature of attributes and the similarity between the values they can take.
100
100
70
89
97
95
70
46
47
Dight
Figure 17: Heatmap showing CS for each attribute combination the A compositionality task in Colored MNIST generation with partial support (row 10 in Fig. 4a)
G.4 COIND IMPROVES CONDITIONAL GENERATION
Support
Configuration
CS
Uniform
Vanilla
99.98
Support
Configuration
R²
CS
Uniform
COIND(A = 1)
100
Uniform
Vanilla
0.99
100
Non-uniform
Vanilla
99.98
Non-uniform
CoIND(A = 1)
99.98
Uniform
COIND(A 1)
0.99
100
Diagonal partial
Vanilla
33.14
Orthogonal partial
Vanilla
0.97
95.88
Diagonal partial
CoIND(A = 0.5)
68.82
Orthogonal partial
COIND(A 1)
0.99
99.57
(a) Colored MNIST
(b) Shapes3D
Table 7: Overall Performance Metrics for Conditional generation
32

---

Published as a conference paper at ICLR 2025
Given ordered n-tuple from the attribute space not observed during training, CoIND generate images corresponding to this sampled from joint distribution, Px(X(C)) To answer this question, Colored MNIST and Shapes3d datasets. Tab. shows the
train CoIND and the baselines results. As expected, the vanilla model, under full support, generates samples corresponding to the joint distribution. However, demonstrated in $ models trained partial support fail generate samples for unseen attribute compositions. In addition to the improved performance on logical compositionality, enforcing conditional independence explicitly improves conditional generation as well and produces better results on partial support compared to vanilla diffusion models for both Colored MNIST and Shapes3D datasets.
G.5 COIND CAN INTERPOLATE BETWEEN DISCRETE ATTRIBUTES
In cases, it may be necessary to have control over continuous-valuee attributes such as height or thickness. However, the datasets with continuous annotations may not be available to train such models. Or we may be in- terested in using pre-trained model that was trained generate images with discrete attributes. In such cases, can we generate samples where attributes take arbitrary values that do not belong to the set of training annota- tions? We show that COIND can interpolate between the discrete values of an attribute on which it was orig- inally trained and thus essentially produce images with continuous-valued attributes.
Observed
Interpolated
Observed
26
28
30
Figure 18: Although CoIND was only trained to generate images with orienta- tions 26 and 30", it succesfully gener- ated a sample with 28 orientation.
As mentioned in the main paper, we trained COIND to generate images from the Shapes3d dataset using the la- bels provided (Kim & Mnih 2018). The labels provided for the orientation attribute were discrete, although orientation itself continuous.
In Fig. 18 we highlight the images generated by COIND where the subject has orientations 26 and 30% We interpolate between observed discrete values linearly and generate the samples shown in the second column of Fig. [8] By carefully observing the variation in the gap between the corner of the cube and the corner of the room, notice that CoIND generated an image where the orientation of the cube is midway between those of 26 and 30% This demonstrates that CoIND offers a promising direction where training on datasets with discrete annotations sufficient generate samples with continuous-valued attributes.
33


Reviews:
[{"review_id": "MqpMwvdrk4", "reviewer": "Reviewer_5mnE", "summary": "This paper considers the problem of conditional generation for logical compositions of attributes. The authors claim that existing conditional diffusion models violate the conditional marginal independence assumption of the attributes. This makes them struggle in generating samples given arbitrary logical compositions of attributes, even if trained with uniform data over all possible combinations of attributes, and even worse if there is a statistical correlation or only some attribute combinations are seen. The paper then tackles this problem by training score-based models (with their diffusion model interpretation) with an additional loss term to enforce this conditional independence assumption, motivated by Fisher’s divergence between the joint and product of marginals of attributes. The assumption about marginal independence empirically validated by observing that Jensen-Shannon divergence of joint distribution and the product of conditional marginal distributions has a negative correlation with an accuracy metric called conformity score (CS).", "questions": "I found the definition of conformity score to be not very clear. First, I assume results are reported in percentage rather than as defined (between 0 and 1). My main confusion is about the distribution p(X,C) in the definition (line 199). For the generated samples, is the score measuring whether all consistent combinations of attributes appear following a certain distribution, or is it only measuring whether the attributes (given by the classifier) satisfy the logical composition?\n\nCan this method be used for compositions using general logical formulas beyond single AND and NOT operations?\n\nCan CoInD generate attributes having non-uniform probability (while still being conditionally independent)? For instance, to generate shapes in a scene following a certain distribution, or images for demographic groups following a certain population distribution.\n\nThe causal graph in Fig 2a implies independence among the attributes, but not necessarily conditional independence given X. It wasn’t clear whether (around line 121) the claim is that conditional independence is implied by this causal graph or it is an additional assumption made by the authors.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the problem of generating samples for logical compositions of attributes, including combinations previously not seen in data, which is highly relevant.\n\nThe proposed approach trains a single model to handle arbitrary compositions of attributes, and thus scales with the number of attributes. The proposed loss function is derived from a theoretically sound objective based on the Wasserstein distance between the true conditional likelihood and the learned likelihood satisfying the conditional independence assumption.\n\nExperiments are thorough, including different support settings (uniform, non-uniform, partial/orthogonal) for two benchmark datasets, and the results support the claims about conditional independence assumption. \n\nThe paper is overall well structured, although some details could be clarified (see below).", "weaknesses": "I have a couple of concerns about soundness and clarity. While the proposed loss function (Eq 6 and 7) has a clear theoretical justification of bounding Wasserstein distance, this is somewhat lost in the actual training objective of CoInD (Eq 8 and 9) due to simplifications for practical implementation. It was also not very clear how the modified L_{CI} in Eq 9 is derived.\n\nIn several places (including the problem statement), the authors seem to conflate the distribution of attributes being uniform vs. conditionally independent. The conditional distribution of attributes can be factorized without necessarily being uniform. Another issue is that even if each attribute had uniform probability, after conditioning on a logical formula in general, each marginal probability may no longer be uniform. \n\nMoreover, the paper mentions “incorrect marginals” multiple times as being a limitation of current approaches, but I believe the main claim is about conditional independence (factorizability). It would be clearer to refer to independence directly because it’s possible to have the correct marginal probability of each attribute while still violating the conditional independence assumption.\n\nAs the authors also discuss, the proposed method is restricted to a composition of a closet set of attributes, and cannot easily add compositional attributes without retraining from scratch. However, while a limitation, open-set compositions are outside of the scope of this paper and not a major weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFRlNltgJo", "reviewer": "Reviewer_oaF6", "summary": "This paper shows that standard conditional diffusion models violate this assumption, even when all attribute compositions are observed during training. The authors propose COIND to address this problem. It explicitly enforces statistical independence between the conditional marginal distributions. Quantitative experiments demonstrate a significantly more faithful and controlled generation of samples for arbitrary logical compositions of attributes. The benefit is more pronounced for scenarios that current solutions relying on the assumption of conditionally independent marginals struggle with, namely, logical compositions involving the NOT operation and when only a subset of compositions are observed during training.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The problem is interesting and the authors proposed a solution to the problem. The writing of the paper is clear and easy to follow. The method is simple and effective. The authors provide experiments to validate the proposed method.", "weaknesses": "1. The datasets in the paper are simple MNIST and synthetic 3D shapes. Experiments on real-world images could strengthen the paper. Experiments on natural images could provide more convincing evidence for the proposed method.\n\n2. What are the practical applications of NOT operation in the composition generation? What are the motivations for applying logic operations on the attributes with multiple values, e.g. c_i  in  [0, 1, 2, 3, 4, 5]?\n\n3. How does the model process logic expressions? How is the model implemented to process logic expressions? If not implemented, what are the motivations for applying logic operations? Are AND operations sufficient for composition generation with different attributes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ajmxZHM35f", "reviewer": "Reviewer_BxVf", "summary": "This paper performs compositional image generation with and, or, and not operators. The proposed method trains the diffusion models with a score function that enforces the attributes in the generated images to be independent. Finally, the authors show performance on two synthetic datasets.", "questions": "### Questions:\n\n* What happens if $\\lambda$ is set to a very high value? Would that give perfect conditional independence? Will that be an issue while minimizing $L_{\\text{score}}$?\n* What issues will the proposed algorithm face if the attributes are actually dependent in the true data-generating process?\n* How many attributes can be composed together? How does the proposed approach behave when you increase the number of attributes?\n* Based on Figure 3: if the training dataset has no [color: green, digit: 9], can the model generate such images? The proposed approach has a 55% conformity score (CS) for partial support. Does that mean that, among the generated images with the attributes [color: green, digit: 9], 55% contain these attributes correctly?\n* Can the authors provide images with minimal attribute combinations where the proposed method, along with all baselines, fails?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written. Every detail is provided in a clear manner. The description is intuitive. The authors also provided insightful observations.", "weaknesses": "### Minor weakness:\n* Some motivational examples of the utility of the proposed method in the introduction would be nice.\n* The conformity score with notation should be defined/discussed briefly in the experiment section again.\n* An initial look at Figure 3 might give readers the impression that the proposed approach is effective for such scenarios as well. This should be made clearer.\n* The numbers in line 417 can be written in a more detailed way.\n* The authors should give some examples of open and closed set attribute compositions.\n\n### Major weakness:\n* The only major issue with this paper is that there is no experiment on a real-world image dataset. One of the baselines, LACE, presents results for human faces with independent attributes, e.g., smile, eyeglasses, etc. Can the authors show performance on such datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lJIt5A4I4", "reviewer": "Reviewer_D8GZ", "summary": "This paper addresses the limitations of standard diffusion models in sampling data with arbitrary logical compositions of statistically independent attributes. It reformulates the problem as minimizing the Fisher divergence between the joint distribution and the product of marginals, demonstrating the effectiveness of this approach.", "questions": "(1) How is the failure of standard diffusion models linked to the violation of independence assumptions? Could you provide evidence or proof supporting this claim?\n\n(2) How does the model perform on more diverse datasets? Could additional results on simulated data be included to further validate its effectiveness?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a clear and well-articulated statement of the problem, supported by intuitive examples that help the reader understand the limitations of current conditional diffusion models. It effectively highlights the insufficiencies in these models, particularly in capturing accurate marginal distributions, and offers a novel solution to address these gaps. The authors ensure that the proposed model is designed to learn precise marginal distributions, an essential improvement over existing methods. The paper also includes rigorous experimental validation, utilizing a range of extensive datasets, which adds credibility to the findings and demonstrates the model’s effectiveness across diverse scenarios.", "weaknesses": "(1) Some claims in the paper need additional rigor. For instance, the assertion that the drop in conformity scores is due to the violation of independence relations in the learned model lacks clear justification. Providing stronger reasoning or citing relevant references would help substantiate this point.\n\n(2) The proof process could be clarified for readers. For example, the derivation of Equation 6 from previous equations is not immediately clear, and similar issues appear in other parts of the theoretical development. Additionally, some expected theoretical support is missing from the appendix, which could help make the proofs more accessible.\n\n(3) In the experiments section, the model is only evaluated on two real-world datasets, with no simulation results to provide more quantitative insights into the model’s effectiveness. Including simulation studies would strengthen the empirical validation by offering additional quantitative measures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLjW9qWT0H", "reviewer": "Reviewer_jrRB", "summary": "The authors introduce COIND, a novel approach that enforces conditional independence between marginal distributions by minimizing Fisher's divergence between joint and marginal distributions. Traditional models struggle to handle arbitrary compositions, especially when only partial data compositions are available during training. In comparison, COIND leverages causal independence to improve the generation of images that meet desired attribute compositions. The experiments show COIND’s performance in generating images with diverse and accurate compositions.", "questions": "1. As pointed out by (Karras et al. (2022), Fu et al. (2024)), generative diffusion models often introduce an early stopping time rather than reverting completely to time zero in the generation process to avoid numerical instability. Did you consider the impact of setting an early stopping time in your experiments? If so, could you elaborate on it? If not, could you explain why this setting might not significantly affect the results?\n\n2. The selection of the hyperparameter λ in the loss function is crucial. Could you share the principles you followed when choosing it (e.g., Bayesian optimization, cross-validation)? The analysis in Section 5.2 does not seem to fully address this aspect.\n\n3. In the experimental scenarios considered in this paper, the features are all  discretized. Do you believe it is possible to extend this method to continuous features, or potentially adapt it with certain techniques to handle continuous feature scenarios?\n\n4. The paper uses Jensen-Shannon Divergence (JSD) to measure conditional independence, though there are many metrics for conditional independence. For instance, Conditional Mutual Information (CMI) (Mukherjee et al., 2020; Li et al., 2023) has shown strong performance for continuous feature scenarios mentioned in Q3. Could you explain the advantages of using the JSD metric in this context?\n\nReference:\n1.\tLiu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022, October). Compositional visual generation with composable diffusion models. In European Conference on Computer Vision (pp. 423-439). Cham: Springer Nature Switzerland.\n\n2.\tKarras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.\n\n3.\tFu, H., Yang, Z., Wang, M., & Chen, M. (2024). Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arxiv preprint arxiv:2403.11968.\n\n4.\tMukherjee, S., Asnani, H., & Kannan, S. (2020, August). CCMI: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence (pp. 1083-1093). PMLR.\n\n5. Li, S., Zhang, Y., Zhu, H., Wang, C., Shu, H., Chen, Z., ... & Yang, Y. (2023). K-nearest-neighbor local sampling based conditional independence testing. Advances in Neural Information Processing Systems, 36.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces conditional independence analysis into the training of conditional diffusion models, integrating insights from both causal structure learning and generative modeling. By uncovering latent causal structures, it enhances the performance of traditional conditional diffusion models, which is quite innovative.\n\n2. The derivation of error decomposition in Section 4 is particularly impressive. By decomposing the Wasserstein distance, the paper introduces a novel loss function, providing a strong theoretical justification for incorporating causal structures like conditional independence, which is very intriguing.\n\n3. On the selected datasets, the proposed method consistently outperforms comparative approaches across various settings, demonstrating a significant improvement. Moreover, the presentation of experimental results is comprehensive and meticulous. Notably, Appendix A.7 provides an exhaustive comparison of experimental settings across methods, which is commendable.", "weaknesses": "1. While the paper uses illustrative images to present the experimental settings for Non-uniform and Partial support scenarios, the lack of mathematical descriptions limits the rigor of this section.\n\n2. The paper’s exploration of causal structures introduced during training remains relatively simple, focusing primarily on conditional independence, and lacks investigation into more complex causal graphs and mechanisms.\n\n3. The study only examines the model's performance on the Colored MNIST and Shapes3D datasets, which, compared to papers like Liu et al. (2023), leaves it without results on more complex, real-world datasets to substantiate the research conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)