Improving Compositional Generation with Diffusion Models Using Lift Scores
Authors: Chenning Yu
Sicun Gao
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
Training-free
Rejection Sampling
Primary area: applications->computer_vision
TL;DR: We introduce a training-free resampling criterion for compositional generation with diffusion models, which is computational efficient and requires no additional modules.

Abstract: We introduce a novel resampling criterion using lift scores, for improving compositional generation in diffusion models. By leveraging the lift scores, we evaluate whether generated samples align with each single condition and then compose the results to determine whether the composed prompt is satisfied. Our key insight is that lift scores can be efficiently approximated using only the original diffusion model, requiring no additional training or external modules. We develop an optimized variant that achieves relatively lower computational overhead during inference while maintaining effectiveness. Through extensive experiments, we demonstrate that lift scores significantly improved the condition alignment for compositional generation across 2D synthetic data, CLEVR position tasks, and text-to-image synthesis. Our code is available at github.com/rainorangelemon/complift.

Improving Compositional Generation with Diffusion Models Using Lift Scores
Chenning Yu?
Sicun Gao
Abstract
diffusion model to improve its ability to generate samples that satisfy complex prompts (Huang al., 2023; Bao et al., 2024), and (2) training-free methods, which either refine the sampling process, or apply rejection sampling to refine the generated samples without modifying the underlying model (Feng et al., 2022; Chefer et al., 2023). These cat- egories of methods are not necessarily mutually exclusive, and they can be combined achieve better performance.
We introduce novel resampling criterion using lift scores, for improving compositional generation in diffusion models. By leveraging the lift scores, we evaluate whether generated samples align with each single condition and then compose the results determine whether the composed prompt is satisfied. Our key insight is that lift scores can be efficiently approximated using only the original diffusion model, requiring no additional training or external modules. We de- velop an optimized variant that achieves relatively lower computational overhead during inference Through
We introduce CompLift, a training-free frree rejection criterion for improving compositional generation in diffusion models. The concept of lift score (Brin et al., 1997) builds on a sim- ple observation: a generated sample truly satisfies a con- dition, the model should perform better at denoising it when given that condition compared unconditional denoising. This insight leads to an efficient criterion for accepting or rejecting samples. Similar to Diffusion Classifier (Li et al., 2023), we show that lift scores can be efficiently approxi- mated using only the original diffusion model's predictions, requiring no additional training or external modules. By de- composing the satisfaction a complex prompt into the sat- isfaction of multiple simpler sub-conditions using lift scores, our approach can boost the model's alignment with complex prompts without changing the sampling process. Through extensive experiments on 2D synthetic data, CLEVR posi- tion tasks, and text-to-image generation, we demonstrate that CompLifi significantly improves compositional genera- tion while maintaining computational efficiency.
while maintaining effectiveness. extensive experiments, we demonstrate that lift scores significantly improved the condition alignment for compositional generation across 2D synthetic data, CLEVR position tasks, and text-to-image synthesis. Our code is available at raincrangelemon git giithub..... 1o/ complifte
1. Introduction
Despite the success of diffusion models in generating high- quality images (Sohl-Dickstein et al., 2015; Ho et al., 2020; Nichol & Dhariwal, 2021; Song et al., 2020), they some- times fail to produce coherent and consistent results when the condition complex or involves multiple objects (Huang al., 2023; Feng et al., 2022; Chefer et al., 2023). This limi- tation is particularly evident in tasks that generate the images with specific combinations of attributes or objects, where the generated samples may only partially satisfy the conditions or exhibit inconsistencies across different attributes.
Contributions. Our contributions are threefold: 1 We in- troduce CompLift, as a novel resampling criterion using lift scores for compositional generation, requiring additional training (Section 3). We explore the design space of Cofp including noise sampling strategies, timestep se- lection, and caching techniques to optimize computational efficiency (Section 4). @ We scale our approach to text-to- image generation, demonstrating its effectiveness in improv- ing compositional generation in this challenging domain (Sections 5 and 6).
To mitigate this issue, several methods have been proposed improve the compositional generation with diffusion mod- els. These methods can be described 2 main categories: (1) training-based methods, which train or finetune the
'Department of Computer Science and Engineering, UC San Diego La Jolla, USA. Correspondence to: Chenning schyOl0@uusd.cdtt Sicun Gao <sicung @ucsd.edu>>.
2. Related Work
Recent studies have revealed significant limitations in diffu- sion models (Dhariwal & Nichol, 2021) for handling compo- sitional generation, particularly as missing objects, incorrect spatial relationships, and attribute inconsistencies (Huang
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Pr(x)
Py(*)
Ground Truth
Composable Diffission
Compl.jf
I
x
+
Figure 1. An illustration of product, mixture, and negation compositional models, and the improved sampling performance using CompLift. Left to right: Component distributions, ground truth composed distribution, composable diffusion samples, samples accepted by Complift. Top: product, center: mixture, bottom: negation. 2 represents the empty set no samples generated or accepted. Each component distribution is trained independently using a 2D score-based diffusion model. Accuracy is evaluated based on whether generated samples fall into the support or within the 3r region of the composed distribution (details in Section 6, Appendix E).
et al., 2023; Feng et al., 2022; Chefer al., 2023).
with a whole single prompt. Kong et. al (Kong et al., 2023) regards lift scores as point-wiss mutual information for the explanability of the generated images. In this work, we investigate the use of lift scores as compositional criterion for rejection and resampling in compositional generation.
Current approaches to address these challenges fall into several categories. One line of work leverages the cross- attention mechanisms, modifying attention values during synthesis to emphasize overlooked tokens (Feng et al., 2022; Chefer et al., 2023; Wu et al., 2023; Wang et al., 2024). An- other approach introduces layout control for precise spatial arrangement of objects (Feng et al., 2024; Chen et al., 2024). While effective, these methods require specific model archi- tectural knowledge, unlike our model-agnostic approach.
3. Using Lift Scores for Rejection Sampling
3.1. Diffusion Model Preliminaries
Diffusion model is a class of generative models that generate samples by denoising noisy input iteratively, Given a clean sample In. the forward diffusion process ((x7|<x--) sequentially adds Gaussian noise to Φ;-1 at each timestep I, whereas the learned reverse diffusion pro- cess pe (x;-1 (x;, c) denoises æ, to reconstruct T;-1, op- tionally conditioned on a condition e. The conditional prob- ability of ano can be defined as:
The diffusion process itself can also be modified to improve compositional generation. Composable Diffusion (Liu et al., 2022) guides generation by combining individual condition scores. Discriminator Guidance (Kim al., 2022) employs an additional discriminator to reduce the distribution mis- match of the generated samples. Recent methods introduce advanced sampling techniques like Markov Chain Monte- Carlo (Du et al., 2023), Restart Sampling (Xu et al., 2023), rejection sampling (Na et al., 2024), and particle filtering (Liu et al., 2024). We complement these methods by provid- ing a technique that can be integrated without assumptions about the underlying sampling process.
Pe
NITT [ p(æT)
II (o c) dai,T,
where p(ær) is typically standard Gaussian N I).
The diffuusion model is often parameterized as neural net- work cg to represent the denoising function. Using the fact that q(ær, ax)) == N(a;; varan, (1 - āx a)) the model trained by maximizing the variational lower bound (VLB) of the log-likelihood of the data, which can be approximated as the Evidence Lower Bound (ELBO), i... diffusion loss:
Most closely related to our work are resampling strategies that operate as either selection of the final image (Karthik et al., 2023) or search of the initial noise (Qi et al., 2024; Ma et al., 2025). However, these methods typically require external models for quality assessment. In contrast, our approach leverages the denoising properties of the diffusion model itself (Li et al., 2023; Chen al., 2023; Kong al., 2023), climinating the need for additional models.
log Po (xalc) )) 2 1. log Po(#nrr c) g(
- [walle e ( c) + C. Following the previous works (Ho et al., 2020; Li et al., 2023; Chen et al., 2023), we set wy = for all t as achieves good performance in practice.
Our work is not the first to apply lift scores to diffusion models. Condition Alignment Score (Hong et al., 2024) proposes an equivalence to rank the alignment of images
2

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Algorithm 1 Rejection with CompLift
Type
Algebra
Acceptance Criterion
1: Input: test sample 2o, condition {c: # of trials T,,
Product
C1 C2
min lift((//c)) >
algebra A
2: Initialize L. Ft(c:] = list() for each c;
Mixture
C1 V C2
[1. (x/(()))))) > 0
3: for trial j 1,... . T' do
Negation
C1
lif((a))e)) I
4: Sample t ~ [1, 1000]; c~ (O( I)
5: 3t = -ā + /I dGE
Table
Examples of Compose for multiple conditions.
6: for condition E {c:) do
7: (aut,() = | c0( Ø) |E co(
8: Lift|ckk (ittlcclaappend(tffff... (x))
model should improve the reconstruction quality to æ from the noisy 3S if given to the model, compared to the reconstruction quality of a; when e is not known. Hence, the criterion of CompLift for single condition is:
9: end for
10: end for
11: (Li((=)))::]]= = (mean (t)]
12: return Compose (F cc)
> Table
if Equation (3)> 0; 0 (reject & resample) if Equation (3) 0.
(accept)
T < c) =
3.2. Approximating Lift Scores
Given a sample z, we wish to check whether it aligns with a condition c. In other words, we want to see how well the association æ has with c. This brings us to the concept of lift, which is an existing concept in data mining:
The Compose function encodes the logical structure of the prompt. This compositional logic mirrors logical inference over sub-condition satisfaction and enables generalization to complex prompts (see examples in Figure I and Table 1). For instance:
lift(a)))) = log p(c|æ) = log p(ac c) log p(x]cc) p(z)p(()
p(c)
p(æ²)
(product) Compose accepts the sample only all compo- nent conditions have positive lift scores. (mixture) The sample is accepted if least one lift score positive.
We note that the definition of lift scores is slightly different from the traditional data mining definition (Brin et al., 1997) as it is defined in the logarithmic space. We abuse the notation since it is easier to implement in the logrithm space, and we call it lift score throughout this paper.
(negation) The sample is rejected if the corresponding lift score is positive.
We describe the general form of Compose in Appendix B.
One perspective to understand lift scores is to regard p as perfectly-learned classifier. If x is arbitrary unknown sample, this classifier tries its best to predict the probability of æ: belonging to class as p(c). However, once æ is given as known sample, the classifier will utilize the information in æ to increase its prediction accuracy, as p(c|z)). If x has any positive association with c, we would expect p(c|æ) > p(c). which means lift(())c)= > 0.
4. Design Space of CompLift
We present the naive version of CompLift in Algorithm 1, where the lift score in Equation (3) is estimated using Monte- Carlo sampling, In this section, we discuss design choices of Compliift We wish to answer the following questions: (Section 4.1) How does the noise sampling affect the esti- mation accuracy of lift scores?
Approximating the lift scores. In preliminaries, we that both p(z|c) and p(z) can be approximated using ELBO:
(Section 4.2) What role do different timesteps play in the effectiveness of our criterion?
p(z|c) zs exp( - Es, ||c co 1² + C), p(x) Rs exp(- Be,ellc- co 0112 + C).
(1)
(Section 4.3) Can we optimize the computation overhead via caching strategies?
(2)
A Running Example: 2D Synthetic Dataset. To illustrate the behavior of lift scores and the effect of different design choices, we use synthetic 2D datasets composed from com- ponent distributions such as Gaussian mixtures on ring or uniform strips. Each composed distribution is defined by applying algebraic combinations to these components (see Figure I as an example). Samples are evaluated based on accuracy: whether they fall within the support of the target composed distribution. We refer readers to Section 6 and Appendix E for training details and metrics.
Notice that the constant C' is independent of æ and c, there- fore dividing p(x|c) cc by p(z) cancels C. This gives us the approximation of lift in the form of log p(ale). p(z)
lift(()) c) R Ee,. flccfllc-oofe (2,,
c)...) (3)
Equation (3) gives us another explanation of lift scores from a denoising perspective: c is positively associated with ar then given a sample x, by adding noise to x, a diffusion
3

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
1.0
product (share each trie)
Algorithm 2 Optimized Rejection with Cached CompLift
0.9
product (share tlllss
product (idepeedem))
1: Input: conditions {<i)==>> algebra A, # of reverse
0.8
timesteps N
Accuracy
micture (shere each tris))
Initialize aT ~ NNO, T)
0.7
misture (shore all mals)
misture (Idepandeen)
3: for timestep Ej [tv, -1 t1] do
0.6
4: > Standard generation process
negation (thare each triss)
0.5
negation (share al triats)
5: Tt;-1 = st.ep fe, {4)
negation (Indepaindeen))
0.4
6: > Add predictions to cache
10'
10
7: if classifier-free guidance then
Number of Trials
8: Add precomputed ( 2) cache
Figure 2. The accuracy of Lff with different noise sam-
9: end
pling strategies on synthetic dataset. See Section 4.1.
10: if Composable Diffusion then
11: Add precomputed (M),, G)) to cache
100
12: end
Accuracy
13: Add æt, to cache
60-
14:
end for
40-
15:
run Algorithm 1 with cache, where # of trials T = N,
and Tt,, ca (Z,,,, Ø), and co es) are reused.
200
400
600
800
1000
Timestep Used (1 Trial)
Figure 3. Accuracy of acceptancerejeettoon over single sam- pled timestep for pretrained model (Liu et al., 2022) on CLEVR Position dataset. We found that models trained with importance sampling require importance sampling for ELBO estimation.
uniform timestep sampling, sampling the timestep uniformly for ELBO estimation (Line 4 in Algorithm 1) is sufficient.
More surprisingly. we also found that the timestep for ELBO estimation needs importance sampling when the diffusion model is trained under importance sampling, where the importance sampling enhances the stability of optimizing LVLD (Nichol & Dhariwal, 2021). In particular, we discov- ered a pretrained model for CLEVR Position dataset (Liu et al., 2022) requires the importance sampling because of the above reason. We show the effect in Figure 3.
4.1. Effeec of Noise
In Line 7 in Algorithm 1, choose share the when estimating both p(æ|c) and p(z). To validate this design, we conduct the experiment on 2D synthetic dataset with the following 3 settings:
(1) Independent Noise: We sample two independent noises and e' for p(æ cc) and p(æ), respectively.
This suggests that the timestep sampling of ELBO estima- tion to stay consistent with the sampling strategy at training stage. Therefore, we choose use importance sampling for the CLEVR pretrained model, and use uniform sampling as default for other tasks, i.e., 2D dataset and text-to-image
(2) Shared Noise for Each Trial: We share the noise
between p(æ] c) and p(z), each trial samples new noise. (3) Shared Noise Across All Trials: We share the noise
across all trials for both p(æ|c) and p(z).
4.3. Improving Computational Efficiency with Cached Prediction
In Figure 2, we observe that sharing noise for each trial is mostly robust over the number of trials, which is consistent with the conclusion in Diffusion Classifier (Li et al., 2023). We choose to use this design as the default setting. The independent strategy increases the accuracy with more trials, especially for mixture and negation algebra. We also find small regression in the sharing strategies across all tasks. Our hypothesis is that sharing the same noise introduces some bias in the estimation, and it may over-reject samples as a conservative way. With more trials, the bias is amplified, thus makes more samples over rejected.
In the naive version of CompLiff in Algorithm 1, the algo- rithm requires (n + 1) T times of forward passes for conditions. To reduce the computational cost, we propose caching the diffusion model's intermediate predictions.
We find that we can reuse the prediction of e (æx, 2) and (< o,)) at intermediate timestep during the gen- eration process of To. Note that the unconditional pre- diction cø 2) is typically precomputed for generation with classifier-free guidance (CFG) (Ho & Salimans, 2022). For conditional prediction {Ee(# cx) )) they are pre- computed under the Composable Diffusion framework (Liu et al., 2022). Under this framework, we can reduce the number of extra forward passes from (n + 1) to O.
4.2. Effect of Timesteps
Similar to Diffusson Classifier (Li et al., 2023), we found that in most cases, if diffusion model cg trained with
4

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
turtle and a blue clock
frog with bow
an orange backpack and purple car
an elephant with gl glasses
a cat and a horse
black crown and car
Accepted with CompLift
Rejected with CompLift
Figure Accepted rejected SDXL examples using CompLift criterion. Objects in blue are composed through the given prompts. Here we show the images from the text prompts with the most-improver CLIP scores. See more examples in Appendix G.
Image (1024x1024)
Eqn.5 for Black Car
Egn.3 for Black Car
23.67s
0.08
Baselines
20.15s
uaae
20
Ours
Running
11.13s 11.71s
0294
7.62s
0.002
0.24s
0.25s
0.79s
Composabioii
Orched Compler
(Fa50))
Conpure
(7a50))
EBM (ULA)
EBM (MALA)
EBM HAC))
EDM (MAC)
Latent (128x128)
Een.5 for White Clock
Eqn.3 for White Clock
0.0G
8.006
0.903
Figure 5. Average rumning time 2D synthetic dataset. T indicates number of trials. Note that our methods can be further optimized the lateney of only forward pass with parallelization, while MCMC methods require sequential computation. See Table 2 for performance comparison all the methods.
-05
0.002
Figure 6. Replacing in the differential loss of Equation (3) by e (a, Ceoeppse) ) reduces the estimation variance. Prompt: a black and white clock. Pixels with negative scores masked out.
For more general framework, computing the conditional predictions (Ca(x,, c.) } additional computation on- the-fly during generation is beneficial for caching. With sufficient amount of GPU memory, this operation typically would not bring extra time overhead, since all computations can be done parallel to the original generation task. In this context, we can reduce the number of extra forward passes T, if CFG is used.
of only forward pass by parallelization, while MCMC methods (Du et al., 2023) require sequential computation. For more details, see pseudo-code in Appendix D.
5. Scaling to Text-to-Image Generation
We mainly consider the problem of missing objects the text-to-image compositional task. Given multiple objects [c:]² address simultaneously, prominent error of diffusion models is that the generated images sometimes do not include object(s) (cylū E [1,n]] mentioned in the text.
We provide the optimized version of CompLift in Algo- rithm 2. When running Line 4 in Algorithm I, is not i.i.d. any more, but computed as e = (æc, vāgzo)/ VI & instead using the cached w. Another difference is that T is constrained to be exactly the same as a number of inference steps N. In practice, we found that these 2 changes do not lead to a significant performance drop as long T is not too small. We show the comparison of the running time on 2D synthetic task in Figure 5. The overhead introduced by the cached CompLift is negligible for the Composable Diffusion baseline (Liu et al., 2022). With enough GPU memory, our method can be further optimized the latency
5.1. Counting Activated Pixels Existence of Objects
Our approach is to count the number of activated pixels in the latent space to determine the existence of objects. A pixel at position [a, 6) in the latent z is activated for condi- tion c, if lift C.) > 0. Here we regard lift , c;) as a matrix with the same shape of z. This is achieved by aver- aging only over the timestep and not the feature dimension
5

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Image (1024x1024)
Latent (128x128)
Activated Pixels:
Actiivated Pixels:
xtort
Activated Pixels: 4178
15
10
0.000
1.
glass bottle with message drifting past starfish
05
HUA00
1,0
3.0010
00
H00202
500000
00200
0,0
000000
Image (1024x1024)
Latent (128x128)
Activated Pixels: 2297
Activated Pixels: 4490
# #ctivvated Pixels: 924
0.2001
00006
0.0015
steaming teacup besides an open book and candle
05
0,0003
0.0000
30001
40,0012
405
100000
0.0001
000055
0,00000
0,0000
0,0000
Image (1024x1024)
Latent (128x128)
# Activaated Pixels: 631
Activated Pixels: 4395
Activated Pixels:
1210
p0
00.0000
wooden ladder reaching into treehouse under stars
05
00,0014
00
00,0022
-1.0
Image (1024x1024)
Latent (128x122 28)
# Activated Pixels: 5507
Activated Pixels: 2420
#Activated Pixels: 2066
10
40,00120
old gramophone on a windowsill with falling autumn leaves
05
HUADC
0.00015
00
00,00110
05
0101
-10
Figure 7. CompLift for text-to-image compositional task. Pixels with negative scores are masked out. From left to right: original image ær the latent the heatmaps of lift score (ift(z, c;) for each object component c,. The border color represents the corresponding object. We use random prompts and observe the clear pixel separation in general. Here the images with higher aesthetic qualities are presented.
when calculating the mean, i.e., Line 12 in Algorithm 1. Definition of CompLift in image space. In particular, an object considered to exist in the image æ; the number of activated pixels in its latent is greater than threshold T:
5.2. Reduce Variance of ELBO Estimation
Our initial attempt is to see whether the pixels in the latent z align with the lift score. Intuitively, we expect for an arbitrary pixel with position [a, b] from a target object e, its lift score lift (2, ca) should be greater than 0 in most cases. However, we found that the estimation variance is high when applying Equation (3) directly. As illustrated in the 3rd column in Figure 6, the lift score is noisy overall.
lift((,,)))
Σ a.bE[l.m]
(Uifi(z.cciia.) >
(4)
where is the latent in the shape of (m, m)) T is hyper- parameter be tuned for our approach. We set 250 the median activated pixel count among all images. Tests at 25th and 75th percentiles showed the median works best.
We hypothesize that the sampled noise in the L2 loss in Equation (3) is major factor of the variance. In the ELBO estimation, we first add the noise to the image and make predictions, similar to adversarial attacks. The model may or may not have a good prediction on the new noisy image, therefore, the model's bias might lead to unstable estimation. To reduce the variance, we can find candidate to replace when calculating the L2 loss in Equation (3). Ideally, the candidate is close enough to but inherently cancels out the bias from the model's prediction.
Equation (4) of CompLift for images makes the identifica- tion of objects more robust and interpretable. The rationale that the activated pixels in the latent space are more likely be the ones that correspond the objects in the image. We present examples in Figure As illustrated, we can see clear consistency between the pixels activated by lift((z c,) in the latent and the object e; in the image. Since the object often occupies only fraction of the image, simply taking the mean along the feature dimension as Equation (3), may not be as effective as Equation (4), as those unrelated pixels with lift(z, e, ex) < 0 will dominate the averaged score.
Our finding is that the composed score eø (21, Coompose) is a good candidate to replace e, where Ceompose the original composed prompt to generate latent 2;. For example, in Figure 6, Commpose is "a black and white clock". Since
6

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Method
Product
Mixture
Negation
I Shape
3 Shapes
Shapes
Acc 1 CD
Acc
CD /
Acc
CD
Composable Diffusion
56.5
0.061
70.7
0.042
7.9
0.207
Ground Truth
ERM (ULA)
51.7 0.076
71.8
0.044
11.4
0.186
EBM (U-HMC)
56.3
0.036
73.1
0.063
9.4
0.269
EBM (MALA)
62.6
0.054
73.8
0.036
6.8
0.203
Composable
EBM (HMC)
64.8
0.021
73.7
0.078
4.2
0.340
Diffusion
Composable Diffusion
+ Cached CompLift
99.5
0.009
86.5
0.023
62.0
0.137
C Composable Diffusion
+ CompLift (T = 50)
99.1
0.015
98.6
0.029
75.0
0.154
Compl.ift
Composable Diffusion
99.9
0.009
100.0
0.023
78.7
0.131
+ CompLift (T- = 1000)
Figure 8. Examples of samples on CLEVR Position dataset.
Table 2. Quantitative results 2D compositional generation.
Ace (%) mcans accuracy, and CD means Chamfer Distance.
points randomly for each component distribution, and train individual diffusion model for each distribution. We use the diffusion process of 50 timesteps for both training and inference, which means the cached version of Compliift uses 50 trials. The vanilla version of CompLift uses 50 or 1000 trials. All the methods generate 8000 samples for inference. For CompLift algorithms,, we use Composable Diffusion (Liu et al., 2022) to generate the initial 8000 samples, then apply the CompLift criterion to accept or reject samples. We test MCMC methods with Energy-Baan Based Models (EBM) as additional baselines (Du et al., 2023), including Unad- justed Langevin Annealing (ULA), Unadjusted Hamiltonian Monte Carlo (U-HMC), Metropolis-Adjuste Langevin Al- gorithm (MALA) and Hamiltonian Monte Carlo (HMC). We provide more details in Appendix E.
eo (Zz, ) mainly guides the generation process of Zt, it is reasonable to assume that it is good at reconstructing the original latent, therefore is close to e. Furthermore, since co(Zt. Ccompose) and (0(2,, C;) (or co(z), 2)) come from the same model's prediction, it helps cancel the model's bias. The results are shown in the 2nd column of Figure 6, We observe that the variance is significantly reduced, and the activated pixels are more consistent with the objects.
The improved lift score for condition e and latent z is:
Iif( c) = Fr,c (leeo(ze; Ccompose) ) EZ( 0)2 (1), Ceumpase) co cz( e) [|]]
(5)
Intuitively, if object c exists in image x, then for most noisy latents Zt, E0( c) should be closer to c0 Ccompose) than the unconditional cg(2.. 2) in the corresponding pixels. The combination of Equations (4) and (5) represents our final criterion for text-to-image compositional task. We first esti- mate the lift score for each latent pixel via Equation (5), and then count the activated pixels to determine the existence of objects in the image via Equation (4).
Metrics. We define that a sample satisfies a uniform distri- bution if it falls into the nonzero-density regions. A sample satisfies Gaussian mixtures if it is within 30 of any Gaus- sian component, where is the standard deviation of the component. The negation condition is satisfied, if the above description is not satisfied. We calculate the accuracy of the algorithm on a composed distribtuion, as the percentage of generated samples that satisfy all the conditions. We use the Chamfer Distance to measure the similarity between the generated samples and the dataset, as we find an efficient and general metric, applicable to all kinds of algebra.
6. Experiments
In the following sections, we evaluate the effectiveness of our Complift criterion on 2D synthetic dataset, CLEVR Position dataset, and text-to-image compositional task.
It is genuinely hard define universal co 2) for the 2D synthetic task, since in this task, the 2D distribution can be of arbitrary shapes. Therefore, we find an effective strategy by defining co (2t,, Ø) == ac ( e) with a = 0.9 for each c. Consequently, the lift criterion is simplified as Rr,, {lle - (x, e) - ||e - co(a,, l)). We find this criterion is surprisingly effective for this 2D task. One hypothesis is that if has some correlation with c, then well-trained ep should be good at predicting the noise en (2,, c) that is closer to and further from O in most cases.
6.1. 2D Synthetic Dataset
Experiment setup. We consider 2D synthetic dataset with 3 types of compositional algebra: product, mixture, and nega- tion. We generate the distributions following the way in Du et. al (Du et al., 2023)- they either Gaussian mixtures or uniform distribution. We use the same architecture of networks from (Du et al., 2023), which regards the diffuuion model Co as Va Eg. Eg is a 3-layer MLP and trained with the same loss function as the standard diffusion model by minimizing FFt,e/V. a (o (x,,t) t) - <ll We sample 8000 data
The overall result is shown in Table 2. We observe that the CompLift criterion is effective in all 3 types of compo- sitional algebra. Compared to MCMC methods, rejection
7

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
CompLift
Cached CompLift
Composable Diffusion
Method
Animals IR
Ohject&AAnimal
Objects
100
100.0
100.0
99.7
97.5
CLIP
C
CIPP
Stable Diffasion 1.4
0.310
-0.191
0.343
0.432
0.333
-0.684
99.0
SD 1.4 + EBM (ULA)
0.311
0.026
0.342
0.387
0.344
-0.380
90.3
SD 1.4 Cacked Compla3
0.319
0.128
0.356
0.990
0.344
-0.131
90
SD 1.4 CompLij (T=30)
0.320
0.241
0.355
0.987
0.344
-0.154
Accuracy (%)
90.8
84,3
50-
SD 1.4 Compl ift (TH 200)
0.322
0.293
0.358
1,093
0.347
.0.050
Stable DifTusion 2.1 2.1 EBM (ULA)
0.330
0.532
0.354
0.924
0.342
-0.112
80-
0.829
0.981
0.218
78.7
SD 2.1 + Cached CompLifi
0.339
0,880
0.361
1.252
0.354
0.353
CompLift (7=)0)
0.992
1.263
0.454
SD 2.1 CompLift ( (T=200)
0,340
0.975
0.362
1.283
0.355
0.489
40-
Stable Diffusion XL
0.338
1.025
0.363
1.621
0.359
0.662
SD XL EBM (ULA)
0.335
0.913
0.362
1.676
0.361
0.872
SD XI. Cached CompLif
0.341
1.244
0.364
1.687
0.368
0.896
SD XI.
Complijt (T=50)
0.342
1.222
0.364
1.700
0.365
0.842
2
SD XI.
CompLift (T=200)
0.342
1,216
0.364
1.706
0.367
0.890
Number of Constraints
Number of Constraints
Figure 9. Quantitative results on CLEVR Position dataset.
Table 3. Quantitative results on Attend-anddExcie Bench-
marks (Chefer al., 2023). IR is ImageReward (Xu et al., 2024).
with Compl.ift criterion is able to generate empty sets by construction. We show some examples in Figure 1, and more examples are illustrated in Appendix E.
that the Compliift criterion is able to generate samples that satisfy all conditions more effectively than the baseline. As the number of constraints increases, the performance of the baseline drops significantly, while the performance of the CompLift criterion remains relatively stable. We also show some examples in Figure 8. Meanwhile, we observe some regressions in the FID scores. We hypothesize that the rejection reduces the diversity of samples, which results in the higher FID scores, since the image quality has no significant difference from Composable Diffusion.
6.2. CLEVR Position Dataset
Experiment setup. The CLEVR Position dataset (Johnson et al., 2017) is a dataset of rendered images with a variety of objects placed in different positions. Given a set of 1-5 spec- ified positions, the model needs to generate an image that contains all objects in the specified positions (i.e., product algebra). We use the pretrained diffusion model from (Liu et al., 2022). The model is trained with importance sam- pling (Nichol& & Dhariwal, 2021). Thus, we use importance sampling for ELBO estimation. In particular, we find that sampling solely as t = 928 is sufficient to give us a decent performance by observing Figure 3. This leads to a compu- tational efficiency via only trial at 928 for both vanilla and cached versions of Complaft. For each composition of conditions, our algorithm uses Composable Diffusion (Liu et al., 2022) to generate the initial 10 samples, then apply the Lifi criterion to accept or reject samples. We test all methods with 5000 combinations of positions with various numbers of constraints. All methods use classifier-fre free guidance with a guidance scale of 7.5 (Ho & Salimans, 2022).
6.3. Im-gge Compositional Task
Experiment setup. We use the pretrained Stable Diffu- sion 1.4, 2.1, and SDXL (Rombach et al., 2022: Podell et al., 2023). The models are trained with large-scale text-to- image dataset LAION (Schuhmann et al., 2022). We use the same composed prompts as Attend-and (Chefer al., 2023), which considers classes of composi- tions 2 animal categories, animal and I object category, and 2 object categories (i.e., product algebra). There are 66 unique animal combinations, 66 unique object combi- nations, and 144 unique animal-object combinations. We use the diffusion model to generate 5 initial samples using each combined prompt, then apply the CompLift criterion to accept or reject samples. For inference, we use 50 diffusion timesteps, which means the cached version of CompLift uses 50 trials. The vanilla version of CompLift uses 200 trials. We use CFG with a guidance scale of 7.5 for all methods. We fix r = 250 for all experiments.
Benchmark
Metrics. We use Segment Anything Model 2 (SAM2) (Ravi et al., 2024) as a generalizable verifier, to check whether an object is in a specified position. We first extract the background mask, which is the mask with the largest area, using the automatic mask generator of SAM2, then label a condition as satisfied if the targeted position is not in the background mask. We find this new verifier is more robust compared to the pretrained classifier in Liu et al. (2022). We provide more details in Appendix F. We calculate the accuracy as the ratio of samples that satisfy all given condi- tions. To calculate FID, we compare the generated samples to the training set provided by Liu et al. (2022).
Metrics. We use CLIP (Radford et al., 2021) and ImageRe- ward (Xu et al., 2024) assess the alignment between the generated images and the prompts. For combination where CompLifi is invalid for all generated samples, we uniformly sample one of the images as the final output. The metrics are averaged over the samples within each combination class.
The quantitative results are shown in Table 3. We observe that with Complift, the performance for each version of SD)
We show the quantitative results in Figure 9. We observe
8

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Pearson Correlation: 0.8246
Rejected
Rejected
Acknowledgements
0.30
CLIP Score of White Clock
We thank the anonymous reviewers for their helpful com- ments in revising the paper. This material is based on work supported by NSF Career CCF 2047034, NSF CCF DASS 2217723, and NSF AI Institute CCF 2112665.
0.28
0.24
Accepted
Accepted
Impact Statement
0.20
We introduce CompLift, a novel resampling criterion lever- aging lift scores to advance the compositional capabilities of diffusion models. While we mainly focus on efficient approximation of lift scores without additional training and improving compositional generation, acknowledge sev- eral broader implications, similar to other works on diffusion models and image generation.
0.18
500
1030
#CompLift Activated Pixels A White Clock
Figure 10. The correlation between the number of activated pixels and the CLIP score, The correlation is calculated using the Pearson correlation coeflicient. Each point represents generated image. Prompt: a black car and a white clock.
As we demonstrate in this work, CompLift could benefit image generation applications through more reliable gener- ation of complex, side, Complaft could enhance creative tools through more precise and reliable image generation, potentially enabling more sophisticated applications in art, design, and enter- tainment. However, like other works on diffusion models, CompLift could potentially be misused to create misleading or deepfake contents, if not properly managed (Mirsky & Lee, 2021). These contents could be more consistent with the user intention, leading to potential societal harms such as scams and misinformation. Additionally, biases in train- ing data could be amplified through the resampling process, potentially perpetuating societal biases and underrepresenta- tion in the generated images. We encourage work building on CompLifi to evaluate not only technical performance but also the risks and ethical implications of the approach.
can be boosted be comparable their next generation for some combination classes. For example, SD 1.4 with CompLift is comparable to the vanilla SD 2.1 on object- animal combinations and object combinations. SD 2.1 with CompLift is comparable to the vanilla SDXL on animal combinations. Since the CompLift criterion is a training- free criterion and does not depend on additional verifier, also shows novel way of how diffusion model could self-improve during test time without extra training cost (Ma al., 2025). Additionally, show its correlation with CLIP in Figure 10, and leave the discussion in Appendix G.
outputs. On the positive
7. Conclusion
We propose novel rejection criterion Complift for com- positional tasks, which is based on the difference of the ELBO between the generated samples with and without the given conditions. We show that the CompLift criterion is effective on 2D synthetic dataset, CLEVR Position dataset, and text-to-image compositional task. We also demonstrate that the CompLift criterion can be further optimized with caching strategies.
References
Bao, 7.,, Li, Y., Singh, K. K., Wang, Y.-.., and Hebert, M. Separate-and-enhance Compositional finetuning for text- to-image diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-10, 2024.
Some limitations include that the performance of Complifft is dependent on the quality of the underlying generative model. For example, if the model is not well-trained, it may not be able to generate any acceptable samples. An- other limitation is that while we tested all algebras in the 2D dataset, we did not have a sysmatic test on OR/NOT al- gebras for text-to-image generation due to a lack of existing mature benchmarks for OR/OOT algebras. would also be interesting use segmentation models like Grounded Segment Anything (Ren et al., 2024) to provide grounded evaluation of the accuracy of the Compl.ift criterion.
Brin, S., Motwani, R., Ullman, J. D., and Tsur, S. Dynamic itemset counting and implication rules for market basket data. In Proceedings of the 1997 ACM SIGMOD interna- tional conference on Management of data, pp. 255-264, 1997.
Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L.,, and Cohen-Or, D. Attend-and andi-exiieee Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4): 2023.
In the future, we plan to explore the potential of the Com- pLift criterion on more complex compositional tasks, such as video generation and music generation.
Chen, H., Dong, Y., Wang, Z., Yang, X., Duan, C., Su, H., and Zhu, J. Robust classification via a single diffusion model. arXiv preprint arXiv:2305.15244 2023.

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Chen, M., Laina, I., and Vedaldi, A. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision. pp. 3343-5353, 2024.
Karthik, S., Roth, K., Mancini, M., and Akata, Z. If first you don't succeed, try, try again: Faithful diffusion-based texl-to-image generation by selection. arXiv preprint arXiv:2305.1333 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 87844, 2021.
Kim, D., Kim, Y., Kwon, S. J., Kang. W., and Moon, I.- Refining generative process with discriminator guid- ance in basee diffusion models. arXiv preprint arXiv:2211.1709) 2022.
Du, Y., Durkan, C., Strudel, R., Tenenbaum, B., Diele- man, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. S. Reduce, reuse, recycle: Composi- tional generation with energy-based diffusion models and memc. In International conference on machine learning, pp. 8489-8510. PMLR, 2023.
Kong, X., Liu, O., Li, H., Yogatama, D., and Steeg, G. Interpretable diffusion via information decomposition. arXiv preprint arXiv:2310.07972 2023.
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E., and Pathak, D. Your diffusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2206-2217, 2023.
Feng, W. He, X., Fu, T.-.,, Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X. E., and Wang, W. Y. Training-free free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05033 2022.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable dif- fusion models. In European Conference on Computer Vision, pp. 223-439. Springer, 2022.
Feng, W. Zhu, W., Fu, T.-... Jampani, V., Akula, A., He, X., Basu, S., Wang, X. E., and Wang, W. Y. Layout- gpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Liu, Y., Zhang, Y., Jaakkola, T., and Chang, S. Correcting diffusion generation through resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8713-8723, 2024.
Ho, J. and Salimans, Classifier-free free diffusion guidance. arXiv preprint arXiv:2207.12550 2022.
Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.-.., Zhang, M., Yang. X., Li, Y., Jaakkola, T., Jia, X., et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840 6851, 2020.
Mirsky, Y. and Lee, W. The creation and detection of deep- fakes: A survey. ACM computing surveys (CSUR), 54(1): 1-41, 2021.
Hong, C., Cha, B., and Oh, T.-H. Cas: A probability- based approach for universal condition alignment score. In The Twelfth International Conference on Learning Representations, 2024.
Na, B., Kim, Y., Park, M., Shin, D., Kang, W., and Moon, I.-C. Diffusion rejection sampling. arXiv preprint wwXh:2405.17888. 2024.
Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., and Smith, A. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20406-20417, 2023.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMIR, 2021.
Huang. K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i- compbench: A comprehensive benchmark for worndworld compositional text-to-image -moge generation. Advances in Neu- ral Informatonn Processing Systems, 36.78723-78747, 2023.
Podell, D., English, Z., Lacey, K., Blattmann, A.. Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307. 00950. 2023.
Qi, 7.,, Bai, L.,, Xiong, H., and Xie, 7. Not all noises are cre- ated equally: Diffusion noise selection and optimization. arXiv preprint 2rXiv:2407.14041 2024.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 29010 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural
10

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
language supervision. In International conference on machine learning, pp. 8748-8760 PMLR, 2021.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L.,, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00 007 14, 2024.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprini arXiv:240..14159 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b; An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294, 2022.
Sohl-Dickstein, Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference machine learming, pp. 2256-2265. PMLR, 2015.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A.. Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Wang, R., Chen, Z., Chen, C., Ma, J., Lu, H., and Lin, X. Compositional text-to-image synthesis with attention map control of diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5544-5552, 2024.
Wu, Q., Liu, Y., Zhao, H., Bui, T., Lin, Z., Zhang, Y., and Chang, S. Harnessing the spatial-temporal attention of diffusion models for high-fidelin text-to-image synthesis. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 7766-7776, 2023.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating hu- man preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Xu, Y., Deng, M., Cheng, X., Tian, Y.. Liu, Z., and Jaakkola, T. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 36: 76806-76838, 2023.

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
In this appendix, we provide additional details on the experiments and results presented in the main paper. Section A discusses the connection between our method and Classifier-Free Guidance (CFG) (Ho & Salimans, 2022; Liu et al., 2022). Section B describes the Compose algorithm used in our experiments. Section C and D provide the full details of the vanilla and cached versions of the CompLift algorithm for text-to-image generation. Section E, Section F, and Section G provide additional results for 2D synthetic distribution compositions, the CLEVR Position Task, and text-to-image generation, respectively.
A. Connection with Classifier-Free Guidance (CFG) (Ho & Salimans, 2022; Liu et al., 2022)
A brief summary is that our method tries to solve the primal form Equation (6), while CFG tries solve the dual form using soft Lagrangian regularization to satisfy the constraint as Equation (9). The two formulations are equivalent in the Lagrangian sense, but CFG is not guaranteed to strictly satisfy the constraints in practice. Though in practice, we will lose the theoretical guarantee of CompLift using ELBO estimation, but we hope this theoretical perspective can help the readers understand how Complift aims to directly solve the primal form.
We provide rough proof follows. The formulation of the final probability can be regarded constrained distribution:
2g ~ Pgenerator s.t. p( cx) > p(zo), Vc,.
(6)
Now we show that CFG (Ho & Salimans, 2022), or Composable Diffusion (which uses CFG in the compositional generation context) (Liu et al., 2022), try to satisfy the constraint using soft regularization. For convenience, we take the logarithm (assuming all probabilities are strictly positive). Ignoring the boundary condition, this transforms the constraint p(z0 c₂) p(zo) into:
log p c.)
log ((zo) 2
Vc,.
Suppose our goal to maximize the log-likelihood of the generator while ensuring the above constraints are met. The optimization problem can be written as:
max soo
log Ppenerator((FD)
(8)
subject to logo c.) p(o)(()))))))))))))))))))))))p 2 0, Veg.
To handle the constraints, introduce non-negative Lagrange multipliers A 0 for each constraint. The Lagrangian function is then defined as:
L( A) = log Pgencrator (Do) + Σ A loggp(oo (.) - log bogp)))). A 2 0.
(9)
C:
In this formulation:
The first term, log Prencrator (xo), is our original objective.
The second term penalizes any violation of the constraint log p(æ0 cs) - p(ao)) 2 0. If the constraint is violated, the penalty term will lower the overall value of the Lagrangian.
We assume the derivative of log po at æ, is proportional to c0 Ø):
Vz, logpo((x)) Re V., Ee, ((e Ø) - +) x c0(a,, Ø).
(10)
Consequently, assuming that Pgenerator (xo) FJ pe (a Ø), then the derivative of the Lagrangian at æ;;
x; L(wo, A) ea( Ør, Ø) Σ A (Ee(())) - eø ca 2)).
A; Σ 0,
(11)
which derives the exact CFG-like formulation as Equation 11 in Composable Diffusion (Liu et al., 2022).
Some notes:
12

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
CFG Composable Diffusion fix the Lagrangian coefficient X fixed weight value w. Yet, it unknown whether the constraints are satisfied indeed.
In sect-to-image tasks, CompLift uses Pgeuerator as Pe ( Ccompose), while CFG / Composable Diffusion use Pyenerator (2)) as po (o Ø). Using po (To; Ceompuse) in Complift increases the probability of the samples be accepted in practice. It would be interesting test whether changing the Pgenerator (3g) pe (Fu Ceompose) in CFG Composable Diffusion can also improve the performance.
13

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
B. Compose Algorithm
describe the Compose algorithm used our experiments in Algorithm 3. The algorithm takes as input a set lift scores (lift(Znp Y== and an algebra A, and returns boolean value indicating whether the composed prompt is satisfied. The algorithm first initializes dictionary to store partial results. It then converts the algebra A conjunctive normal form (CNF) and iterates each disjunction A& in A. For each literal Lm in As, the algorithm calculates the lift score based on the input lift scores and updates s [A.]] Finally, it returns whether the minimum of the results is greater than zero
Algorithm 3 Compose for multiple algebraic operations
Input: lift scores (lif((zolc,)) ))) algebra A {}
2: Initialize
Dictionary to store partial results
3: Convert A to conjunctive normal form (CNF)
4: for disjunction A& E A do
5: Initialize s s[AAa] = 0
6: for literal fm E A& do
7: if L. of form e; then
8: s[Ap]] max( c.), s(Aa))
9:
else if m. of form -c then
10: s[AA}] max(-lift(zoll))) s(An))
11: end if
12: end for
13:
end for
14:
return minA.E
C. CompLift for Text-to-Image Generation (Vanilla Version)
We combine Algorithm 1, Equation (4), and Equation (5), and provide the full algorithm of CompLift for t2i tasks.
Algorithm 4 Text-to-Image Generation with CompLift
1: Input: image Øo, conditions {c,) composed prompt Coompose- # of trials T, threshold r
2: Initialize Pixellis ft (cs] list() for each c;
3: Encode 2o to latent Zo using VAE encoder
4: for trial j 1,..., T do
5: Sample ~ [1, 1000]; e 1)
6: 20.20 + v1 āac
7: ccompoce = c0 Ceompace)
D- Get composed prediction
8: for condition C E {c) do
9: (2ty(2)lc)) compose (z, 2] composs c0(2,, 12
PixelLift (xxelLift(cxlapaee (apppeeaa.appp (zolcx))
11: end for
12: end for
13: for condition C₁ E [c do
14: lifi(2o,C4) = mean(PixelLift [t(xf((()
Average over trials
15: lift(Zo, Ct.) = Σ
T(lif(zo c+ (anl) > 0)
a.
16: end for
17: return Compose ((lit(( |c))) A)
> Run Algorithm 3
14

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
D. Cached CompLift for Text-to-lmage Generation (Cached Version)
Similar to the vanilla version, we combine Algorithm 2, Equation (4), and Equation (5), and provide the full algorithm of Cached LifpLift for t2i tasks. The algorithm divided into two parts: Algorithm 5 and Algorithm The first part caches the latent vectors and predictions at each timestep, while the second part calculates the lift scores using the cached values.
Algorithm 5 Cache Values During text-to-Image Generation
1: Input: conditions {c:)-r composed prompt Ceomposes # of timesteps
2: Initialize zT
(0.
3: Initialize Caehe = ()
Dictionary store intermediate values
4: for timestep E [1N, IN [1] do
5: > Standard generation process
6: -t;-; = step(zt,, Eg, Coompose)
7: > Add predictions to cache 21 9: Cachelt)J.null.ppt = EZ(,, R)
8: Cache|t,)
10: for condition C E (c))._ do
11:
Cache(t,)-comd pred(cu) c (2t,, Ck)
12: end for [acce(ty).cooppl pred = c0 (Zzz, Crompuse
13:
end for
15:
Decode 2n to image By using VAE decoder
16:
return Zo, Cache
Algorithm 6 CompLift Using Cached Values
Input: image Øo, conditions {c Cache, threshold
2: Initialize PixelIA ft [cs] list() for each c;
3: Encode zo to latent #o using VAE encoder
4: for timestep 'j in Cache do
5: 21 [
6: Ccompose = Cache (tccchettt..cmpelll cooposs pred
7: for condition Ca (c do
8: lift, (x))
[cooppuee Cacheet,].nul1. pred|] compusse Cachejty].cond pred[c.]]|
9: PixelLifte ft [cx] (tcefftcccpppeeeeee appenddlif,, (2olcc))
10: end for
11: end for
12: for condition Cs E [c do
13: lift(zo, Cx) = meean(iieeliiffppppp..
D Average over timesteps
14: lift(Zo, C+. = Σ Illif((zo. cx) > 0) -
a.be[l.m
15: end for
16: return Compose ((lpoos (l 1
> Run Algorithm
15

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
E. 2D Synthetic Distribution Compositions
(x)
Py(x)
Ground Truth
Composable Diffusion
Compláft
x
X
X
Figure 11. 2D synthetic result for product composition with Composable Diffusion and CompLift.
(x)
P2(x)
Ground Truth
Composable Diffusion
CompLift
:
+
+
+
Figure synthetic result mixture composition with Composable Diffusion and CompLift.
(x)
Px(x)
Ground Truth
Composable Diffusion
CompLift
Figure 13, 2D synthetic result for negation composition with Composable Diffusion and CompLift.
16

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Ground Truth
ULA
U-HMC
MALA
HMC
Figure 14. 2D synthetic result for product composition with Encrgy-Based Models and MCMC. The component distributions are the same ones Figure!
Ground Truth
ULA
U-HMC
MALA
HMC
:
:
:
Figure 15. 2D synthetic result for mixture composition with Energy-Based -Based Models and MCMC. The component distributions the ones in Figure 12.
Ground
U-HMC
HMC
Figure 16. 2D synthetic result for negation composition with Energy-Base Based Models and MCMC, The component distributions are the same ones in Figure 13.
17

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
In Figures 11 to I6, we provide additional visualizations of the 2D synthetic distribution compositions. The figures show the generated samples for product, mixture, and negation compositions, respectively. We find that the summation result is slightly different from the visualization result in Du et. al (Du et al., 2023), which is likely due to the difference of the temperature used for the composed distribution. We simply set temperature as the default 1 for the composed distribution of the summation algebra, while Du et. al (Du et al., 2023) sets a temperature as 3.5 (see the implemen- in https: : /github. com/ yi myluundu/ai reducee searrerrruuueeeeeeeeeee
tation of MixturEBMMiiffsioooMoon walnn/notebooe / s immll_eee _i strrrtiiiuu intrruutioon composed distribution can achieve in practice.
/blob/
ipynb). We think our choice may better represent what the original
Table 4 shows the detailed results for the 2D synthetic distribution compositions. We provide the accuracy (Ace), Chamfer distance (CD), and acceptance ratio (Ratio) of samples using the CompLift algorithm for each composition. We find that Composable Diffusion enhanced with CompLift (T' = 1000) consistently outperforms baseline methods across all scenarios, achieving perfect or near-perfect accuracy in many cases. While all methods perform reasonably well with Product algebra, performance generally degrades with Mixture and especially with Negation algebra, where CompLift still maintains a significant edge over other approaches. The results demonstrate that Compl.ift effectively improves both accuracy and sample quality (measured by Chamfer Distance), though this comes with varying sampling efficiency as shown by the acceptance ratios.
Figure 17 shows the histograms of the Compl.ift scores for each composed 2D synthetic distributions following the order in the Figure 11, Figure 12, and Figure 13, where the groud-truth positive data points are in color blue, and the negative data points are in color orange. We observe that ! positive data samples have a higher CompLift score and negative data samples (i.e., the data sample out of the desired distribution) have a lower CompLift score, and @ the boundary condition of is a good threshold to separate the positive and negative data samples for product. For mixture and negation, the boundary condition is less effective and slightly conservative, but the positive data samples still have higher CompLifi score than the negative data samples, and the performance still achieves better when applying CompLift to the baseline methods.
Histog ram of CompLitt Scores for Product
Histogram of CompL//t Scores for Product
Histogram of Complitt Scores for Product gt positive
1750
gt positive
5000
gt positive
#Data Points
1500
1250
gt negative
#Datt Points
4000
negative
Data Points
gt negative
2000
3000-
Product
750
2000
500
50
1000
-2.9
-1.5
-10
-0.5
0.0
-7
-6
-5
-4
-3
CompLift Score
CompLit Score
CompLift Score
16-1
Histogram of Complift Scores for Mixture A
Histogram of CompLift Scores Mixture
Histogram of Complift Scores Mixture
150
gt positive
1209
gt pasitive
5000
gt positive
#Data Points
125
g: negative
*Data Points
1000
gt negative
*Data Points
4000
gt negative
100-
800
2000
Mixture
75
600
2000
50
400
1000
-1.75 -1. 50 -125 10.00-1..5 75-00-1.--0050 -0.25
CompLift Score
1e
CompLift Score
lc 1
CompLit Score
16 1
Histogram of CompLift Scores for Negation
Histogram of CompLiit Scores for Negation B
Histogram of Complift Scores for Negation C
1400
gt positive
1400
gt positive
7000
positive
#Data Points
100e
nogative
#Data Points
1200
3000
gt negative
DData Points
6000
5000
gt negative
eoc
Negation
800
600
600
3000
400
460
2000
200
1000
1.0 CompLitt Score
0.4
CompLlt Score
CompLint Score
Figure 17. Histograms of the CompLift scores for 2D synthetic distribution compesitions. The histograms show the distribution of the lift scores for the three compositions. The samples are 8000 samples generated from Composable Diffusion. The x-axis represents the composed lift scores using Table 1, and the y-axis represents the number of samples.
18

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Algebra
Method
Scenario 1
Scenario
Scenario 3
Acc
CD
Ratio
Acc t
CD /
Ratio
Acc 1
CD
Ratio
Composable Diffusion
81.9
0.030
87.6
0.093
0.0
EBM (ULA)
80.3
0.010
74.7
0.141
0.0
EBM (U-HMC)
84.9
0.007
83.9
0.065
0.0
Product
EBM (MALA)
95.8
0.005
92.0
0.103
0.0
EBM (HMC)
95.7
0.003
98.8
0.039
0.0
Composable Diffusion
98.5
0.005
100.0
0.013
100.0
+ Cached CompLifi
Composable Diffusion
97.4
0.011
100.0
0.019
100.0
CompLift (T 50)
Composable Diffusion
99.8
0.005
79.3
100.0
0.013
55.1
100.0
0.0
+ Lifp (T 1000)
Composable Diffusion
22.1
0.067
90.0
0.025
99.9
0.033
EBM (ULA)
28.2
0.062
87.9
0.021
99.2
0.049
EBM (U-HMC)
30.5
0.060
89.5
0.020
99.5
0.108
Mixture
EBM (MALA)
28.4
0.059
92.9
0.018
100.0
0.030
EBM (HMC)
28.7
0.059
92.3
0.019
100.0
0.157
Composable Diffusion
61.0
0.030
98.5
0.022
100.0
0.018
+ Cached CompLift
Composable Diffusion
95.7
0.020
100.0
0.041
100.0
0.026
+ Complift iift (T 50)
Composable Diffusion
100.0
0.017
9.2
100.0
0.038
56.8
100.0
0.014
57.1
CompLifi (T 1000)
Composable Diffusion
11.8
0.191
11.7
0.222
0.0
EBM (ULA)
28.7
0.129
5.6
0.244
0.0
EBM (U-HMC)
25.9
0.206
2.3
0.333
0.0
Negation
EBM (MALA)
17.8
0.150
2.7
0.257
0.0
EBM (HMC)
12.4
0.258
0.2
0.421
0.0
Composable Diffusion
40.4
0.152
45.6
0.123
100.0
+ Cached CompLift
Composable Diffusion
68.6
0.150
56.4
0.158
100.0
+ CompLift (T = 50)
Composable Diffusion
78.7
0.127
3.2
57.2
0.135
6.3
100.0
0.0
+ CompLifi (T 1000)
Table 4. Quantitative results on 2D composition. Ace means accuracy, CD means Chamfer Distance, and Ratio means the acceptance ratio of samples using CompLift.
E.1. Parameters of Distributions
The distributions follow the generation way in Du et. al (Du et al., 2023) they are either Gaussian mixtures or uniform distribution. We sample 8000 data points randomly for each component distribution, and train 1 diffusion model for each distribution. We define that sample satisfies a uniform distribution it falls into the nonzero-density regions. A sample satisfies Gaussian mixtures if it within 30 of any Gaussian component, where is the standard deviation of the component. The negation condition is satisfied, if the above description is not satisfied.
Product, Component A1: A mixture of 8 Gaussians evenly placed circle of radius 0.5 centered the origin. Each Gaussian has standard deviation of 0.3.
Product, Component A2: uniform distribution vertical strip: x 1-0.1, [-1,1].
Product, A1 A A2: Two Gaussian blobs centered at (0, 0.5) and (o, -0.5), each with standard deviation 0.3, The accurate samples are those that fall within 30 of either Gaussian blob.
19

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Product, Component B1: Uniform samples along a circle of radius centered at (-0.5, 0).
Product, Component B2: Identical B1 but centered at (0.5, 0).
Product, B1 A B2: Two deterministic point masses at (0, ) (i.e., zero-variance Gaussians), Since the distribution
has 0 area of the support region, we define the accurate samples as those that fall within the distance of 0.1 to either point mass.
Product, Component Cl: Uniform distribution in a left rectangle: x E [-1, -0.5), [-1,1].
Product, Component C2: Uniform distribution in a right rectangle: x E [0.5, 1], E [-1,1].
Product, C1 A C2: Empty dataset (no samples are generated for the composed case).
Summation, Component A1: Three Gaussians centered at (-0.25, 0.5), (-0.25, 0), and (-0.25, -0.5), each with standard deviation 0.3.
Summation, Component A2: Three Gaussians centered at (0.25, 0.5), (0.25, 0), and (0.25, -0.5,, each with standard deviation 0.3.
Summation, A1 V A2: Union of all six Gaussians from A1 and A2, each with standard deviation 0.3. The accurate samples are those that fall within 3a of any Gaussian component.
Summation, Component B1: Uniform distribution in left rectangle: x E [-1,- [-1,
Summation, Component B2: Uniform distribution in right rectangle: E [0.5, 1), [-1,
Summation, B1 V B2: Union of samples from both rectangular regions. The accurate samples are those that fall on the support region of either rectangle.
Summation, Component C1: Uniform samples along a unit circle centered at (-0.5, 0).
Summation, Component C2: Identical to C1 but centered at (0.5, 0)-
Summation, C1 V C2: Union of both circular distributions. Since the distribution as 0 area of the support region, we define the accurate samples as those that fall within the distance of [0.9, 1.1] to either center of the circle.
Negation, Component A1: Uniform distribution over square: y E [-1,
Negation, Component A2: Uniform distribution over the center square: x, [-0.5, 0.5).
Negation, A1 A- A2: Union of four rectangular corner regions (i.e., all of Al excluding A2). The accurate samples are those that fall on the support region of either rectangle corner.
Negation, Component B1: Uniform distribution over square [-1, [-1,1].
Negation, Component B2: Uniform in vertical strip: E [-0.5,0.55 [-2,
Negation, B1 A- B2: Union of two vertical slabs: -05 and [0.5, 1]. y [-1,1]. The accurate samples are those that fall the support region of either slab.
Negation, Component C1: Uniform in center square: x, y E 0.55,0.55.
Negation, Component C2: Uniform in larger square: z, [-1,
Negation, C1 A- C2: Empty dataset (intended to represent the subtraction of C1 from C2, but not explicitly sampled).
E.2. Notes on Why We Use Chamfer Distance as the Metric
The reader might ask why choose Chamfer Distance as opposed to other metrics, e.g., Kullback-Lebbler divergence, Wasserstein Distance, others. We summarize the reasons as follows:
We chose Chamfer Distance because it (1) applies to uniform distributions and (2) is sensitive to out-of-distribution samples. KL inapplicable for oft-of- distribbutinnn samples with undefined density ratio in uniform distribution settings. Wasserstein Distance more robust to outliers, which doesn't meet our requirement to sensitively capture unaligned samples.
20

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
F. CLEVR Position Task
Method
Combinations
Method
Combinations
Composable Diffusion
100.0 100.0 99.0
90.8 78.7
Composable Diffusion
37.6
37.7 38,8
48.8
50.1
EBM (U-HMC)
82.0
66.0
34.0
24.0 11.0
EBM (U-HMC)
158.9
139.7
111.3
101.4
84.4
EBM (ULA)
86.0
74.0
48.0
27.0
19.0
EBM (ULA)
180.9
153.1
122.4
103.9
84.5
Composable Diffusion
Composable Diffusion
100.0 100.0
99.7 91.6 84.3
36.8
36.2
36.4
50.2
57.3
Cached CompLift
+ Cached CompLifi
Composable Diffusion 100.0 100.0 99.9 97.5 90.3
Composable Diffusion
38.2
36.4
40.1
53.2
56.2
CofpLift
+ CompLijf
Table 5. Quantitative accuracy results on CLEVR compositional generation (higher better).
Table 6. FID scores on CLEVR compositional generation (lower is better).
Original Image
Segmentation Mask
Object Mask
SAM2 Labels
Pretrained Classifier
98%
Composable Diffusion Shape
99%
CompLift
0%
Composable Diffusion w/ 3 Shapes
63%
96%
CompLiff
0%
Composable Diffusion w/5 5 Shapes
0%
9999 %
CompLift
Figure 18. More examples of samples on CLEVR Position dataset. We find that the SAM verifier is more robust and generalizable compared the pretrained classifier provided by the Composable Diffusion work (Liu et al., 2022). From left right: the first column shows the original images, the second column shows the segmentation masks generated by SAM2, the third column shows the object masks (by excluding the pixcls from the background mask), the fourth column shows the labels given by SAM2, and the fifth column shows the labels given by the pretrained classifier. The percentage numbers represent the confidence score predicted by the classifier.
21

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
We Segment Anything Model 2 (SAM2) (Ravi al., 2024) generalizable verifier, check whether an object is in specified position. We first extract the background mask, which is the mask with the largest area using the automatic mask generator of SAM2 (SAM2Aut omat McMas
then label a condition as satisfied if the targeted position is not in the background mask. We find this new verifier is more robust than the pretrained classifier in Liu et al. (2022). As shown in Figure 18, the pretrained classifier often fails to generalize when more shapes are required to be composed, while the SAM2 verifier can effectively detect the samples due to the fact that it is pretrained with massive segmentation dataset.
For the MCMC implementation. we use the same PyTorch samplers shared by Du et al. (Du et al., 2023) '. However, we find hard reproduce the result of the MCMC method on the CLEVR Position Task. The generated objects in the synthesized images are often in strange shapes that are dissimilar from the trained data. We suspect that the MCMC method may require more careful tuning of hyperparameters, as not all hyperparameters are mentioned as the optimal ones in the codebase. Therefore. we remind the readers that the results of the MCMC method on the CLEVR Position Task in this section should be interpreted with caution they mainly serve as references and do not represent the optimal results.
Nevertheless, we are still interested in how the methods perform using the pretrained classifier from Liu et (Liu et al., 2022). In Table 7, we find that the Composable Diffusion model with CompLift consistently outperforms the baseline methods across all combinations, and achieves a nonzero accuracy in the most challenging object composition setting.
Method
Combinations
2
5
Composable Diftusion
52.7
20.5 3.1 0.3 0.0
EBM (U-HMC)
1.0
0.0
0.0
0.0 0.0
EBM (ULA)
6.0
0.0
0.0
0.0 0.0
Composable Diffusion
55.7 25.3
6.7 0.4 0.0
+ Cached CompLift
Composable Difflusion
56.2 25.3 5.7 0.1
+ CompLift
Table 7. Quantitative accuracy results on CLEVR using the pretrained classifier from Liu et al. (Liu et al., 2022).
Figure 19. Example of failed samples on CLEVR Position dataset using MCMC. The generated objects in the synthesized images are often in strange shapes which is dissimilar from the trained data. We suspect that the MCMC method may require more careful tuning of byperparameters. As result, we remind the readers that the results of the MCMC method on the CLEVR Position Task should be interpreted with caution.
1amppllll_papeee camplers.pp. ers.
22

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
G. Text-to-lmage Generation
G.I. More Examples on Correlation Between CLIP and CompLift
In Figures 10 and 20, we provide examples the correlation between CLIP and CompLift for text-to-image generation. The figures show the generated images for 3 prompts involving a black car and white clock, turtle with a bow and frog and a mouse. For these prompts, find that the diffusion models would typically ignore the clock, bow, and mouse components, leading to low CLIP scores for many generated images. Meanwhile, we observe strong correlation between the CLIP and CompLift scores for these prompts, indicating the effectiveness of CompLift for complex prompts. For the images with low CLIP scores, they typically have lower number of activated pixels using CompLift for these ignored objects.
One note that for the a turtle with a bow prompt, we observe that most of the images would fail to generate the bow component. This explains why many data samples have low CLIP scores on the left part of Figure 20, However, the CompLift algorithm can effectively capture the bow component, and reject those samples with low CLIP score.
Pearson Correlation: 0.8572
Rejected
Rejected
Pearson Correlation: 0.6763
Rejected
Rejected
0.300
0.28
0.275-
0.26
CLIP Score of AA Bow
CLIP Score of Mouse
0.250
0:24
0.225
0.22
Accepted
Accepted
Accepted
Accepted
0.200
0.20
0.175
0.18
0.150
0.16
500
1000
1500
2001
400
ComppLit Activated Pixels of Bow
#CompLift Activated Pixels of A Mouse
Figure 20. More examples correlation between CLIP and CompLift. Prompt to generate left images: a turtle with a bow. Prompt to generate right images: frog and mouse. Text in blue indicates the object components to compose.
G.2. Ablation Study: Improvement on & Excite Generator
We conducted new experiment using Attent-and-Exett (Chefer al., 2023), focusing on the additional improvement achieved by incorporating CompLift. We observed consistent performance gains with both SD 1.4 and SD 2.1, Note that SD XL is not included due to the lack of support in the original Attent-and-Exen code.
Method
Animals
Object& Animal
Objects
CLIP 1
IR
CLIP t
IR
CLIP 1
IR 1
A&E (SD 1.4)
0.330
0.831
0.357
1.339
0.357
0.815
A&E (SD 1.4) Cached CompLift
0.338
1.156
0.361
1.469
0.362
0.934
A&E (SD 1.4) CompLifi (T=200)
0.337
1.160
0.361
1.458
0.361
0.990
A&E (SD 2.1)
0.342
1.225
0.360
1.471
0.366
1.219
A&E (SD 2.1) + Cached CompLift
0.344
1.298
0.364
1.488
0.371
1.245
A&E (SD 2.1) + CompLifi (T=200)
0.346
1.337
0.365
1,516
0.370
1.246
Table 8. CLIP and IR scores across categories for Attend-and-Exen (A&E) with CompLift variants.
G.3. Ablation Study: Advantage of Compositional Criteria
One concern is that the compositional acceptance / rejection task can also be framed using single criterion that works directly the whole prompt, as addressed by CAS (Hong et al., 2024). To test how CAS-like variant performs for prompts containing multiple objects, we have conducted a new ablation study.
Here, CAS-like variant means that we use the single criteria logp(> Coompose) p(x Ø) as the latent lift score, which replaces the composed criteria from multiple individual lift scores Compl.ift. Note that this is controlled experiment to
23

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
check the advantage of compositional criteria, thus, keep the same estimation method using ELBO.
We provide Table the result. We observe only modest improvement when using the CAS-like variant. We hypothesize that CAS-like variant might face similar problem as the original Diffusion Model for multi-object prompts the attention to the missing object is relatively weak in the attention layers. More similar discussions can be found in previous works such as Mtend-andd-xxite (Chefer et al., 2023), where diffusion model co (x, Ceompose) sometimes ignores some condition ci
in Ceompose
Method
Animals
Object&Animal
Objects
CLIP
IR f
CLIP t
IR t
CLIP
IR f
SD 1.4
0.310
-0.191
0.343
0.432
0.333
-0.684
SD 1.4 CAS Variant
0.312
-0.153
0.348
0.708
0.337
-0.373
SD 1.4 + Complift (T=200)
0.322
0.293
0.358
1.093
0.347
-0.050
SD 2.1
0.330
0.532
0.354
0.924
0.342
-0.112
SD 2.1 + CAS Variant
0.333
0.626
0.355
1.080
0.347
0.144
SD 2.1 + CompLift (T=200)
0.340
0.975
0.362
1.283
0.355
0.489
SD XL
0.338
1.025
0.363
1.621
0.359
0.662
SD XL + CAS Variant
0.338
1.064
0.363
1.628
0.362
0.702
SD XL CompLift (T=200)
0.342
1.216
0.364
1.706
0.367
0.890
Table Performance comparison (CLIP and IR) across categories for SD backbones, with CAS Variant and CompLift.
G.4. More Examples on Accepted and Rejected Images
In Figures 21 and 22, we provide additional examples of accepted and rejected images using SDXL CompLift. We observe that the CompLift algorithm can effectively reject samples that fail to capture the object components specified in the prompt in general. In addition, we find that the current CompLift criterion is relatively weak in determining the color attribute, leading to some accepted images with incorrect colors (e.g., a green backpack and a purple bench). This suggests that future work could explore more sophisticated criteria to further improve the quality of generated images.
24

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
white bow and white
purple chair and
horse and mouse
yellow backpack and gray apple bow
elophant with
and elephant
turtle and mouse
bear and mouse
bird and horse
elephant and rabbit
a green backpack and purple bench
red glasses and red suitcase
an elephant and turtle
frog and rabbit
gray backpack and yellow glasses
cat
gray crown and white clock
with bow
and mouse
red backpack and yellow bowl
bird with glasses
lion and elephant
elephant and pink backpack
bird and yellow
horse and rabbit
Figure 21, More examples of accepted and rejected images using SDXI. Compl-ift for text-to-image generation.
25

---

Improving Compositional Generation with Diffusion Models Using Lift Scores
purple and pink apple
white glasses and orange balloon
green balloon and pink bowl
white chair and gray balloon
yellow glasses and
blue balloon and blue
bird
turtle and gray backpack
white bow and black clock
lion and mouse
dog and rabbit
pink crown and purple bow
and
black backpack
11
mouse
suitcase and
dog and frog
frog and à turtle
gray and purple apple
a frog and pink bench
and
à red
white chair
yellow suitcase and yellow
yellow glasses and gray bowl
horse with bow
monkey and mouse
black and pink balloon
backpack
lion and horse
Pigure 22, More examples of accepted and rejected images using SDXI. Compl-ift for text-to-image generation.
26

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
H. Q&A during Rebuttal
During the rebuttal, we have received some questions from the reviewers. We appreciate the valuable feedback from anonymous reviewers, and we incorperate our answers in the rebuttal follows:
Q: Though CompLift shows significant improvement over Composable Stable Diffusion synthetic datasets, the improvement on real-world text-to-image generative model is trivial as shown in Table 3.
A: The seemingly trivial CLIP improvement is due to the low magnitude of CLIP scores. Here, we provide another perspective to interpret the numbers, We compare the CompLift selector to the perfect best-of-n selector, which has direct access to the metric function. The percentage gain is calculated as (CompLift metric baseline metric) (perfect selector metric baseline metric). On average, the gains are 40% for vanilla Compl Lift and 30% for cached Compl.ift.
Method
Animals
Pojjct&Annmin
Objects
CLIP gain% t
IR gain% t CLIP gain% t IR gain% t CLIP gain% t
IR gain%
SD
Cached CompLift
31.58
30.58
42.62
62.74
37.04
54.99
SD 1.4
CompLift (T=200)
42.11
46.40
49.18
74.32
47.14
63.05
SD 2.1
Cached CompLift
35.71
44.13
28.34
55.07
37.97
57.82
SD 2.1
CompLift (T=200)
39.68
56.18
32.39
60.28
41.14
74.73
+ Cached Complift
16.95
55.88
26.13
46.15
24.49
46.06
SD XI. + Complift (T=200)
22.60
48.74
26.13
59.44
32.65
44.88
Table 10. CLIP and IR gain (%) across categories with CompLift variants.
Q: Would it be feasible to evaluate with the TIFA score as in Karthik et al (Karthik et al., 2023; Hu et al., 2023)?
A: New TIFA experiments show improvements across categories:
Method
Animals TIFA Object&Animal TIFA 1
Objects TIFA 1
Stable Diffusion 1.4 SD 1.4 + Cached CompLift
0.692
0.822
0.629
0.750
0.886
0.685
SD 1.4 CompLiff (T=200)
0.794
0.902
0.682
Stable Diffusion 2.1 SD 2.1 Cached CompLift
0.833
0.873
0.668
0.905
0.911
0.731
SD 2.1 CompLift (T=200)
0.927
0.912
0.726
Stable Diffusion XL
0.913
0.964
0.755
SD XL + Cached CompLift
0.949
0.972
0.790
SD XL + CompLift (T=200)
0.946
0.974
0.782
Table 11. TIFA results across categories. Compl.ift variants show consistent improvements over baselines.
27

---

Improving Compasitional Generation with Diffusion Models Using Lift Scores
Q: For assessing alignment via CLIP do you the entire prompt (including all conditions) assess the CLIP alignment each condition individually? I believe it's the former but do you think they latter might work better? (do you have any evidence this?)
A: We used entire prompt. We add new experiment with minCLIP (minimum CLIP score across subjects) in Table 12. Similar performance gains were observed, indicating CompLift primarily improves the weaker condition (typically missing object).
Method
Animals 1 Object&Animal minCLIP
Objects minCLIP
Stable Diftusion 1.4 1.4 + Cached CompLift 1.4 + Complijf ift (T=200)
0.218
0.248
0.237
0.225
0.260
0,249
0.228
0.263
0.252
Stable Diffusion 2.1 2.1 + Cached CompLift
0.237
0.258
0.247
0.248
0.265
0.260
SD 2.1 + CompLift (T=200)
0.249
0.265
0.261
Stable Diftusion XI.
0.243
0.269
0.264
XL Cached CompLift
0.248
0.271
0,269
XI. Lft (7==200)
0.250
0.271
Table 12. minCLIP scores across categories, showing improvements with CompLift variants.
Q: Do you have any way to assess whether CLIP vs ImageReward a more appropriate metric, and how well each of them actually checks whether all the conditions are present in the composition (e.g. for AND).
A: We manually labeled 100 samples from Figure 10 for presence of both black car and white clock. With 62 positive and 38 negative samples, we calculated metric performance in Table 13. CLIP and ImageReward perform similarly, both better than TIFA. CLIP slightly preferred due to imbalanced data.
Metries
CLIP
ImageReward
TIFA
ROC AUC
0.949
0.955
0.857
PR AUC
0.972
0.968
0.901
Table 13. ROC AUC and PR AUC for CLIP, ImageReward, and TIFA the correspondence of the presence of both black and white clock. Experiments were conducted on 100 sampled images from Figure 10.
28


Reviews:
[{"review_id": "yReeqFLJYw", "reviewer": "Reviewer_25c6", "summary": "This paper proposes a training-free post-processing approach, CompLift, to select images with specified concepts from diffusion model-generated image candidates. The main idea is to use the lift score, which is equivalent to point-wise mutual information, to evaluate if conditioning $c$ reduces uncertainty of variable $\\textbf{x}$. As a post-processing approach, the performance of CompLift hinges on the generative model (i.e., composable diffusion model) it is based on. If composable diffusion model cannot generate accurate images at all, then CompLift cannot make any improvement. Experimental results show improved generation accuracy by using the proposed approach.\n\n### Most of my concerns are addressed. I maintain the rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions. We address your concerns below:\n\n> On CompLift's dependence on underlying generative model\n\nWe agree and will add this theoretical limitation to our Conclusion. While theoretically CompLift cannot improve if the base method produces no accurate images, in practice even weak generators often improve with ≤5 candidate images.\n\n> On \"minimal computational overhead\" claim\n\nWe'll remove the ambiguous term \"minimal\" and claim only \"requiring no additional training.\" For text-to-image generation, the (n + 2) · T additional forward passes can be parallelized to reduce latency. Currently, generation takes ~15s and lift score calculation ~30s on a 4090 GPU, with GPU memory as the bottleneck. Ideally, we can further parallelize to the latency of O(1) forward pass given enough GPU memory.\n\n> On cached CompLift's accuracy-speed tradeoff\n\nEvery column in Fig. 5 has a corresponding accuracy row in Table 2 (Cached CompLift has T=50 by default). We'll add a footnote clarifying this. In practice, we perceive small accuracy regression on Mixture and Negation task when switching from vanilla CompLift to Cached CompLift. However, the accuracy remains significantly higher than other baselines. The tradeoff exists, but seems to be mild and acceptable given the substantial speed improvement.\n\n> On lift score equivalence to point-wise mutual information [1]\n\nWe'll update the Related Work section to reflect this equivalence. Our paper applies point-wise mutual information (PMI) as an acceptance/rejection criterion, focusing on missing-object cases and composing PMI for multiple objects.\n\n> On real-world text-to-image improvement\n\nThe seemingly trivial CLIP improvement is due to the low magnitude of CLIP scores. Here, we provide another perspective to interpret the numbers. We compare the CompLift selector to the perfect best-of-n selector, which has direct access to the metric function. The percentage gain is calculated as (CompLift metric - baseline metric) / (perfect selector metric - baseline metric). On average, the gains are ~40% for vanilla CompLift and ~30% for cached CompLift. We will add more explanation to the Appendix.\n\n| Method | Animals |  | Object&Animal |  | Objects |  |\n|--------|---------|---------|---------|---------|---------|---------|\n|        | CLIP gain% ↑ | IR gain% ↑ | CLIP gain% ↑ | IR gain% ↑ | CLIP gain% ↑ | IR gain% ↑ |\n| SD 1.4 + *Cached CompLift* | 31.58 | 30.58 | 42.62 | 62.74 | 37.04 | 54.99 |\n| SD 1.4 + *CompLift* | **42.11** | **46.40** | **49.18** | **74.32** | **47.14** | **63.05** |\n||||||||\n| SD 2.1 + *Cached CompLift* | 35.71 | 44.13 | 28.34 | 55.07 | 37.97 | 57.82 |\n| SD 2.1 + *CompLift* | **39.68** | 56.18 | **32.39** | **60.28** | **41.14** | **74.73** |\n||||||||\n| SD XL + *Cached CompLift* | 16.95 | **55.88** | **26.13** | 46.15 | 24.49 | **46.06** |\n| SD XL + *CompLift* | **22.60** | 48.74 | **26.13** | **59.44** | **32.65** | 44.88 |\n\n> On CompLift's compositionality\n\nOur approach is to (1) evaluate individual concepts separately, and (2) compose an acceptance/rejection criteria from these multiple individual criteria, as Algorithm 3. Thus, the CompLift criteria seems compositional from our perspective. We wish to mention Composable Diffusion [2] as an example to elucidate such a perspective. Essentially, the approach uses (1) individual score on each concept, and (2) compose the score from these multiple individual scores. Such a factorize-and-compose way helps reduce the hardness to comply with complex prompt. As shown in our experiments, it improves performance on complex multi-object generation by evaluating individual conceptual alignment before making a composed decision.\n\n> On limited algebraic operations for text-to-image\n\nWe acknowledge this limitation. While we tested all algebras in the 2D dataset, we found no existing mature benchmark for OR/NOT algebras in text-to-image generation. We'll note this in our Conclusion for future work.\n\n> On fair comparison with Composable Diffusion\n\nWe recognize the challenge in comparing these approaches. Composable Diffusion generates candidates with no internal selection mechanism, while CompLift is a post-hoc filter. They serve different purposes and are not mutually exclusive—CompLift can enhance generation models by leveraging semantic alignment for filtering results. Our main goal in the experiments is to show such an enhancement, instead of a direct replacement, of other baselines such as Composable Diffusion.\n\n[1] https://arxiv.org/abs/2310.07972\n\n[2] https://arxiv.org/abs/2302.11552", "claims_and_evidence": "The paper claims that \"as a novel resampling criterion using lift scores for compositional generation, requiring minimal computational overhead\". This assertion seems somewhat overstated, and the term \"minimal\" is ambiguous without a clear criterion. While in some cases, the cached strategy results in no additional computational overhead, this is not universally true.  For text-to-image generation, when replacing $\\epsilon$ with $\\epsilon_\\theta(z, c_{\\text{ccompose}})$, additional computational overhead (n + 2) · T forward passes is involved.", "methods_and_evaluation_criteria": "The proposed method make sense for the application.", "theoretical_claims": "No theoretical claims are provided in the paper.", "experimental_designs_or_analyses": "The paper states in Fig. 5 that the overhead introduced by the cached CompLift is negligible for the Composable Diffusion baseline (Liu et al., 2022). However, this experiment only shows running time without giving accuracy evaluation. It is not clear if it is trading accuracy performance with running time. It might be helpful to show both in a single figure.", "relation_to_broader_scientific_literature": "The proposed approach uses lift score as a criterion to evaluate if a concept appears in an image. The estimated lift score in equation (4) is actually equivalent to the point-wise mutual information discussed in equation (5) in [1]. The proposed approach can be seen as an application of point-wise mutual information in [1].\n\n[1] Kong, X., Liu, O., Li, H., Yogatama, D., and Steeg, G. V. Interpretable diffusion via information decomposition. arXiv preprint arXiv:2310.07972, 2023.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strength**\n\n1. The proposed approach is training-free and requires little or no additional computational resources by cache design.\n\n2. Extensive experiments are conducted and show improvements over baselines.\n\n3. The writing is smooth and easy to follow.\n\n**Weakness**\n\n1. Involving no extra training or guidance at inference time not only can be an advantage but also can be a limitation of the proposed model. This means that the proposed model cannot correct the generated images but can only select some from them. As a result, the performance of CompLift largely hinges on the generative model (like Composable Diffusion) it builds upon, because if Composable Diffusion does not generate accurate images, the CompLift cannot select accurate images from the generated images. \n\n2. Though CompLift shows significant improvement over Composable Stable Diffusion on synthetic datasets, the improvement on real-world text-to-image generative model is trivial as shown in Table 3. Again, this hinges on the performance of Composable Stable Diffusion that has limited ability to generate accurate multi-object images. \n\n3. Though called CompLift, the proposed approach itself is not compositional because it evaluate individual concepts separately. It is not evaluating a joint appearance of all concepts. For text-to-Image generation, only AND operation is considered by evaluating individual concepts.\n\n4. For text-to-image generation, when replacing $\\epsilon$ with $\\epsilon_\\theta(z, c_{\\text{ccompose}})$, additional computational\noverhead (n + 2) · T forward passes is involved. This contradicts the minimal computational overhead requirement claim, and should be discussed to avoid overclaim.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "It is somewhat unclear about fair comparison with Composable Diffusion Model. Consider that Composable Diffusion Model generates 5 images, and one of them is accurate. CompLift selects the accurate one with lift score. Then how to determine if Composable Diffusion Model or CompLift is more accurate?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nWwkd8SGU6", "reviewer": "Reviewer_kyUu", "summary": "This work proposes CompLift, a resampling criterion based on the concept of lift scores used to improve the compositional generation capabilities of pretrained diffusion models. CompLift approximates the lift scores with the diffusion modules noise estimation, without requiring any external reward modules to measure its alignment to the given condition. The authors additionally propose a caching technique for CompLift and achieves computationally effecient pipeline. Through evaluations on both simple synthetic generation and more complex text-to-image generation, the paper shows that CompLift leads to accurate compositional generation without additional training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We address your concerns as follows. \n\n> Relationship to CAS [1]\n\nThank you for pointing out this important related work, which we previously overlooked. We will add reference to this valuable work, and incorporate a discussion of CAS in the \"Related Work\" section of the revised version. A summary of the relationship is as follows:\n\nOur work can be seen as an extension of CAS, which investigates the potential of using CAS as a compositional criterion to decompose the alignment requirements of a complex prompt into multiple acceptance criteria. To approximate CAS, we employ ELBO estimation to reduce computational cost, as an alternative to the Skilling-Hutchinson estimator used in the original CAS paper. It would be interesting to explore how Skilling-Hutchinson-based estimation performs in a compositional setting. It may yield higher accuracy at the expense of greater computational overhead, which we plan to investigate in future work.\n\n> Comparison with Attent-and-Excite [2] on text-to-image generation\n\nThank you for the great question. We conducted a new experiment using Attent-and-Excite [2], focusing on the additional improvement achieved by incorporating CompLift. We observed consistent performance gains with both SD 1.4 and SD 2.1. Note that SD XL is not included due to the lack of support in the original Attent-and-Excite code.\n\n| Method | Animals |  | Object & Animal |  | Objects |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | CLIP ↑ | IR ↑ | CLIP ↑ | IR ↑ | CLIP ↑ | IR ↑ |\n| A&E (SD 1.4) | 0.330 | 0.831 | 0.357 | 1.339 | 0.357 | 0.815 |\n| A&E (SD 1.4) + *Cached CompLift* | **0.338** | 1.156 | **0.361** | **1.469** | **0.362** | 0.934 |\n| A&E (SD 1.4) + *CompLift* | 0.337 | **1.160** | **0.361** | 1.458 | 0.361 | **0.990** |\n||||||||\n| A&E (SD 2.1) | 0.342 | 1.225 | 0.360 | 1.471 | 0.366 | 1.219 |\n| A&E (SD 2.1) + *Cached CompLift* | 0.344 | 1.298 | 0.364 | 1.488 | **0.371** | 1.245 |\n| A&E (SD 2.1) + *CompLift* | **0.346** | **1.337** | **0.365** | **1.516** | 0.370 | **1.246** |\n\n> The idea of \"counting the activated pixels\" in Section 5.1 seems quite confusing... How did you set the threshold $\\tau$?\n\nThank you for raising this thoughtful concern. We agree that performance could be further improved by making $\\tau$ an object-specific hyperparameter. For simplicity, we currently set $\\tau$ as a uniform threshold.\n\nWe chose $\\tau = 250$ as the median value among the number of activated pixels across all images. We also experimented with the 25th and 75th percentiles, and found the median performed best in practice. A lower $\\tau$ leads to less accurate rejection due to ELBO variance, while a higher $\\tau$ increases the rejection rate.\n\nWe will include more details on the derivation of $\\tau$ in the Appendix. Intuitively, $\\tau = 250$ corresponds to ~1.5% of the total number of the latent pixels in SDXL (128x128 latent space) and ~6.1% in SD 1.4/2.1 (64x64). We find this small threshold sufficient, since our focus is on identifying \"missing object\" issues. If an object is missing, it tends to result in almost no activated pixels. Additional discussion will be added to the Appendix.\n\n> Does the same trend hold for the recent work \"Reduce, Reuse, Recycle\" [3] based on MCMC?\n\nThank you for this suggestion. We also observed a clear overall advantage of our method over MCMC-based methods like \"Reduce, Reuse, Recycle\" [3]. While certain MCMC variants such as U-HMC and MALA perform comparably in specific scenarios (e.g., the first test case in Product and second in Mixture), they often generate samples outside the target distribution in other settings, unlike our method.\n\nWe will update Appendix D to include visualizations comparing results from the MCMC-based methods.\n\n> Typo: page 8 line 408: ImageResizer -> ImageReward\n\nThank you for catching this typo. We will correct it in the revised version.\n\n[1] https://openreview.net/forum?id=E78OaH2s3f  \n\n[2] https://arxiv.org/abs/2301.13826  \n\n[3] https://arxiv.org/abs/2302.11552", "claims_and_evidence": "The overall writing of the paper is well-structured, with a clear problem definition and a simple but effective solution. The idea of adopting the concept of lift scores for improving diffusion model's compositional generation is interesting.\n\nHowever, the paper lacks reference and discussions on an important related work, as detailed below: \n \nCompLift seems to resemble a closely related work CAS [1], in which the authors define a novel \"condition alignment score\" CAS as $\\log p (x_0 | c) - \\log p (x_0)$ (Fig. 3 of the paper). The main argument of CAS is that this term can effectively measure the alignment between the generated output $x_0$ and the given condition $c$, and therefore it can be used as a alignment metric without the need of external modules. This claim is similar to the main contribution of this work.\n\nIn this regard, the proposed formulation of lift scores in Eq. (2)-(4) of this work seems to be quite similar to CAS, and the authors will need to provide a discussion on the difference between the two approaches in order to claim the novelty of CompLift.\n\n[1] CAS: A Probability-Based Approach for Universal Condition Alignment Score, Hong et al., ICLR 2024", "methods_and_evaluation_criteria": "1. For the request for the justification on the novelty of CompLift, please refer to the above section.\n\n2. Evaluations on the effect of CompLift on text-to-image generation is not very convincing as it does not include comparisons with the baselines. While the authors evaluate using the benchmark from Attend-and-Excite [1], I couldn't find the comparisons with the Attend-and-Excite itself. Since Attend-and-Excite (or its follow-ups) also does not require external modules for measuring the alignment with the given condition, I believe it should be a valid baseline for comparison. Otherwise, as stated in the introduction, it would be nice if the authors show that CompLift can indeed to applied together with such methods, yielding additional improvements.\n\n[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models, Chefer et al., SIGGRAPH 2023", "theoretical_claims": "This paper doesn't provide theoretical claim or proof, and instead focuses on the empirical evidence on numerous types of data.", "experimental_designs_or_analyses": "1. Comparisons on the running time in Fig. 5 clearly shows the advantage of CompLift over the MCMC-based approaches in terms of efficiency.\n\n2. The idea of \"counting the activated pixels\" in Section 5.1 seems quite confusing. I'm curious whether this design choice can consider a variety of objects. For instance, some objects are likely to take up a large area of the image, while other objects might be generated in smaller sizes. If the same threshold for number of pixels is applied for checking the existence, can it handle both cases? How did you set the threshold $\\tau$?", "relation_to_broader_scientific_literature": "As also mentioned in the paper, the idea of using the diffusion model itself as a source of a conditional reward function could be useful for inference-time scaling methods for diffusion models aiming for better condition alignment.", "essential_references_not_discussed": "A critical related work CAS [1] is missing, as mentioned in the \"Claims\" section.\n\n[1] CAS: A Probability-Based Approach for Universal Condition Alignment Score, Hong et al., ICLR 2024", "other_strengths_and_weaknesses": "I agree with the fundamental goal of the paper that \"diffusion models should be able to assess its own alignment to the given condition\". And the method is quite simple and intuitive. However, for now it is hard to give a positive score as the paper fails to address a critical previous work that has proposed a similar claim and solution.", "other_comments_or_suggestions": "typo: page 8 line 408: ImageResizer -> ImageReward", "questions_for_authors": "Crucial questions are included in the \"Claims\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ViWNnAbky0", "reviewer": "Reviewer_kdTw", "summary": "- The paper introduces a novel criterion CompLift for rejecting samples of conditional diffusion models based on lift scores.\n- For compositional generation, i.e., cases in which the condition for sampling, e.g, a text prompt can be described as a composition of conditions (like desired individual objects in the image), CompLift intuitively evaluates whether final samples are more likely given each individual condition than without it and therefore whether the conditions have been properly considered throughout the generation process.\n- Formally, this criterion can be described in terms of lift scores, an existing concept in data mining, for which the authors introduce an approximation using the same conditional diffusion model as for sampling.\n- In an exploration of the design space, the authors evaluate the effect of noise and timestep sampling for the approximation and propose a more efficient algorithm that caches intermediate results from the generation process for later evaluation of the rejection criterion.\n- An evaluation on synthetic data, a toy image dataset, as well as text-to-image generation shows improved alignment with the conditions for compositional generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We address your concerns below.\n\n> Overstatement of minimal\n\nWe will modify the abstract to make the contribution accurate as \"significantly improved the condition alignment for compositional generation\".\n\n> Limited comparisons: no T=50 for vanilla CompLift; missing EBM baselines on CLEVR/text-to-image\n\nWe have conducted additional experiments to address both concerns:\n\n1) We compared against EBM+ULA [1] on text-to-image generation (ULA is the default text-to-image sampler in their repo).\n\n2) We ran vanilla CompLift with T=50 to enable fair comparison.\n\nThe consolidated results are in the table below. CompLift outperforms EBM+ULA across all model variants, and the cached version performs similarly to vanilla CompLift with the same T=50, with CLIP scores very close while ImageReward scores are slightly lower for the cached version.\n\nWe wish to clarify that our main focus is demonstrating vertical improvement - how CompLift boosts the base method's performance. The horizontal comparison to other baselines serves as supportive evidence that this boost helps achieve state-of-the-art performance.\n\n| Method | Animals |  | Object&Animal |  | Objects |  |\n|--------|---------|---------|---------|---------|---------|---------|\n|        | CLIP ↑ | IR ↑ | CLIP ↑ | IR ↑ | CLIP ↑ | IR ↑ |\n| Stable Diffusion 1.4 | 0.310 | -0.191 | 0.343 | 0.432 | 0.333 | -0.684 |\n| SD 1.4 + EBM (ULA) | 0.311 | 0.026 | 0.342 | 0.387 | 0.344 | -0.380 |\n| SD 1.4 + *Cached CompLift* | 0.319 | 0.128 | 0.356 | 0.990 | 0.344 | -0.131 |\n| SD 1.4 + *CompLift* (T=50) | 0.320 | 0.241 | 0.355 | 0.987 | 0.344 | -0.154 |\n| SD 1.4 + *CompLift* (T=200) | **0.322** | **0.293** | **0.358** | **1.093** | **0.347** | **-0.050** |\n||||||||\n| Stable Diffusion 2.1 | 0.330 | 0.532 | 0.354 | 0.924 | 0.342 | -0.112 |\n| SD 2.1 + EBM (ULA) | 0.330 | 0.829 | 0.357 | 0.981 | 0.348 | 0.218 |\n| SD 2.1 + *Cached CompLift* | 0.339 | 0.880 | 0.361 | 1.252 | 0.354 | 0.353 |\n| SD 2.1 + *CompLift* (T=50) | **0.340** | **0.992** | 0.361 | 1.263 | 0.354 | 0.454 |\n| SD 2.1 + *CompLift* (T=200) | **0.340** | 0.975 | **0.362** | **1.283** | **0.355** | **0.489** |\n||||||||\n| Stable Diffusion XL | 0.338 | 1.025 | 0.363 | 1.621 | 0.359 | 0.662 |\n| SD XL + EBM (ULA) | 0.335 | 0.913 | 0.362 | 1.676 | 0.361 | 0.872 |\n| SD XL + *Cached CompLift* | 0.341 | **1.244** | **0.364** | 1.687 | 0.365 | **0.896** |\n| SD XL + *CompLift* (T=50) | **0.342** | 1.222 | **0.364** | 1.700 | 0.365 | 0.842 |\n| SD XL + *CompLift* (T=200) | **0.342** | 1.216 | **0.364** | **1.706** | **0.367** | 0.890 |\n\n\n> Structure issues\n\nThank you for the constructive feedback. We will try our best to make the concept more clear. In particular, we will make the following modifications:\n1. Self-inclusive caption - summarize the experiment setup in the caption of Figure 1, including the component distribution, the algebra, the data generation, and the training.\n2. Early introduction of 2D dataset - briefly mention the 2D synthetic dataset in Section 3, including the description about the data generation, the component distribution, the algebra, and the accuracy metric. We will add a sentence to refer reader to Section 6 and Appendix D for more details.\n3. Intuitive explanation - add several more explanation of the Compose function, using the examples in the 2D dataset.\n\n> 2D dataset missing generation details\n\nWe will add more text to Section 6.1 about the dataset generation. In short, the distributions follow the generation way in [1] - they are either Gaussian mixtures or uniform distribution. We sample 8000 data points randomly for each component distribution, and train 1 diffusion model for each distribution. We will include more parameters of the distributions in Appendix D.\n\n> Unclear noise sharing performance loss in negation\n\nOur hypothesis is that sharing the same noise introduces some bias in the estimation, and it makes CompLift over-reject samples as a conservative way. With more trials, the bias of the estimation is amplified, thus makes more samples over rejected. After taking a deeper look in Figure 2, we also observe similar  sharing-noise regression for Product and Mixture, though the regression is very slight for those 2 algebras. We will modify the explanation in the paper to make this hypothesis more clear.\n\n> Cached CompLift description unclear\n\nWe provide more details about the algorithm in Appendix C. Algorithm 5 and 6 are in pseudo-code styles and might be easier for the reviewer to parse. We will add a sentence in Section 4.3 to refer readers to Appendix C for more context. Please let us know if there remains such an issue, we will keep making the paper easier to read.\n\n> EBM-related abbreviations\n\nThanks. We'll add explanations in Section 6.1 for all abbreviations (EBM, ULA, U-HMC, MALA, HMC).\n\n> Typos\n\nThanks. We'll remove the self-reference and change $z$ to $z_t$ in paragraph L317.\n\n[1] https://arxiv.org/abs/2302.11552", "claims_and_evidence": "Most claims in the submission are supported by clear and convincing evidence except for:\n\n- The paper claims \"significantly improve[d] compositional generation\" (lines 24 ff., left column) while the quantitative results on the CLEVR position dataset mainly show accuracy improvements with 4 and 5 constraints, for which the FID however is worse than the Composable Diffusion baseline as also mentioned by the authors (lines 424 ff., left column)\n  - If the rejection of samples reduces sample diversity as hypothesized by the authors, the kind of improvements for compositional generation (condition / prompt alignment) should be specified to avoid misunderstandings.\n- The paper compares quantitatively on the 2D synthetic dataset against additional baselines (EBM [1]), but limits qualitative comparisons to the Composable Diffusion baseline only.\n- The main paper compares only on the 2D synthetic dataset against baselines (EBM [1]), but misses to do so on the CLEVR and text-to-image setups. The appendix provides comparisons on CLEVR.\n\n[1] Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC. ICML 2023", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense except for:\n- For the text-to-image compositional task, the number of trials is different for the vanilla version and the cached version. While I understand that the number of trials for the cached version is equal to the number of sampling steps, a fair comparison of both versions in terms of number of trials would be interesting to see for this task in order to evaluate the effect of caching.\n- While providing results on the 2D synthetic dataset makes sense, there is a severe lack of clarity regarding this benchmark:\n  - Figure 1 showing results on this dataset is already on page 2 but first referenced on page 7. The caption of it also does not describe the experimental setup. I found this figure to be unclear if the dataset is not introduced yet.\n  - As a result of that, the Compose function (lines 119 ff., right column) together with table 1 lack intuition. Having the synthetic 2D dataset described earlier for an example of different compositions of conditions (or possibly a text-to-image example in the introduction using different compose functions) would be helpful.\n  - The introduction of the dataset in section 6 and metrics after the design space exploration / ablation (section 4) with quantitative resutls on this benchmark and figures 2 and 3 also raise questions about the dataset and how the accuracy is measured while reading the paper.\n  - The description of the 2D synthetic dataset at the beginning of section 6.1 is lacking information about how the dataset is generated.", "theoretical_claims": "There are no theoretical claims that require proofs.", "experimental_designs_or_analyses": "All experimental designs and analyses seem to be valid.", "relation_to_broader_scientific_literature": "Given complex conditions such as text prompts, diffusion models are known to hallucinate samples not following the correct conditional distribution. If conditions can be decomposed into smaller conditions, prior work like Composable Diffusion [1] has shown that this compositionality can be effectively leveraged to generate samples with better alignment to complex prompts. This paper proposes a method orthogonal to prior work by introducing a rejection / resampling criterion to ensure that the final sample is positively correlated with the conditions.", "essential_references_not_discussed": "I am not aware of any essential missing references.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is mostly well-written and easy to follow. Abstract, introduction, and related work provide a good motivation and introduction into the topic of compositional generation.\n- The method section includes formal derivations of the lift score approximation as well as the intuition behind the equations, which I found very helpful.\n- The qualitative results are convincing:\n  - Once the 2D synthetic dataset and task is understood, the qualitative results illustrate the effect of CompLift and the different Compose functions well.\n  - The pixel-wise scores for text-to-image generation clearly decompose the image according to the individual conditions.\n- The quantitative results show consistent improvements over the Composable Diffusion baseline.\n\nWeaknesses:\n- As already indicated in review section \"Methods And Evaluation Criteria\", the structure of the paper w.r.t. the 2D synthetic dataset, figure 1, the different compose functions with table 1, and section 4 consisting of ablations before the dataset description is suboptimal and results in a lack of clarity.\n  - And the dataset description itself also lacks information about how it was generated.\n- More lack of clarity:\n  - In lines 154 ff., right column, the explanation for the small performance loss using noise sharing strategies in negation tasks is unclear to me.\n  - I find the description of the cached CompLift version in Section 4.3. and Algorithm 2 difficult to understand, even though the idea itself is quite simple and intuitive.\n- The paper never introduces the abbreviations for the baselines from EBM and also not EBM itself.", "other_comments_or_suggestions": "- In line 194 f., right column, you reference section 4.3. in section 4.3. itself.\n- In line 317, left column, the $z$ should be a $z_t$, if I am not mistaken.", "questions_for_authors": "I do not have any particular questions to the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FxjjgPuEdn", "reviewer": "Reviewer_xPQc", "summary": "This paper aims to improve compositional generation at inference time via rejection sampling using Lift scores on each condition to be composed.\n\n**Update after rebuttal**\nI appreciate the additional data (which I found convincing) and clarifications provided by the authors during the rebuttal. I was previously unaware of the prior work CAS mentioned by reviewer kyUu and agree that it may impact the novelty, but I also found the authors' clarifications about their focus specifically on compositions and systematic evaluation fairly convincing. Overall, I will keep my score at 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We address your concerns as follows:\n\n> Chamfer Distance\n\nWe chose Chamfer Distance because it (1) applies to uniform distributions and (2) is sensitive to out-of-distribution samples. KL is inapplicable for out-of-distribution samples with undefined density ratio in uniform distribution settings. Wasserstein Distance is more robust to outliers, which doesn't meet our requirement to sensitively capture unaligned samples.\n\n>  Segment Anything for t2i\n\nVanilla Segment Anything lacks text prompt support. Grounded Segment Anything [1] would be a good candidate. We used CLIP/ImageReward following standards in previous works [2,3], but we will add a discussion in Conclusion.\n\n> TIFA score [2, 4]\n\nNew TIFA experiments show improvements across categories:\n\n| Method | Animals | Object&Animal | Objects |\n| --- | :---: | :---: | :---: |\n|  | TIFA ↑ | TIFA ↑ | TIFA ↑ |\n| Stable Diffusion 1.4 | 0.692 | 0.822 | 0.629 |\n| SD 1.4 + *Cached CompLift* | 0.750 | 0.886 | **0.685** |\n| SD 1.4 + *CompLift* | **0.794** | **0.902** | 0.682 |\n| | | | |\n| Stable Diffusion 2.1 | 0.833 | 0.873 | 0.668 |\n| SD 2.1 + *Cached CompLift* | 0.905 | 0.911 | **0.731** |\n| SD 2.1 + *CompLift* | **0.927** | **0.912** | 0.726 |\n| | | | |\n| Stable Diffusion XL | 0.913 | 0.964 | 0.755 |\n| SD XL + *Cached CompLift* | **0.949** | 0.972 | **0.790** |\n| SD XL + *CompLift* | 0.946 | **0.974** | 0.782 |\n\n> entire prompt vs individual condition\n\nWe used entire prompt. We add new experiment with minCLIP (minimum CLIP score across subjects):\n\n| Method | Animals | Object&Animal | Objects |\n| --- | :---: | :---: | :---: |\n|  | minCLIP ↑ | minCLIP ↑ | minCLIP ↑ |\n| Stable Diffusion 1.4 | 0.218 | 0.248 | 0.237 |\n| SD 1.4 + *Cached CompLift* | 0.225 | 0.260 | 0.249 |\n| SD 1.4 + *CompLift* | **0.228** | **0.263** | **0.252** |\n|  |  |  |  |\n| Stable Diffusion 2.1 | 0.237 | 0.258 | 0.247 |\n| SD 2.1 + *Cached CompLift* | 0.248 | **0.265** | 0.260 |\n| SD 2.1 + *CompLift* | **0.249** | **0.265** | **0.261** |\n|  | |  |  |\n| Stable Diffusion XL | 0.243 | 0.269 | 0.264 |\n| SD XL + *Cached CompLift* | 0.248 | **0.271** | 0.269 |\n| SD XL + *CompLift* | **0.250** | **0.271** | **0.271** |\n\nSimilar performance gains were observed, indicating CompLift primarily improves the weaker condition (typically missing object).\n\n> CLIP vs ImageReward\n\nWe manually labeled 100 samples from Fig. 10 for presence of both black car and white clock. With 62 positive and 38 negative samples, we calculated metric performance:\n\n| Metrics | CLIP | ImageReward | TIFA |\n| --- | :---: | :---: | :---: |\n| ROC AUC | 0.949 | 0.955 | 0.857 |\n| PR AUC | 0.972 | 0.968 | 0.901 |\n\nCLIP and ImageReward perform similarly, both better than TIFA. CLIP slightly preferred due to imbalanced data.\n\n> decisions feel a bit ad-hoc\n\nWe choose $\\tau=250$ as the median activated pixel count among all images. Tests at 25th and 75th percentiles showed the median works best - lower $\\tau$ reduces accuracy due to estimation variance, while higher $\\tau$ increase rejection rates.\n\nRegarding $\\epsilon_\\theta(x_t, c_{compose})$: this design is from empirical observations. Intuitively, if object $c$ exists in image $x$, then for most noisy images $x_t$, $\\epsilon_\\theta(x_t, c)$ should be closer to $\\epsilon_\\theta(x_t, c_{compose})$ than the unconditional $\\epsilon_\\theta(x_t, \\varnothing)$ in the corresponding pixels. We'll add this explanation to the paper.\n\n> connections with CFG\n\nOur paper uses the constrained distribution:\n\\begin{equation}\n x_0 \\sim p_{\\text{generator}}(x_0), \\quad \\text{s.t. } \\log p(x_0 \\mid c_i) - \\log p(x_0) > 0, \\quad \\forall\\, c_i.\n\\end{equation}\n\nNow we show that [5,6] tries to satisfy the constraint using soft regularization. Using Lagrangian relaxation and multipliers $\\lambda_i \\geq 0$, the objective can be transformed into:\n\\begin{equation}\n\\mathcal{L}(x_0, \\lambda) = \\log p_{\\text{generator}}(x_0) + \\sum_{c_i} \\lambda_i \\Bigl( \\log p(x_0 \\mid c_i) - \\log p(x_0) \\Bigr), \\quad \\lambda_i \\geq 0.\n\\end{equation}\n\nSince \n$\\nabla_{x_t}\\log p_\\theta(x_0) \\propto \\epsilon_\\theta(x_t, t)$, and [5, 6] assume an unconditional generator,\nthe derivative matches Equation 11 in [6]:\n\\begin{equation}\n\\nabla_{x_t}\\mathcal{L}(x_0, \\lambda) \\propto \\epsilon_\\theta(x_t, t) + \\sum_{c_i} \\lambda_i \\Bigl( \\epsilon_\\theta(x_t, t \\mid c_i) - \\epsilon_\\theta(x_t, t) \\Bigr), \\quad \\lambda_i \\geq 0.\n\\end{equation}\n\nCFG [5,6] uses fixed $\\lambda_i=w$, not guaranteeing constraint satisfaction.\n\n> Are Figure 4, 7, samples cherrypicked?\n\nYes: Figure 4 shows the ones with most-improved CLIP scores. Figure 7 uses random prompts with clear pixel separation and aesthetic quality. We will add more samples to Appendix and captions to make the selection clear. \n\n[1] https://arxiv.org/abs/2401.14159\n\n[2] https://arxiv.org/abs/2305.13308\n\n[3] https://arxiv.org/abs/2301.13826\n\n[4] https://arxiv.org/abs/2303.11897\n\n[5] https://arxiv.org/abs/2207.12598\n\n[6] https://arxiv.org/abs/2206.01714", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Quantitative metrics are important for this study. I feel that this was done thoroughly and mostly reasonable choices were made, although I have a few questions.\n\n2D synthetic: (Q) Why did you choose Chamfer Distance (as opposed to other metrics e.g. KL, Wasserstein etc.)?\n\nCLEVR: SAM2 and Liu pretrained classifier both make sense, it is nice that they were both tried and compared.\n\nSDXL: CLIP and ImageReward seem reasonable here.  \n(Q) Could Segment Anything potentially be used in this context? (why not?). \n(Q) Would it be feasible to evaluate with the TIFA score as in Karthik et al (https://arxiv.org/pdf/2305.13308)?\n(Q) (L407) For assessing alignment via CLIP do you use the entire prompt (including all conditions) or assess the CLIP alignment on each condition individually? I believe it’s the former but do you think they latter might work better? (do you have any evidence on this?)\n(Q) Do you have any way to assess whether CLIP vs ImageReward is a more appropriate metric, and how well each of them actually checks whether *all* the conditions are present in the composition (e.g. for AND).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Please see Methods And Evaluation Criteria", "relation_to_broader_scientific_literature": "This approach builds on various existing approaches in diffusion, composition, rejection sampling, Lift scores, etc. I don't feel that it represents a big conceptual leap beyond these existing methods, however I think it cites prior work appropriately and focuses mainly on getting a conceptually simple idea to work well empirically with suitable validation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: I appreciate the focus on the missing objects problem. I find the samples and metrics fairly convincing (though please see Questions).\n\nWeaknesses: A lot of the decisions feel a bit ad-hoc (e.g. how many activated pixels “count” e.g. tau = 250 on L245 and the choice to replace epsilon by eps_theta(x, c_compose) in Fig 6) although I don’t consider this a deal-breaker for a methods paper.", "other_comments_or_suggestions": "I would appreciate a clearer discussion of the final probability you are actually optimizing for (or energy you are minimizing) after your rejection-sampling procedure, if there is anything you can say theoretically about it. I did not find the variance-reduction interpretation of replacing epsilon by eps_theta(x, c_compose) very clear. I also wonder if there are any connections with CFG that you know of?", "questions_for_authors": "Please see the questions in Methods And Evaluation Criteria.\n\nA few others:\nAre Figure 4, 7, samples cherrypicked? If so I feel this should be acknowledged in the caption, and possibly some not-so-great examples included in appendix.\n\nAnswers to these questions would help me find the samples and metrics more convincing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers find the paper well written, and the evaluation demonstrates that the proposed method consistently outperforms the baseline, particularly on the synthetic dataset. However, as noted by Reviewer kyUu, the literature review is incomplete, with a highly relevant work omitted which impacts the novelty and contribution of the paper. Additionally, the reviewers find the evaluation on real-world data less convincing. These concerns were addressed in the rebuttal, leading to improved overall recommendations. Area Chair agrees with the reviewers and considers the paper above the acceptance threshold, provided the authors appropriately incorporate the content and clarifications presented in the rebuttal into the final revision.
Decision: Accept (poster)