Compositional Scene Understanding through Inverse Generative Modeling
Authors: Yanbo Wang
Justin Dauwels
Yilun Du
Venue: ICML 2025 poster (2025)
Keywords: Scene Understanding
Compositional Generative Modeling
Generalization
Primary area: deep_learning
TL;DR: We present a way of using generative models for scene understanding tasks with strong out-of-distribution generalization

Abstract: Generative models have demonstrated remarkable abilities in generating high-fidelity visual content. In this work, we explore how generative models can further be used not only to synthesize visual content but also to understand the properties of a scene given a natural image. We formulate scene understanding as an inverse generative modeling problem, where we seek to find conditional parameters of a visual generative model to best fit a given natural image. To enable this procedure to infer scene structure from images substantially different than those seen during training, we further propose to build this visual generative model compositionally from smaller models over pieces of a scene. We illustrate how this procedure enables us to infer the set of objects in a scene, enabling robust generalization to new test scenes with an increased number of objects of new shapes. We further illustrate how this enables us to infer global scene factors, likewise enabling robust generalization to new scenes. Finally, we illustrate how this approach can be directly applied to existing pretrained text-to-image generative models for zero-shot multi-object perception. Code and visualizations are at https://energy-based-model.github.io/compositional-inference.

Compositional Scene Understanding through Inverse Generative Modeling
Yanbo Wang
Justin Dauwels'
Yilun Du
Abstract
(1) Infer object locutions
Generative models have demonstrated remarkable abilities in generating nigh-fidelitg visual content. In this work, we explore how generative mod- els can further be used not only to synthesize vi- sual content but also to understand the properties of scene given natural image. We formulate scene understanding as an inverse generative mod- eling problem, where we seek to find conditional parameters of a visual generative model to best fit given natural image. To enable this proce- dure infer scene structure from images substan- tially different than those seen during training, we further propose to build this visual generative model compositionally from smaller models over pieces of a scene. We illustrate how this proce- dure enables us to infer the set objects a scene, enabling robust generalization to new test scenes with an increased number of objects of shapes. We further illustrate how this enables us to infer global scene factors, likewise enabling robust generalization to new scenes. Finally, we illustrate how this approach can be directly ap- plied to existing pretrained text-to-image gener- ative models for zero-shot multi-object percep- tion. Code and visualizations are at https://energy- based-model. github, wwt.eell.ggthuub.ooooooooo htto///omppiitionaa-ffeeeeeeeeee
shecicted lcsation
preeice
In-distribution
Result
Out-of-distrrbutioo
Result
query
query
Infer facial attributes
Black-hair
Black-huir
Eyeglasses
Eyeglasses
Smiling v
Smiling
In-distribution
Result
Out-of-distrbuttion
Result
query
query
(c) Infer object eategories
A dog
dog
cat
A cat
A rabbit
A rabbit v
Zero-shot query
Result
Zero-shot query
Result
Figure 1: Compositional Scene Understanding, Our approach demonstrates strong generalization across various scene under- standing tasks. For object location inference (first row), the model is trained on CLEVR images containing 3-5 objects, while the test set is CLEVRTex, which contains 6-8 objects. For multi-facial attribute inference (second row), the model trained only fe- male faces from CelebA, and tested exlusively on male faces. For object category inference (third row), we use pretrained Sta- ble Diffusion without any additional fine-tune fine-tuning, and the test set consists of multi-object natural images.
Introduction
world (Shepard & Metzler, 1971). Such a generative learn- ing mechanism the key for accurately parse scenes that have never encountered before, i.e., zero-shot scene understanding (Chomsky, 1965; Fodor & Pylyshyn, 1988; Bengio, 2019). We are interested in equipping machines with such generalizable scene-understan seeee-nnersstnndin abilities by leveraging recent advances generative models.
"What cannot create, do not understand."
Richard Feynman
To understand surrounding physical scenes, human intel- ligence is able to learn abstract visual concepts from the physical world and compositionally reuse them (Biederman, 1987: Greff et al., 2020; Fodor & Lepore, 2002). Given an image of an object, we can then easily imagine how the object would look if it were rotated or moved in 3D
Conventionally, scene understanding tasks have been dom- inated by discriminative models that learn direct map- ping from input images to visual attributes (Vapnik et al., 1998; Krizhevsky et al., 2012; Redmon, 2016), which, how- ever, is demonstrated to struggle with generalizing to even slightly shifted test distributions (Geirhos et al., 2018; Recht et al., 2019; Taori et al., 2020; Hendrycks & Gimpel, 2016; Geirhos al., 2020). In contrast, generative models have
'TU Delft Harvard University. Correspondence Yanbo Wang wang 27 @tudelft.nl>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Compositional Seene Understanding through Inverse Generative Modeling
long been advocated for solving inference problems with the promise of better generalization brought by data generation modeling (Ng & Jordan, 2001; Hinton, 2007). Yet, only very recently have generative models begun to show promis- ing results for visual inference tasks (Li et al., 2023a; 2024; Clark & Jaini, 2024), thanks to the highly expressive model- ing abilities of diffusion models SSoh--Dickstein et al., 2015; Ho et al., 2020). Despite this progress, these newly proposed generative inference approaches focus only on single-label classification tasks, and how to perform broader range of scene understanding tasks (e.g., object discovery or multi- object classification) on scenes significantly more complex than those seen during training remains elusive.
2 Related Work
Generative Models for Visual Understanding. Recent work has explored applying generative models to tasks beyond visual generation, such as classification (Li et al., 2023a; 2024; Jaini et al., 2023; Clark & Jaini, 2024; Maha- jan et al., 2024; Chen et al., 2024), personalization (Gal et al., 2022; 2023; Avrahami et al., 2023), and segmentation (Amit et al., 2021; Brempong et al., 2022; Zhao et al., 2023; Wang et al., 2024). Most relevant to our work, generative classi- fiers (Li al., 2024) leverage generative models to tackle single-label classification tasks. In contrast, our framework does not limit itself to solving single-label classification problems; instead, it demonstrates how generative models with flexible conditioning can address a broader range of visual understanding tasks, such as object discovery and zero-shot multi-object perception. More importantly, our approach composes a generative model from smaller sub- models each capturing a specific visual concept, enabling generalizing to unseen scenes that differ substantially from training set.
In this work, we propose an inverse generative modeling framework that is broadly applicable across various scene understanding tasks, including those involving scenes more complex than that encountered during training. Our frame- work builds a visual generative model compositionally (Du & Kaelbling, 2024) from smaller generative pieces repre- senting individual parts of a scene. During inference, to understand scene, we aim to find the conditional param- eters for a composed set of generative models that best fit given natural image, enabling to fit more complex scenes by fitting larger set of conditional parameters for more generative models.
Compositional Generative Models. There has been sig- nificant recent progress in incorporating compositionality into generative models (Du & Kaelbling, 2024) to enable generalization beyond training distribution (Du & Mordatch, 2019; Cho et al.. 2023; Shi al., 2023; Sohn et al., 2023;
In Fig. 1, we show how our approach can be used to compo- sitionally interpret scenes across different visual understand- ing tasks. In the top row of Fig. 1, we illustrate how our approach can discover objects in scene by predicting ob- ject positions and generalize effectivly to nut-of-distribution images. For this task, the model is trained on CLEVR dataset with each image containing 3-5 objects, while tested on a different dataset CLEVVRTe with 6-8 objects. The substan- tial difference in object number, shape, color, texture and background between the training set and the test set demon- strates the strong generalization ability of our approach. In the middle row of Fig. 1, we demonstrate how our approach can simultaneously classify multiple facial attributes on CelebA dataset and likewise generalize faithfully, where the training set contains only female faces while the test set con- tains only male faces. Finally, in the bottom row of Fig. 1, we show how our approach can adopt pretrained diffusion models to perform zero-shot multi-object perception task on web images without any additional training.
Du et al., 2020; 2021; 2023; Nie al., 2021; Feng et al., 2022; Li et al., 2022: Liu et al., 2021; 2022; Huang et al.,
2023; Cong et al., 2023; Wang et al., 2023; Su et al., 2024;
Zhou et al., 2024; Netanyahu et al., 2024). While most of these works focus on generating novel scenes, we focus on less explored direction inverse compositional generative modeling for scene understanding. The most similar work in this direction is UCCD (Liu et al., 2023), which requires group of images input to identify common concepts across image clusters. In contrast, our approach takes a single image as input, aiming to discover visual concepts that best interpret it. Furthermore, unlike UCCD relying on text-to-image generative models, our approach leverages generative models with flexible conditioning and can be applied a wider range of visual understanding tasks.
Image Captioning. Our work is also related to image cap- tioning. By leveraging pre-trained eext-to-maaee generative models (e.g., Stable Diffusion), our model can be applied to image captioning tasks like BLIP-2 (Li et al., 2023b). How- ever, our approach is applicable to a broader range of scene understanding tasks beyond image captioning. For example, by conditioning on object coordinates, our approach can per- form object discovery tasks and even enable generalization to more complex scenes (many more objects) than seen at training. This flexibility and generalizability distinguishes our approach from traditional image captioning models.
Our contributions are as follows: (1) We propose a generic inverse generative modeling framework to tackle several scene understanding tasks such as object discovery and zero-shot perception. (2) We build the inverse generative model compositionally, enabling strong generalization be- yond training set. (3) Our approach significantly outper- forms generative classifier baselines for several scene under- standing tasks on both synthetic and realistic image datasets.
2

---

Compositional Seene Understanding through Inverse Generative Modeling
3 Compositional Scene Understanding through Inverse Generative Modeling
To construct the composed noise prediction model in Eqn (4), prior work has focused on learning each denois- ing function eo ea(æ", //ce) in isolation, combining denoising functions at test-time dependent on the composition needed. However, such a test-time composition of denoising func- tions can lead to the accumulation of error between score functions. To more accurately model the composed score function egumb (z' .t), we directly train the composed score function with the denoising diffusion objective:
In this section, we introduce our inverse generative modeling approach for scene understanding tasks. Given an image I, we aim to infer a set of K visual components (c'..... .... c*) that describe the image, where image & will often contain a larger or more complex combinations of concepts than those seen at training time. We first illustrate how we can model more complex test scenes by modeling the data generation process as composition of a set of generative models. Next, we formulate how we can invert the generation process to infer the set of concepts that describe a given image.
Co Ex,,, ||e
Eg
-g (x',t)l'
K
(5)
= =E,,,,, I|e
Σ
Ea (2' f|ct))1
where each of individual term es ca t|c²²) is parameterized by a neural network. This enables the composed denoising functions behave more accurately together, and at test time, we can still compose additional terms of ea ca t|c²) to construct more complex scenes. We provide overview of this training approach in Algorithm 1.
3.1 Compositional Generative Modeling
In a visual domain, given set of conditioned concepts fcl, c², , }, we aim to construct a generative model that can accurately represent the probability distribution p(a]cc .o,,....c"),
(1)
over the space of images æ. The set of scenes with concepts fcl, c², ..,,...,,,, .k) can be much more complex at test time than those seen at training time, making it difficult to directly fit generative model on the data.
3.2 Compositional Scene Understanding
Given a generative model p(x]cc, ,eppccc,c"....... ,."....,ct c,),,,))))) we can then formulate scene understanding given an image x as an in- verse problem of finding parameters of the model that ex- plain the image. Concretely, we seek to find set of visual concepts c² ,² that maximize the log-likelihood of the observed image:
One approach to model p(z)c'', ce,..,,,,,, ,x) is to factorize the probability distribution (Du & Kaelbling, 2024) and ap- proximate it a product of simpler conditional distributions p(x)c=)):
é¹ ei...., c argmaxloggparcc .e,......,,,
(6)
p(zlc' ......"" ) x II p(
et....,ck
The inferred set of concepts c then corresponds to de- scription of the scene, where individual concepts can flexibly
While this approach is a biased approximation of p(a|c)..... )²). prior work (Liu et al., 2022) has found that it enables effective compositional generalization to a larger number of visual concepts. We can model each p(#]c*)) as an energy-based model (EBM) e EE(aale"), The product distribution p(z|c',.... c²) takes the form of a summation of a set of energy functions:
describe individual objects of the scene as well as global features.
We can approximate the optimization of likelihood in Eqn (6) in diffusion models through the variational lower bound. The variational bound corresponds to a weighted form of the objective:
p( c
xk)) e Ek_ Eo(ale")
(3)
c),..., cK argmin Ee,r|/e - co(x", [|c',.... cK )|
el c'.....e
To parameterize this product of EBMs, similar to (Liu et al., 2022), we can represent each EBM Es(x)c*) using the denoising function diffusion model ((x")cc", which approximately represents (zrle)) To sample from the product distribution in Eqn (3) we construct the composed denoising function:
where similar to prior work, we approximate the likeli- hood by ignoring the weighting terms on each loss (Li et al., 2023a). Thus, for a given image, we can optimize for set of visual concepts by minimizing the above objective.
We can then use the approach discussed in Section 3.1 to directly parameterize denoising functions for more complex scenes with more concepts by optimizing Eqn (4), leading to the optimization objective of:
comb (x'
Σ cola" )
(4)
which approximately corresponds to V. IK )o(
K Σ Efl 1P,
We can then use the composed denoising function Eg coumb(rr', 1) in the standard diffusion sampling process to ap- proximately sample from the product distribution in Eqn (3).
é¹ c',...,'
Ck argmin E.,elee
(7)
el c'....ee
k=1
where we use a set of samples with different sampled
3

---

Compositional Scene Understanding through Inverse Generative Modeling
Compositional Inverse Generative Modeling
ch no black hair c no eyeglasses
Concept
Diffusion Model
cx smiling
(x(x,((ec))
argmin E, - Eell ct,
Diffusion Model
Eu (xt,,elle*)
c² (0.25, 0.28)
Ec
Concopt c*
c²: (0.31, 0.49)
cx: (0.75, 0.74)
Input image
Inferred concepts
Figure 2: Compositional Scene Understanding. Our model achieves scene understanding by identifying the optimal conditioning concepts (e.g.. facial attributes or object coordinates) that best interpret the input test image. Its compositional structure allows for simultaneous inference of multiple concepts and enables robust generalization to images that differ substantially from the training data.
Algorithm 1 Training Algorithm
Algorithm 2 Discrete Concept Inference Algorithm
1: Input: data distribution pp, denoising model
1: Require: an image æ, trained denoising model ep
2: while not converged do
2: Determine all possible M K concept configurations
3: (ro, c¹, c², ,.,,,...., c.) ~ PD
Ctuple = {(cc, c² ..... c/ E {et, ey..... ..., 1
4: >. Compute denoising direction
& Evaluate denoising error for each configuration
5: <~ (,(, 1), Unif([]..... T))
4: Initialize denoising error list E = zeros
6: x' va,a0 + vI dGE
5: for n L,.... ......,Naune .Vsample do
7: AO < Volle Ek=1 co(=", t, c)12
6: Oz ~ ~O, tn ~ (1..(( TT)
8: end while
7: x²n = varz + -
9: return Eg
8: for j = 1...,, MK do
E[j] += l|en Σ co by et (i)lle
timesteps and noise to estimate the objective.
10: end for
11: end for
Optimizing Visual Concepts. We can solve Eqn (7) with different optimization algorithms, depending on the concepts e² are discrete or continuous in specific visual understanding tasks. When each visual concept c E es..e.... ( is a discrete variable with finite set of possibilities, we can directly optimize Eqn (7) by enumerat- ing through each possible configuration of c and evaluating the average denoising error. We illustrate this optimization in Algorithm 2. To scale to a large number of discrete concepts and reduce inference time, we further propose a gradient-based search method in Algorithm 5 in See A.2. In contrast, when c* is continuous, such as when they de- scribe the locations of objects in the scene, optimization is substantially more complex, exhaustive search is not fea- sible and gradient-based optimization is easily susceptible to local minima. We describe more complex algorithms for inference when dealing with this setting in Section 3.3.
12: > Select the configuration with lowest denoising error
13: j = argmings MK) N El]]
14: return Cruptel5]
where Kmin and Km are the minimal and maximal limit of K. Similar to Eqn (6)-Eqn (7), we can approximate the log-likelihood in Eqn (8) with variational lower bound and parameterize the denoising function with composition of multiple functions for generalization purpose:
K
K = argmin KE(K Kmaz) c!.....t E, ||e Σ Ecolat.
In Figure 3, we visually illustrate how K can be deter- mined in object discovery tasks by solving the above ex- pression. We see that the ground truth number con- sistently yields the lowest average denoising error (highest likelihood). demonstrating the effectiveness of our concept number determination approach. We illustrate the object location visualization in Figure XII and the algorithm for determining the number of concepts in Algorithm the Appendix.
Inferring Number of Visual Concepts. In many scene understanding tasks, it is difficult to know beforehand the number of visual concepts K in the scene. For instance, in an object discovery task, the number of concepts (objects)
may differ from one image to another. To determine the number of concepts K for given test scene before inferring concept parameters, we can find number K that maximize the log- of the test image:
Overall, proposed inverse generative modeling (IGM) framework significantly broadens the applicability of gener- ative models to visual understanding tasks with several key elements: (1) flexible conditioning enables the inference of continuous concepts beyond class labels; (2) compositional modeling supports simultaneous multi-concept inference
K argmax KE Wiinn K Kmaal let...ek
max
logp(æ ( c¹
(8) c.....,,))]

---

Compositional Seene Understanding through Inverse Generative Modeling
ground truth number 4
ground truth number
Algorithm 3 Continuous Concept Inference Algorithm
0.14
ground truth number
1: Require: an image x, trained denoising model cØ, À
ground truth number?
2: & Initialize multiple (R) sets of concepts
3: Initialize concepts {ct, ch...., oy } ~ 10),
Averaqe Denoising
0.12
4: > Run Stochastic Gradient Descent
0.11
5: for n = .......,Nttr do On ~ ~,,,
Unif((1...... .T.
0.10
7: xnn Var, ax vI On
0.09
8: Act t Vet ll - -Ei= col2"n tn: ct
0.08
9: ci c -AAck
0.07
10: end for
11: & Evaluate denoising error for each sel
7
Concept Number
12: Initialize a denoising error list E = zeros(R)
13: for i= .N...,NNnn Nsample do Unif([1,.....
Figure 3: Concept Number Inference. Illustration object number inference on CLEVR. Given a test image, our model evaluates each number K E {3..., 8) respectively by using K objects fit the image and obtain corresponding denoising emrors. Out of the potential options E (3,,.., 83, our model determincs the one with the lowest denoising error as object number, which turns out to be consistent with the ground truth number.
14: 4 (0,
15: x = Von, + vT 04
16: for r = I...., R do
17: E[r]+= ||: - E co t4, )1
18: end for
19: end for
besides single-concept inference; (3) compositionality al- lows generalization to scenes substantially different from training data; and (4) the inference algorithm is applica- ble to both domain-trained diffusion models and generic pretrained diffusion models. An overview of our proposed inverse generative modeling approach is illustrated in Fig- ure 2. In Section 4.3, we demonstrate how our model can adopt pretrained text-to-image generative models like Stable Diffusion to solve zero-shot multi-object perception tasks without requiring any additional training.
20: & Select the set with lowest denoising error
21: f argmin, (1.....)
N E[r]
1
22: return ct. G..,.....
values, as the optimal ch is expected to yield a high likeli- hood, This process ultimately converges to single optimal configuration of ct with the highest likelihood. Empirically, we find that this initialization strategy improves the algo- rithm's s ability to escape local minima, thereby significantly enhancing scene understanding accuracy, as demonstrated in the ablation study in Sec 4.1 and Table IV.
3.3 Continuous Visual Concept Inference
Efficient Concept Optimization. To address the sample and computation complexity associated with evaluating the expectation term in Equ (7), we propose leveraging stochas- tic gradient descent (SGD) for optimization. This approach requires a single sample of en and t, at each optimization step to update the concepts ct. As a result, the sample com- plexity is reduced from N 1 per iteration, and eæ (a ) needs to be evaluated only once per iteration in stead of N times. This significantly accelerates the inference speed.
In Section 3.2, we aim to infer set of concepts that best describe a given image by optimizing Eqn (7). When con- cepts cx are continuous, however, the optimization using gradient descent faces several practical challenges. First, the potential non-convexity of the objective function, due to the neural network parametrization of co t/c²), can cause a to converge local minima, resulting in substantial de- viation from the optimal solution. Second, evaluating the expectation term in Eqn (7) at every gradient descent step incurs prohibitively high sample complexity with respect to en and fn. as well as significant computational complexity for evaluating co co(x", f0xx,t/(c ). We propose improved strategies on top of gradient descent to overcome these challenges as illustrated below and outlined in Algorithm 3.
4 Experiments
In this section, we evaluate the scene understanding ca- pabilities of our proposed approach across three different tasks. First, we consider a local factor perception task in Section 4.1, where the objective is to infer the center coordi- nates of objects. We next perform a global factor perception task to predict facial attributes from human faces in Sec- tion 4.2. Finally, we demonstrate how our approach can be adapted to pretrained models for zero-shot multi-object perception without any additional training in Section 4.3.
Effective Concept Initialization. To more effectively pre- vent et from converging to local minima, we propose ini- tializing ch with multiple random starting points, denoted as ot, ch ..., ch and maintaining corresponding updates throughout the optimization process. After every few opti- mization steps, we terminate paths with low likeliiel
5

---

Compositional Scene Understanding through Inverse Generative Modeling
In-distribution (3-5 objects)
Out-of-distribution (6-8 objects)
Models
Perception Rate
Estimation Error
Pereeption Rate t Estimation Error
ResNet-50 (He al., 2016)
5.3%
19.4e
2.9%
19.7e
SlotAttn (Locatello et al., 2020)
80.4%
8.7e
53.3%
1.3e 3
DINOSAUR (Seitzer al., 2022)
82.5%
8.4e
59.0%
1.2e
GC (Li et al., 2024)
82.2%
6.0e
58.7%
1.2e -2
IGM w/o multiple-initialization (Ours)
72.8%
6.9e
68.0%
7.8e
IGM with muultiple-initializzee (Ours)
94.7%
1.4e
85.3%
3.5e
Table I: Accuracy of Object Discovery. Quantitative evaluation of object perception results on CLEVR for both in in-distribution (3-5 objects) and out-of-distributis (6-8 objects) test settings. Perception rate and estimation error reported. Our approach outperforms all the baselines, and the margin is especially significant for the out-of-distribution setting, demonstrating strong generalization capability.
4.1 Local Factor Perception
Test image
ResNet-50
SlotAttn
GC
Ours
We demonstrate how our approach can infer local factors, such as object coordinates, from test image and effectively generalize to scenes containing larger number of objects and more complex objects than those seen during training.
Dataset. We evaluate our approach on the CLEVR dataset (Johnson et al., 2017), where each image is anno- tated with ground truth center coordinates of the objects. The training set consists of images containing 3-5 objects. To evaluate the generalization ability of our approach on out-of-distribution data, we consider two settings: (I) im- ages from the CLEVR dataset containing 6-8 objects; (2) images from the CLEVRTex dataset containing 6-8 objects.
ground-truth location
predicted location
Figure 4: In-distribution Object Discovery. We train our model with CLEVR images containing 3-5 objects. During inference, given in-distribution image (also containing 3-5 objects), our approach accurately identifies object coordinates. Compared with both determinative and generative baselines, our proposed ap- proach demonstrates better coordinates estimation performance.
Baselines. We compare our approach against both dis- criminative and generative baselines, including ResNet-50 (He et al., 2016), Slot Attention (SlotAttn) (Locatello et al., 2020), DINOSAUR (Seitzer et al., 2022), and Generative Classifier (GC) (Li al., 2024). Details on how these base- lines are trained for the object perception task can be found in Appendix A.S.
successfully generalize to images with 6-8 objects despite being trained only on images containing 3-5 objects. In contrast, all baseline models predict object locations that significantly deviate from the ground truth. On the right of Figure 5, we highlight the faithful generalization ability of our model even more challenging scenarios, where the test images are from a different dataset CLEVRTex featur- ing substantially different colors, textures and backgrounds compared to the training set. In this setting, our approach can still predict object location accurately, while baselines predict random location. Additional qualitative results are provided in Figure VIII and Figure IX.
Metrics. We evaluate the object discovery performance in terms of object perception rate and coordinate estimation error. The object perception rate measures the percentage of correctly discovered object relative to the total number of objects. To determine which objects are correctly dis- covered, we use Hungrian algorithm to match predicted coordinates and ground truth coordinates for all object in the scene. An object is considered successfully discovered the mean square error (MSE) between the predicted coor- dinates and the ground truth coordinates less than 0.002. The coordinate estimation error computed by averaging the MSE of the predicted coordinates and the ground truth coordinates across all objects.
Quantitative Results. We quantitatively compare our approach with baselines Table 1. Our method and Gener- ative Classifier demonstrate better perception performance than the discriminative baselines ResNet-50 and Slot At- tention, and our approach, by compositional modeling, achieves a 12.5% higher perception rate than Generative Classifier, with the margin increasing to 26.6% on the out- of-distribution tests. Meanwhile, our approach exhibits significantly lower coordinate estimation error than base-
Qualitative Results. We qualitatively illustrate that our approach can infer object coordinates from test images more accurately than baseline models, as shown in Figure 4. Fur- thermore, we demonstrate how our approach can generalize to images with a larger number of objects and more complex objects in Figure 5. On the left of Figure 5, our model can
6

---

Compositional Seene Understanding through Inverse Generative Modeling
Test image
ResNet-50
SlotAttn
GC
Ours
Test image
ResNet-50
SlotAttn
GC
Ours
0 0
00 4
ground-truth location
predicted location
ground-truth location
predicted location
images; CLEVR images with 6-8 objects (Left) CLEVRTex images with 6-8 objects (Right). Our model is trained with CLEVR images containing 3-5 objects. During inference time, given an oul-of-distribttion image that is substantially different from training data, our proposed approach can still infer the object positions accurately. In contrast, all baseline models predict object locations that significantly deviate from the ground truth.
Figure 5: Out-of-distributioo Object Discovery. Object perception results on
lines, especially in distiiitiitiitoinnnnn tests. Overall, these quantitative results demonstrate the strong generalization ca- pability of our proposed approach to more complex scenes than those seen during training.
In-distribution (female faces)
Out-of-distribution (male faces)
Models
ResNet-50
79.6%
62.2%
GC (Li et al., 2024)
79.1%
61.7%
GC Variant
77.8%
58. %
Ablation Study. We further demonstrate the effectiveness of our proposed random multiple-iiitaalization initialization strategy in Sec 3.3. As shown in Table I, omitting the random multiple- initialization strategy leads to a significant degradation in object perception performance. In this case, the algorithm often converges to local minima that substantially deviate from the ground truth coordinates, even with an increased number of optimization steps. In contrast, adopting our pro- posed random multiple-initaaizzttinn strategy significantly improves the perception performance. Additional ablation study results can be found in Table IV.
IGM (Ours)
80.8% %
65.6 %
Table 2:
Accuracy of Facial Feature Prediction. Quantita- tive evaluation of facial fcature prediction results for both in- distribution (female faces) and out-of-distribution (male faces) set- tings on CelebA. Our model outperforms all baseline approaches in terms of classification accuracy for the in-distribution setting and generalize even much better for the out-of-distributio setting,
Metrics. We evaluate facial attribute prediction perfor- mance with classification accuracy. Classification accuracy is defined as the ratio of correctly classified images the total number of images, where an image is considered cor- rectly classified only if all the three attributes are simultane- ously predicted correctly.
4.2 Global Factor Perception
We further illustrate how our approach can infer global factors, such as facial attributes, from a test image and reliably generalize to images that differ substantially from training data.
Qualitative Results. We demonstrate how our approach can predict the presence of all three facial attributes in given face image in Figure 6. On the left of Figure 6, we show that, by explicitly composing the three attributes with compositional diffusion models, our approach provides more accurate facial attribute prediction results than base- lines. On the right of Figure 6, we further illustrate how our approach can faithfully predict facial attributes even in male faces, despite never having seen that during training, demon- strating stronger generalization compared baselines. Ad- ditional qualitative results are provided in Figure X.
Dataset. We evaluate our approach on the CelebA dataset (Liu et al., 2015) focusing on three attributes: Black Hair, Eyeglasses, and Smiling. The training set consists only female faces labeled with the these attributes, while the out-of-distribution test set comprises solely male faces.
Baselines. We compare our approach with both discrimi- native and generative approaches including ResNet-50 (He et al., 2016), Generative Classifier (GC) (Li et al., 2024), and a variant of Generative Classifier. Details on how these baselines are trained for the facial feature perception task can be found in Appendix A.5.
Quantitative Results. We quantitatively compare our ap- proach with baselines in Table 2. Our approach outperforms all baselines in classification accuracy for in-distribution images, and the performance gap becomes even pronounced for the distr--iistiituti tests. This strong generalization
7

---

Compositional Scene Understanding through Inverse Generative Modeling
Test image
Ground fruth
ResNet-50
GC
Ours
Test image
Ground truth
ResNet-50
GC
Ours
Black bair
Black-hair
Black-hair x
Black bair
Black v
Black-hair
Black-hair
Hlack-hair
Byeglasses
Eyeglasses X
Eyoglasses
Eyoglasses
Eyeglasses
Fyeglasses
Eyeglasses
Ryeglasses
I. Smilings
Smiling/
Smilings
Smiling
Smiling X
Smiling/
-millig
Smiling X
ssence
Black-hair x
Black-hair
Black-hair X
Black-hair X
Black-bair
Black-huir x
Black-hair x
Black-huir
Eyeglasses x
Eycglasses x
Eyeglasses
Eyoglisses
Eyeglasses x
Eyeglusses x
Eyeglasses x
Eyeglasses
Smiling x
Smiling
Smiling X
Smiling x
Smilings/
Smiling
Smilings
Smiling
Black-hair x
Black-bair
Black hair
Black-hair X
Black-hair
Black-hnir
Hlack-hair
Black-hair
Eyoglasses x
Eyeglasses x
Fyeglasses x
Ryeglasses x
Eyeglasses
Eyoglasses
Eyeglasses x
Eyeglasses
Smiling
Smiling
Smiling /
Smillig
Smilings/
Smilling x
Smiling
Smiling /
Figure 6: In-Distribution and cottr--Distribution Facial Feature Prediction. Facial feature prediction results for in-distribution (Left)
and out-of-distribution (Right) CelebA images. Our model is trained on female faces from CelebA. During inference, our model can accurately predict facial features consistent with the ground truth for both in-distribution female faces and out-of-distribution male faces.
to such non-trivee non-trivial distribution shifts demonstrates the ro- bustness of our compositional modeling strategy.
Test image
Ground Truth
DC A dog v A X
DC variant
Ours
dog v
dog v
Aog
Acat v
X
c
4.3 Zero-Shot Multi-Object Perception
A x
A
A rabbit v
A x
Finally, we demonstrate that our approach leverage pretrained diffusion models, such as Stable Diffusion (Rom- bach et al., 2022), for zero-shot multi-object perception tasks without requiring any additional training. Specifically, we compose a set of diffusion models, each conditioned on an individual text prompt, and minimize the average denois- ing error with respect to the text prompts following Eqn (7). The solution can then be obtained using Algorithm 2.
A /
Adog
dog
Aog /
car
v
Acat /
X
rabbit
A
A
rabbit
dog X
A dog
A o
A
car /
X
Acat
rabbit
A rabbiix
Arabbit
A
Figure 7: Zero-Shot Multi-Object Perception. Our approach can faithfully interpret given real-world images by predicting object categories that are consistent with the ground truth.
In our experiment, we evaluate our model on small dataset as detailed in Section A.3, each containing two animals from the set (dog, cat, rabbit}. The prompts corresponding to these object concepts are: "a photo of dog". "a photo of cat", and "a photo of rabbit". Our model composes two dif- fusion models, each conditioned on any two of the prompts to interpret given image: evaluates denoising error for the three possible prompt combinations; and selects the combination with the lowest denoising error as the optimal solution. In contrast, Diffusion Classifier (DC) (Li et al., 2023a) baseline uses a single diffusion model conditioned on compound prompts (e.g.. "a photo of a dog and cat") without explicitly modeling compositionality. The Diffu- sion Classifier Variant (DC variant) baseline also uses single diffusion model but conditioned on individual object prompts, and select the two prompts with smallest denoising error as perception results. Details on Diffusion Classifier and the variant for multi-object perception can be found in Appendix A.5.
Models
Accuracy t
Diffusion Classifer (Li et al., 2023a)
70.4%
Diffusion Classifer Variant IGM (Ours)
73.2%
87.3 %
Table 3: Accuracy of Zero-Shot Multi-Object Perception. By adopting pretrained Stable Diffusion without any additional train- ing, our compositional approach significantly outperforms Diffu- sion Classifier and its variant real-world images in terms of classification accuracy for the multi-object perception task.
sifier by margin of 16.9% in perception accuracy. This demonstrates that our proposed compositional generative modeling framework effectively enables multi-ohject scene understanding through leveraging pretrained models.
5 Limitations and Conclusion
As shown in Figure 7 and Figure XI, our approach can con- sistently recognize multiple objects in realistic images using pretrained generative models without any training. We fur- ther quantitatively compare our approach with basclines in Table 3, where our approach outperforms Diffusion Clas-
Limitations. For multiple discrete concept inference, our compositional modeling approach enumerates through each possible configuration for every concept and evalu- ate denoising error for all concept combinations. This can result in long inference time when the concept number is large. To scale to a large number of concept settings, we
8

---

Compositional Seene Understanding through Inverse Generative Modeling
developed a continuous approximation of our approach that allows gradient-based optimization in Algorithm 5 and Al- gorithm 6, thereby avoiding the exponential inference cost. Alternativly, several additional approaches can potentially significantly mitigate this computational bottleneck. One approach could be to use heuristic search algorithms on discrete values- for instance we can run beam search with a beam width of K over each attribute sequentially which can reduce time complexity from ((M to O(MK), making inference more efficient for large discrete spaces. Finally, since our approach allows parallel processing across the con- figurations, inference time can be drastically reduced given sufficient computational resources, potentially approaching the time required for single configuration evaluation.
References
Amari, S.-i. Backpropagation and stochastic gradient de- scent method. Neurocomputing. 5(4-5):185-196, 1993.
13
Amit, T., Shaharbany, T., Nachmani, E., and Wolf, L. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint (txiv:2//22003300 2021. 2
Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., and Lischinski, D. Break-a-scene: Extracting multiple con- cepts from single image. In SIGGRAPHI Asia 2023 Conference Papers, pp. 1-12, 2023. 2
Bengio, Y. Towards compositional understanding of the world by agent-based deep learning. In NeurIPS'2019 Workshop on Context and Compositionality in Biological and Artificial Neural Networks, 2019.
Another limitation is the assumption of concept indepen- dence. Our compositional generative modeling approach assumes object concept independence given the input image, enabling combinatorial generalization beyond the training distribution. However, one possible limitation of this full independence approximation that it ignores the interaction between objects, which are erucial in many real-world sce- narios. As remedy, we could potentially learn additional models that model interactions between object components, which can also be composed to represent more complex scenes.
Biederman, I. Recognition serriittion----oomee componeess theory of human image understanding. Psychological review, 94 (2):115, 1987.
Brempong, E. A., Kornblith, S., Chen, T., Parmar, N., Min- derer, M., and Norouzi, M. Denoising pretraining for semantic segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4175-4186, 2022. 2
Conclusion. We have presented an inverse generative modeling approach to scene understanding tasks by compo- sitionally combining a set of generative models. We illus- trate how compositionality enables the inference of visual concepts from test images that differ substantially from train- ing data. By adopting pretrained text-to-image generative models, our model can even achieve zero-shot multi-object perception without requiring any additional training. We be- lieve that exploring compositions of various foundation mod- els at test time can be an promising direction to build intelli- gent perception systems that can generalize more effectively.
Chen, H., Dong, Y., Shao, S., Hao, Z., Yang, X., Su, H., and Zhu, J. Your diffusion model is secretly certifiably robust classifier. arXiv preprint 1rX1v:2402.02316, 2024.
2
Cho, W., Ravi, H., Harikumar, M., Khuc, V., Singh, K. K., Lu, J., Inouye, D. I., and Kale, A. Towards en- hanced controllability of diffusion models, 2023. URL https://arxiv.orgaab////3 111223022..1146..
Chomsky, N. Aspects of the Theory of Syntax. The MIT Press, Cambridge, 1965.
Impact Statement
No immediate negative social impact is anticipated from our proposed approach in its current form, as we focus primarily on scene understanding tasks using standard dataset. Given the strong generalization capability to handle more complex scenes than those seen during training, our approach has the potential to benefit various fields such autonomous driving, robot manipulation, and augmented reality, among others. Furthermore, our approach can be environmentally friendly, since our framework can leverage pretained text-to- image generative models directly for zero-shot multi-object scene understanding tasks without requiring any additional training, thereby reducing the carbon footprint.
Clark, K. and Jaini, P. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2024.
Cong. Y., Min, M. R., Li, L. E., Rosenhahn, B., and Yang, M. Y. Attribue-centric compositional text-to-image gen- eration. arXiv preprint DxXiv2301100413. 2023.
Du, Y. and Kaelbling, L. Compositional generative model- ing: A single model is not all you need. arXiv preprint arXiv: a4Xiv:2402.011003 103, 2024. 2,
Du, Y. and Mordatch, I. Implicit generation and gen- eralization in energy- based models. arXiv preprint arXiv: 1903 08689, 2019.

---

Compositional Seene Understanding through Inverse Generative Modeling
Du, Y., Li, S., and Mordatch, I. Compositional visual gener- ation with energy based models. In Advances in Neural Information Processing Systems, 2020. 2
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:/610.02136. 2016.
Du, Y., Li, S., Sharma, Y. Tenenbaum, B. J., and Mordatch, Unsupervised learning of compositional energy con- cepts. In Advances in Neural Information Processing Systems, 2021.
Hinton, G. E. To recognize shapes, first learn to generate images. Progress in brain research, 165:535 547, 2007. 2
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances Neural Information Process- ing Systems, 1116840-68511 2020. 2
Du, Y. Durkan, C., Strudel, R., Tenenbaum, B., Diele- man, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. arXiv preprint a arXiv:2302. 1552, 2023. 2
Huang, L., Chen, D., Liu, Y., Shen, Y., Zhao, D., and
Zhou, J. Composer: Creative and controllable image
synthesis with composable conditions. arXiv preprint arXiv:2302.0977, 2023.
Feng, W. He, X., Fu, T.-J., Jampani, V., Akula, A., Narayana, P. Basu, S., Wang, X. E., and Wang, W. Y. Training- farin structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.005032, 2022. 2
Jaini, P., Clark, K., and Geirhos, R. Intriguing properties of generative classifiers. arXiv preprint arXiv:2309.1677 2023. 2
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 10, 2017. 16, 17
Fodor, J. A. and Lepore, E. The compositionality papers. Oxford University Press, 2002.
Fodor, J. A. and Pylyshyn, Z. W. Connectionism and cogni- tive architecture: critical analysis. Cognition, 28(1-2): 3-71, 1988. 1
Gal, R., Alaluf, Y.. Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint wtX.x:220880061115 2022.
Karazija, L., Laina, I., and Rupprecht, C. Clevrtex: A texture-rich benchmark for unsupervised multi-object seg- mentation. arXiv preprint 1rXiv2211..1026. 2021. 1. 15, 17
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., and Cohen-Or, D. Encoder-based domain tuning for fast personalization of cext-to-image models. ACM Transactions Graphics (TOG), 42(4):1-13, 2023. 2
Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83- 97, 1955. 19
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. Imagenet-traing trained cnns are biased towards texture; increasing shape bias improves ac- curacy and robustness, arXiv preprint 1r11121811,12331, 2018.
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E., and Pathak, D. Your diffusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2, 8, 18, 20
Geirhos, R., Jacobsen, J..H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, A. Shortcut learn- ing deep neural networks. Nature Machine Intelligence, 2(1 676656667, 2020.
2023a.
Li, A. C., Kumar, A., and Pathak, D. Generative classi- fiers avoid shortcut solutions. In ICML 2024 Workshop on Structured Probabilistic Inference f(,&} Generative Modeling, 2024. 2, 6, 7, 17,
Greff, K., Van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 2020.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-199999 19742. PMLR, 2023b.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 6, 7, 17, 18,
2
10

---

Compositional Scene Understanding through Inverse Generative Modeling
Li, S., Du, Y., Tenenbaum, J. B., Torralba, A., and Mor- datch, I. Composing ensembles of pre-trained models via iterative consensus. arXiv preprint arXiv:2210.1152,, 2022. 2
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and
Ommer, B. High-resolution image synthesis with latent diffusion models, 2022. 8
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted InterentionnMICCA. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 334-241.33 Springer, 2015. 17
Liu, N., Li, S., Du, Y., Tenenbaum, J., and Torralba, A. Learning to compose visual relations. Advances Neural Information Processing Systems, 34:23166-23172 2021.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, B. Compositional visual generation with composable diffu- sion models. arXiv preprint arXtv:2206.01714, 2022 2,
Schott, I.., Von Kügelgen, J., Träuble, F., Gehler, P., Rus- sell, C., Bethge, M., Schölkopf, B., Locatello, F, and Brendel, W. Visual representation learning does not gen- eralize strongly within the same domain. arXiv preprint arXi::210700822/ 2021. 17
3
Liu, N., Du, Y., Li, S., Tenenbaum, B., and Torralba, A. Unsupervised compositional concepts discovery with text-to-image generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
Seitzer, M., Horn, M., Zadaianchuk, A., Zietlow, D., Xiao, T., Simon-Gabriel, C.-J,, He, T., Zhang, Z., Schölkopf, B., Brox, T., et al. Bridging the gap to real-world object- centric learning. arXiv preprint arXiv:2209. 0tte::2229.180000 14860, 2022. 6, 19
pp. 2095-20095.20 2023.
Liu, Z., Luo. P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730- 3738, 2015. 7, 17
Shao, R., Wu, T., and Liu, Z. Detecting and recovering se- quential deepfake manipulation. In European Conference Computer Vision, pp. 712-728. Springer, 2022. 17
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention, 2020. 6, 17,
Shepard, R. N. and Metzler, J. Mental rotation of three- dimensional objects. Science, 171(3972)7701-703, 1971.
19
Mahajan, D., Pezeshki, M., Mitliagkas, I., Ahuja, K., and
Shi, C.,, Ni, H., K., Han, S., Liang, M., Mishne, G., and Min, M. R. Compositional image generation and manipulation with latent diffusion models. 2023. 2
Vincent, P. Compositional risk minimization. arXiv preprint arXiv:2410.06303 2024. 2
Netanyahu, A., Du, Y., Bronars, A., Pari, J., Tenenbaum, J.. Shu, T., and Agrawal, P. Few-shot task learning through inverse generative modeling. arXiv preprint arXiv:2411.049977 2024.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015. 2
Ng, A. and Jordan, M. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing sys- tems, 14, 2001. 2
Sohn, K., Shaw, A., Hao, Y., Zhang, H., Polania, L., Chang, H., Jiang, I., and Essa, I. Learning disentan- gled prompts for compositional image synthesis. arXiv preprini 1rXiv:2306.00763 2023. 2
Nie, W., Vahdat, A., and Anandkumar, A. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34:13497-13510, 2021.
Su, J., Liu, N., Wang, Y., Tenenbaum, J. B., and Du, Y. Com- positional image decomposition with diffusion models. arXiv preprint arXiv:2406. 19298, 2024.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize imagenet? In Interna- tional conference on machine learning, pp. 1389-5400. PMLR, 2019.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribu- tion shifts in image classification. Advances in Neural Information Processing Systems, 33: 111188883-8899 2020.
Redmon, J. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Vapnik, V. N., Vapnik, V., et al. Statistical learning theory. 1998. 1

---

Compositional Scene Understanding through Inverse Generative Modeling
Wang, X., Li, S., Kallidromitis, K., Kato, Y., Kozuka, K., and Darrell, T. Hierarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024.2 2
Wang, Y., Liu, L., and Dauwels, J. Slot-vae: Object-centric scene generation with slot attention. In International Con- ference Machine Learning, pp. 36020-36035. PMLR, 2023. 2
Wiedemer, T., Mayilvahanan, P., Bethge, M., and Brendel, W. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36: 6941-6960 2023. 17
Zhao, W. Rao, Y., Liu, Z., Liu, B., Zhou, J., and Lu, Unleashing text-to-image diffusion models for visual per- ception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5729-5739, 2023. 2
Zhou, S., Du, Y., Chen, J., Li, Y., Yeung. D.-Y., and Gan, C. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377 2024. 2
12

---

Compositional Scene Understanding through Inverse Generative Modeling
A Appendix
Test image
Prediction
Test image
Prediction
In the Appendix, we first present additional results for vari- ous scene understanding tasks in Sec. A.I. Next, we conduct further experiments Sec. A.2, and provide dataset details in Sec. A.3, training details in Sec. A.4, and baseline details in Sec. A.5. Finally, we discuss potential extensions of our approach in Sec. A.6.
A.1 Additional Results
Local Factor Perception. We illustrate additional qual- itative results for the object discovery task in Figure VIII and Figure IX. Our model, despite trained only on CLEVR images with 3-5 objects, not only successfully generalizes to CLEVR images containing a larger number (6-8) of objects, but also CLEVRTex images containing objects with sub- stantially different colors, shapes, textures and backgrounds compared to the training images. These qualitative results demonstrate how our proposed compositional modeling by composing a set of diffusion models enables strong general- ization beyond training data.
Global Factor Perception. We further illustrate addi- tional qualitative results for the facial attribute prediction task in Figure X. We train model female face images only from the CelebA dataset to predict facial attributes: Black Hair, Eyeglasses, and Smiling. During inference, our model is tested on male face images that differ signif- icantly from the training images. As shown in Figure X, our model can accurately predict facial attributes from male faces, demonstrating strong ability generalize to non- trival distribution shift.
ground ground-truth location
predicted location
Figure VIII: Object Discovery Generalization. Object discovery results on CLEVR images containing 3-8 objects. Despite trained with CLEVV images containing 3-5 objects, our model can effec- tively generalize
Zero-Shot Multi-Object Perception. We illustrate ad- ditional qualitative results for the zero-shot multi-object perception task Figure XI. We apply our proposed compo- sitional inverse generative modeling framework directly to pretrained Stable Diffusion with any further training. Given real-world images, our model can predict object categories accurately, demonstrating effective zero-shot scene under- standing ability.
with larger number of ohjects.
only the ground truth number can best a given image and leads to small denoising error, as reflected in Figure 3 and Figure XII. Similar to Algorithm 3, we outline the concept number inference algorithm in Algorithm 4, where amine each possible concept number K Kinn,-. ...: Kmar. figure out best concepts e under each K, evaluate average denoising error for each K, and select the one configuration of K with smallest average denoising error as the concept number estimate.
A.2 Additional Experiments
Visualization of Concept Number Inference. In Fig- ure 3, we demonstrate how denoising error can serve criterion for selecting concept number in the object discov- ery task. To more intuitively motivate this approach, we further illustrate visualization results Figure XII show how inferred concepts differ from ground truth concepts when the concept number does match the ground truth concept number. We can that when the concept number differs from the ground truth concept number, either some concepts are missed or extra concepts are inferred. This concepts mismatch results in large denoising error, and
Multiple Random Initialization Strategy Ablation. We discussed the importance of multiple random initialization strategy for continuous concept inference top of Stochas- tic Gradient Descent (Amari, 1993) in Section 3.3 and we presented the results of an ablation study in Table 1. To further demonstrate the effectiveness of this approach, we visually illustrate how single random initialization may con- verge to either an optimal solution local minima Fig-
13

---

Compositional Scene Understanding through Inverse Generative Modeling
In-distribution (3-5 objects)
Out-of-distribution (6-8 objects)
Models
Perception Rate ?
Estimation Error
Perception Rate
Estimation Error +
IGM
l-initialization (Ours)
72.8%
6,9e
68.0%
7.8e
IGM
5-initialization (Ours)
89.6%
2.0e
79.1%
5.4e -4
IGM
10-initialization (Ours)
90.5%
1.9c
81.6%
4.6e
IGM 5-iintialization (Ours)
92.8%
1.6e
84.3%
3.5c -4
IGM 20-initialization (Ours)
94.7% %
1.4e
85.3%
3.5c
Table IV: Ablation Study of Multiple Random Initialization Strategy. We illustrate quantitative evaluation of object perception results CLEVR for both in-distribution (3-5 objects) and out-of-distriuution (6-8 objects) test settings. The object coordinates are inferred using Algorithm: 3 with varying numbers of random initializations for all concepts. The quantitative results indicate that the number of initialization starting points increases, the perception performance improves consistently. This demonstrates that our proposed multiple random initialization strategy can help avoid convergence to local minima for continuous concept inference.
Test image
Prediction
Test image
Prediction
Test image
Prediction
Test image
Prediction
Black-bair x
Black-hair x
Fyeelasses
Eyeglosses
Smiling v
Smiling x
Black-bair
Black-hair x
c
Eyeglasses
Eyeglasses x
Smiling
Smiling /
Black-bair x
Black-hair
Eyoglasses
Fyeglasses
Smiling
Smiling
Black-hair
Black-hair
100
Eyeglasses
Eyeglasses
Smiling
Smiling
Black-hair
Black-hair x
Eyeglasses
Eyeglasses
Smiling
Smiling
Black-hair
Black-hair
Eyeglasses
Eyeglusses x
Smiling X
Smiling
Figure X: Facial Attribute Prediction Generalization. Facial attribute prediction results ClebaA images. Despite trained with only female faces from CclebA, our model can effectively generalize to new scenes containing male faccs.
ground-truth location
predicted location
Figure IX: Object Discovery Generalization. Object discovery results CLEVRTex images containing 3-8 objects. Despite trained with CLEVR images containing 3-5 objects, our model can effectively generalize to new CLEVRTex scenes containing larger number of objects with different colors, shapes and textures.
the capability converge the optimal solution. In Table IV, we further quantitatively demonstrate that as the number of random initialization starting points increases, the perception performance of our model improves.
ure XIII. By employing multiple random initializations, our approach ensembles several starting points and correspond- ing optimization paths, achieving significant improvement
Prompt Weighting for Multi-Concepy Perception. We conducted an additional comparison to see whether naive
14

---

Compositional Seene Understanding through Inverse Generative Modeling
Test image
Prediction
Test image
Prediction
Algorithm 4 Concept Number Inference Algorithm
A dog <
dog
1: Require: an image æ, trained denoising model ég
A cat
A cat
2: Initialize denoising error list E = zeros (aoosKaaa kmmn))
rabbit
A rabbit X
3: for K = -.min....Kmaa do
4: > IInitialize multiple (R) groups of concepts
A dog
A dog v
Initialize concepts {c,, C....,, C, R
N(O,1)
D Run Stochastic Gradient Descent
A cat
A cat X
7: for 1,...: Natep do
A rabbit
A rabbit
8:
En ~ 1),, UUii((1...... ,T)
9:
x²n = Vai, x + v1 Ctn
A dog v
A dog v
10: Ách + Vetelee -E - co(x"" ,t,, ef
A cat
v
A cat X
11:
end for
A rabbit
X
A rabbit
12:
> Evaluate denoising error for each configuration
13:
Initialize denoising error list ER, zeros( R))
A dog x
A dog
14:
for i = 1...., , N Nsample do
15:
C; ~ (0, t) Unif([ TH)
A cat
A cat
16:
x = Vai, x VI- E
rabbit
A rabbit X
17:
ror = 1,........ do
18:
ER(r] += ||E: -EE ca(a" Li cf
A dog x
A dog x
19:
end for
A cat
A cat /
20:
end for
21:
> Select the group with lowest denoising error
A rabbit
A rabbit
22: E[K]= min,.((.....R)) N En,rr
23: end for
A dog v
A dog /
K argmin
De KE (EEKKKmin (mminn.,,KK.. Xmax)
E[K]
A cat X
A cat X
24: return K
A rabbit
A rabbit
Figure XI: Zero-Shot Multi-Object Perception. Zero-shot multi- bbject perception results on natural images containg two animals from finite [dog, cat, rabbit). By leveraging pretrained Sta- ble Diffusion without any additional training, our model predicts object categories accurately.
(CLEVR and CelebA) that are commonly used for object dis- covery and multi-label classification. While these datasets are relatively simple, they serve as strong benchmarks for measuring effectiveness and generalization. Nonetheless, we believe that testing our model on more complex scenes would be interesting. To take a first step towards that end, we conducted additional evaluations on ClevrTex (Karazija et al., 2021), which features diverse object colors, textures, shapes, and complex backgrounds. As shown in the Ta- ble VI, our model outperforms all baselines in object dis- covery on ClevrTex, demonstrating its potential scalability to more complex scenarios.
prompt weighting could solve the zero-shot perception task, using the Compel package. Specifically, we applied prompt weighting to the compound prompts including "a photo of a cat++, a dog, and a rabbit", "a photo of cat, dog++, and rabbit", and "a photo of a cat, dog, and rabbit++*. The underlying idea is that if the image contains specific concepts (e.g., a cat and a dog), then the prompts "a photo of a cat++, a dog, and a rabbit" and "a photo of a cat, a dog++, and a rabbit" are expected to result in lower denoising error (higher likelihood) than the prompt "a photo of a cat, a dog, and a rabbit++". To determine which two objects present in the scene, we choose the two prompt weighted compound prompts that have the lowest denoising error. We report the zero-shot perception accuracy in Table V. The results suggest that simple prompt weighting may not be sufficient for effective multi-concept inference in this setting.
Inference Time. To evaluate inference efficiency, we conducted a comparison of runtime performance between our method and baseline models on an NVIDIA H100 GPU. We evaluate inference time for discrete concept inference on CelebA considering more attributes. As shown in Table VII, the inference time of our approach is comparable to the baseline model Generative Classifier. To further enable our model to work on large number of concept settings (i.e., larger K). we developed continuous approximation of our approach that allows gradient- based optimization, thereby avoiding the exponential (M k) inference cost. The gradient- based discrete concept inference algorithm is outlined in
Object Discovery on CLEVRTex. Our model has been quantitatively evaluated on widely adopted datasets
15

---

Compositional Scene Understanding through Inverse Generative Modeling
Test image
Num =
Num =
Num =
Num = 6
Num 7
Num 8
ground-truth location
predicted location
Figure XII: Visualization Object Number Inference Object Discovery Tasks. 3-5 objects. During inference, given a test image, we try to use K 3, ..., 8 object coordinates to fit the image through our inverse generative modeling approach following Algorithm 3 and Algorithm 4. We can see how the estimated coordinates mismatches ground truth coordinates when the object number differs from ground truth number.
our model CLEVR images containing
Models
OOD Accuracy f
lar to the previous setting, we have developed a continuous approximation for the zero-shot object perception task to im- prove inference efficiency. The gradient-based multi-object perception algorithm outlined in Algorithm
Diffusion Classifer
70.4%
Compel
35.2% %
Ours
87.3%
Specifically, for each concept (e.g.. "a photo of cat"), we assign a learnable weight its corresponding noise prediction in the compositional model. These weights then optimized via gradient descent. After optimization, we select the top two concepts with the highest optimized weights as the predicted objects in the scene. As shown in Table VIII this continuous relaxation leads to a significant reduction inference time, with the time complexity scaling linearly with the number of candidate concepts (O(K))
Table V: Accuracy Zero-Shot Perception with Prompt Weighting. We evaluate the perception accuracy of the prompt weighting approach on the animal dataset and demonstrate that our approach significantly outperforms it.
Algorithm 5.
Specifically, to infer binary labels with gradient-based opti- mization, we relax the learnable binary labels to continuous parameters in the range (0, 1). These continuous parameters are optimized using gradient descent and clamped to (0, 1) at each iteration to remain valid. After optimization, we decide label if the corresponding optimized relaxed parameter is smaller than 0.5, otherwise the label is As is shown in Table VII, the continuous gradient-based approach reduces inference time significantly, making it scale linearly with the number of concepts (O(K)).
A.3 Dataset Details
Object Discovery. We train our compositional generative model on 26132 CLEVR images (Johnson et al., 2017). each containing 3-5 objects of varying color, shape, size and texture. We resize the training images to resolution of 64 x 64. For each image, the 2-D center coordinates of objects are also available, which are of continuous value and normalized between 0 and by dividing the coordinates in terms pixel value by the image resolution. The out-of- distribution test set consist images with 6-8 objects either
Additionally, we also provide the runtime of our approach and baselines on the zero-shot object perception task. Simi-
16

---

Compositional Scene Understanding through Inverse Generative Modeling
In-distribution (CLEVRTex 3-5 objects)
Out-of-distribution (CLEVRTex 6-8 objects)
Models
Perception Rate t
Estimation Error +
Pereeption Rate t
Estimation Error +
ResNet-50 (He et al., 2016) SlotAttn (Locatello et al., 2020)
3.9%
2.0e
1.8%
2.0c
41.9%
1.5e
35.2%
1.6e -3
GC (Li al., 2024)
69.6%
9.8e
52.9%
1 1.4g
Ours
85.2%
5.1e
72.4%
7.8c
Table VI: Accuracy of Object Discovery. Quantitative evaluation of object perception results on CLEVRTex for both in-distribution (3-5 objects) and did-triiiitittii (6-8 objects) test settings. Perception rate and estimation error are reported. Our approach outperforms all the baselines, and the margin is especially significant for outoof-distributio setting, demonstrating strong generalization capability.
Models
OOD Accuracy
Inference Time |
GC (Li et al., 2024)
51%
28.49s
IGM (Ours) IGM (Ours)- continuous approx
60 %
29.10.
55%
22.15s
Table VII: Accuracy and Inference Time of Global Factor Perception. The inference time of both our approach and Generative Classifer are evaluated on CelebA considering attributes including "black hair", "chubby", "cyyglasses", and "smiling". To avoid exponential computation cost for discrete concept inference through enumeration, we further develop a continuous approximation of our approach through gradient-based search algorithm, which significant reduces inference time while maintains generalization performance.
from the CLEVR dataset (Johnson et al., 2017), or from the CLEVRTex dataset (Karazija et al., 2021).
the in-distribution set includes all relevant factors but not all of their possible combinations, while the out-of-distribution set comprises novel combinations of these same factors (Schott et al., 2021; Wiedemer et al., 2023). However, in- ducing such combinatorial shifts in a controlled manner is not feasible with real-world datasets. Our primary goal is to evaluate the generalization ability of our compositional inference framework under challenging out-of-distribution (OOD) conditions by ensuring that the test data is substan- tially different from the training data. For example, in object discovery, the model is trained on Clevr while tested on a different dataset ClevrTex. Following the same spirit, we decided to train our model on CelebA female faces while testing it on CelebA male faces, as there are significant facial characteristic differences between the two groups.
Facial Feature Prediction. For our facial feature experi- ments, we trained our model with 40612 face images from the CelebA dataset (Liu et al., 2015). For each face image, three attributes are available including: Black Hair, Eyeglasses, and Smiling, each represented by categorical values 1-1, We represent the attribute labels with one- hot encoding during training. The -itoo--distribtiinn test set consists of male faces only.
Zero-Shot Multi-Object Perception. For the zero-shot multi-object perception task, our approach directly lever- ages pretrained text-to-image generative requiingg no additional training data. To evaluate the perception per- formance of our proposed approach in this task, we manually collected a small real- world dataset consisting of 71 ran- dom realistic images from the Internet, each containing two animals from (dog, cat, rabbit). Specifically, this dataset consists of 20 images containing a cat and a dog, 22 images containing a cat and a rabbit, and 29 images containing a dog and rabbit. In line with the common practice text- to-image generative models, the prompt corresponding to these object concepts are: "a photo of cat", "a photo of dog", and "a photo of rabbit".
A.4 Training Details
We train a conditional latent diffusion model with latent space of channels and resolution 8 x 8, which uses pre- trained VAE encode input images into the latent space, The latent space image is scaled with factor of 0.18215. The denoising network adopts the Unet architecture (Ron- neberger et al., 2015) as commonly used in diffusion models that takes the latent space image as input along with label conditioning and outputs noise predictions. Specifically, the input for the denoising network is of 8 and the cross attention dimension is 2 (the object coordinates dimension is 2) for object discovery and 6 (the one-hot encoding of facial attributes is of dimension 6) for facial feature predic- tion. We use 1000 diffusion steps and linear beta schedule for training. For other hyperparameters, we use batch size 128 and a learning rate 2e 5
Other Datasets. Although our approach in this work fo- cuses on the binary task of facial feature detection, it has the potential to handle more complex tasks, such as detecting edited facial attribute sequences explored in SeqDeepFake (Shao et al., 2022), which we leave for future work. Fur- thermore, in CelebA, the division between in-distribution and out-of-distribution data can also be defined such that
17

---

Compositional Scene Understanding through Inverse Generative Modeling
Models
OOD Accuracy t Inference Time +
Diffusion Classifer (Li al., 2023a)
70%
99. 99.44s
IGM (Ours)
87
179.96s
IGM (Ours)- continuous approx
75%
101.05s
Table VIII: Accuracy and Inference Time of Zero-Shot Multi-Objeet Perception. The inference time of both approach and Diffusion Classifer evaluated the zero-shot multi-object perception task. To avoid exponential computation cost for discrete concept inference through enumeration, we further develop a continuous approximation our approach through gradient- search algorithm, which significantly reduces inference time while maintains generalization performance.
Test image
Initialization
Initialization
Initialization 3
Algorithm 5 Gradient-based Discrete Concept Inference Algorithm
b
1: Require: an image trained denoising model eg
2: Initialize relaxed continuous labels t' .'....""" .....I E (0, 1)
3: & Construct psudo one-hot encoding for labels
4: c [1, 1- 1,,0000000 0, O..., 0,
e² [0,0,121 12,0... 0,0]
c 00,0,0,000..,, 1
& Run Stochastic Gradient Descent NNstpp do 10: En ~ N),, ta Unif([1......
9: for n
11: x'n = v x /1 0i En
12: I + -V l|en
Σ -i=1 E c0(x""n "la'n,tna tn c )12
13: D Clamp 1 to (0,
14: 1t + .*..lmmp(001)
15: end for
16: if
< 0.5 then
17: 1 +-0
18: else
19: t +-
20: end if
21: return 1', 1²,1² ...,, 1K
A.5 Baselines Details.
We compare our model against multiple discriminative and generative basclines. In this section, we introduce details on how these baselines are trained for scene understanding tasks considered in Section 4.
ground-truth location
predicted location
Figure XIII: Visualization of Random Concept Initialization in Object Discovery Tasks. We train model with CLEVV images containing 3-5 objects. During inference time, given test image, we initialize object cnordinates randomly and run stochas- tic gradient descent to iteratively refine the coordinates. Due to the non-convexity of the problem, the optimization procedure can converge either the optimal solution or local minima, depending on initialization values. We propose to employ multiple random initializations, so that our approach can ensemble several starting points and corresponding optimization paths, achieving significant improvement the capability converge the optimal solu- tion.
ResNet-50 for Object Discovery. For the object discov- ery tasks, the maximal number of objects in images is 5 in the training set and 8 in the test set. To enable ResNet-50 (He et al., 2016) to infer coordinates from images with 8 objects, we append a linear layer with input dimension 2048 and output dimension 16 on top of ResNet-50, followed by sigmoid layer that outputs values between 0 and 1. The outputs is further reorganized into 8 matrix with each row representing center coordinates of an object. We match the model output with ground truth object coordinates by minimizing the MSE loss to train the model. Since the dimension of ground truth coordinates in training data is 2, where 3 < K we pad additional K coordi-
18

---

Compositional Seene Understanding through Inverse Generative Modeling
as described earlier for Slot Attention.
Algorithm 6 Gradient-based Zero-Shot Perception Algo- rithm
Generative Classifier for Object Discovery. Genera- tive Classifier (Li et al., 2024) is originally proposed to solve single-label classification problems by using diffusion models, where they try to find the categorical class that minimize denoising error. To infer object coordinates of continuous values, we adapt Generative Classifier by train- ing generative model that takes multiple object coordinates as conditioning. During inference, can inverse the gen- erative model and solve an optimization problems to find a set of object coordinates that best describe the image. For a fair comparison, we train this model following our proposed model. The only difference is that they train a single denois- ing network taking all coordinates as conditioning, while we train set of denoising networks each taking an indi- vidual object coordinates as conditioning for compositional modeling. During inference, they can follow our proposed inference procedure in Algorithm 3.
1: Require: an image at, trained denoising model c0. prompt' == "A photo of a cat", prompt = "A photo of a dog", prompt = ="A photo of rabbit"
2: >> Get Text Embeddings for Prompts
3: c* = <TextEmbedding(proe (promp!*)
4: Initialize concept weights w², w², w
5: & Run Stochastic Gradient Descent
6: for n = ......,sscp. do
7: Gn (0, tn Unif([1,.... T))
8: x:' --=-arr x + /T- Cit,,.n En
9: c -VVlian Σk wn wn co("", faanttt tn, 1²2
10: end for
11: Select the two indices with largest weights
12: indices- = top2((w', w², w)")
13: return cindices
ResNet-50 for Facial Attribute Prediction. ResNet-50 (He et al., 2016) has been widely used to solve classification problems. is straightforward to apply ResNet-50 to solve the facial attribute prediction task. We append a linear layer with input dimension 2048 and output dimension 3 on top of ResNet-50, followed by a sigmoid layer that probability values between O and 1. We supervise the model outputs with ground truth facial attribute labels by minimizing the BCE loss. During inference, ResNet-50 choose class label with high probability as classification results.
nates with values (1, 1) representing empty coordinates (no object) to match the dimension of model output.
Slot Attention for Object Discovery. Slot Attention (Locatello et al., 2020) is an unsupervised discriminative method for object discovery with strong generalization per- formance, which, however, only provides segmentation masks without giving the center coordinates of objects. For modify and train Slot Attention with object coordinates supervision. Specifically, Slot Attention learns a set of slots that compete with each other through cross attention mechanism to interpret a given image. These slots represent a high level description of objects in the image. To enable slot attention to predict object locations, instead of decoding slots into pixel components, code them into individual object coordinates. We supervise the decoded object coordinates outputs with ground truth coordinates by minimizing the MSE loss, where the coordi- nate matching is achieved with Hungarian Algorithm (Kuhn, 1955). To enable this supervised version of Slot Attention to be able to infer object coordinates from diitr--diitiibutionn images with object number reaching 8, we set the slot num- ber to be 8. Since the number of ground truth coordinates in training data is K, where 3- < K < 5, we pad additional 5 - K coordinates with values (1, 1) representing empty coordinates (no object) to match the dimension of model output.
fair comparison,
Generative Classifier for Facial Attribute Prediction. Generative Classifier (Li et al., 2024) originally can only solve single-label classification tasks. To enable Generative Classifier to perform multi-label classification, we train a diffusion model taking all three facial attributes as condi- tioning. During inference, we can enumerate through all possible facial attribute combinations (e.g., combination 1: "black hair, eyeglasses, smiling", combination 2: "not black hair, eyeglasses, smiling". etc.) and evaluate denoising er- rors. The one combination with smallest denoising error is selected as multi-label classification results. Again, how Generative Classifier in this case differs from our model lies in the lack of compositional modeling.
de-
Generative Classifier Variant for Facial Attribute Pre- diction. For Generative Classifier Variant, the training procedure is the same Generative Classifier, but the infer- ence procedure is different. In stead of enumerating attribute combinations, we can evaluate these attributes separately. Specifically, for attribute "black hair", we can evaluate the denoising error of "black hair" and "not black hair" condi- tioning and determine one of them with smaller denoising error as classification results. We then follow the same pro- cedure classify other attributes. This inference approach is desired to avoid unaffordable computation complexity
DINOSAUR for Object Discovery. DINOSAUR (Seitzer et al., 2022) is an enhanced extension of Slot Atten- tion, which aggregates and reconstructs high-level semantic features extracted from the pretrained self-supervised DINO model, leading to improved object discovery performance. Like Slot Attention, DINOSAUR outputs only segmentation masks and does not provide object center coordinates. We adapt DINOSAUR for object discovery the same manner
19

---

Compositional Scene Understanding through Inverse Generative Modeling
when the number of labels is very large.
Diffusion Classifier for Multi-Object Perception. Dif- fusion Classifier (Li et al., 2023a) is originally propose to solve zero-shot single-label classification problems by using pretrained text-to-image generative models without requiring any training. To adapt Diffusion Classifier for multi-label classification, we can feed Diffusion Classifier prompt that describes a combination of multiple concepts as text conditioning and evaluate denoising error. For example, to determine if an image contains cat and dog in our task, Diffusion Classifier can evaluate the denoising error of the following prompts: "a photo of a cat and a dog". "a photo of a cat and rabbit" and "a photo of dog and a rabbit".. The prompts with smallest denoising error is selected as the classification results.
Diffusion Classifier Variant for Multi-Object Perception. Diffusion Classifier Variant differs from Diffusion Classi- fier by evaluating each prompts separately. Given an image containing two animals from a finite set fdog, cat, rabbit), Diffusion Classifier evaluates the denoising error of follow- ing prompts: "a photo of a dog". "a photo of cat " and "a photo of rabbit", and then choose the two prompts with smallest denoising error multi-label classification results.
A.6 Future Work
Extending our approach dynamic interactive scenes presents exciting direction for future work. In the dy- namic setting, the reconstruction objective can be reformu- lated to involve reconstructing an entire video of the target interaction, rather than a single static image. Each composed generative model can be conditioned on different aspects of the interaction, such as the identity of individual objects or agents within the environment. The inverse generative mod- eling procedure and track objects over time, even when they are temporar- ily occluded, as well as to infer the distinct behaviors of individual agents in the interactive scene.
then be used consistently discover
Another interesting direction is to extend scene understand- ing, such as inferring a complete scene-graph cenne-rrph. In this setting, each composed factor in systems would correspond to an "edge" relation between pair of objects. We could then enumerate possible edges within the graph, seeking combination whose composed set of relations yields an accurate reconstruction the scene.
20


Reviews:
[{"review_id": "PS4Ttd8mMO", "reviewer": "Reviewer_7Q2x", "summary": "This paper presents a computational framework for mining the structural properties of natural scene images by recasting the problem as an inverse generative modeling task. Specifically, the authors propose a generic inverse generative modeling paradigm that integrates compositionality into a diffusion-based generative model, enabling robust generalization beyond the training distribution. During training, the model undergoes optimization by minimizing the average denoising error across discrete concepts. At inference, the framework performs a constrained search within a predefined range of concept cardinalities to identify the configuration that minimizes the average denoising error. For continuous concepts, the authors employ multiple randomized initialization points, iteratively discarding low log-likelihood candidates until convergence to a single optimal solution. Furthermore, they leverage stochastic gradient descent (SGD) to iteratively refine concept representations, enhancing computational efficiency. Empirical evaluations substantiate the efficacy of the proposed methodology, demonstrating substantial performance improvements across three scene understanding tasks—local factor perception, global factor perception, and zero-shot multi-object perception—on the CLEVR, CelebA, and a custom small-animal dataset. These results validate the proposed approach’s effectiveness and establish new state-of-the-art performance benchmarks in the domain of compositional scene understanding.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments! Please see our response below about your concerns.\n\n**1. CelebA images contain only a limited number of attributes (typically 2–3). How does the proposed method perform on tasks involving a greater number of attributes?**\n\nThank you for the insightful question! To demonstrate the effectiveness of our method on a greater number of attributes, we conducted an experiment on CelebA using 4 attributes (Black Hair, Eyeglasses, Smiling, and Wearing Hat), given our time and computational constraints. As shown in the table below, our model consistently outperforms baseline models in both in-distribution and out-of-distribution accuracy. These results demonstrate the potential of our approach to handle more than just 2–3 attributes effectively.\n\n|                           | In-distribution Accuracy (Female only) | Out-of-distribution Accuracy (Male only) |\n|:---:|:---:|:---:|\n| ResNet-50 Classifier      | 0.76                                   | 0.57                                    |\n| Generative Classifier     | 0.78                                   | 0.51                                    |\n| Ours                      | 0.78                                   | 0.60                                    |\n\n\n**2. For CelebA, it may be more appropriate to use the dataset from SeqDeepFake.**\n\nThank you for suggesting this interesting paper and dataset! After reviewing SeqDeepFake, we don’t think   that this dataset is directly applicable for our binary multi-label classification task. Our method requires binary labels for a fixed set of attributes, whereas SeqDeepFake provides edited attribute sequences with varying attribute annotation across images. Nevertheless, we find that the image captioning task in SeqDeepFake is very related and interesting, and would like to explore if our model can detect edited attribute sequences in future work. We will discuss the relevance of SeqDeepFake in the related work section in the next version of our paper.\n\n**3. For the custom small-animal dataset, providing statistical characteristics such as distribution and ensuring a more consistent and diverse set of categories would make the evaluation more convincing.**\n\nThank you for your helpful suggestion! To clarify, our custom small-animal dataset consists of 20 images containing a cat and a dog, 22 images containing a cat and a rabbit, and 28 images containing a dog and a rabbit. We will include this dataset distribution description in the next version to enhance transparency and improve the evaluation's clarity.\n\n**4. Lack sufficient discussion on the relationship between image captioning models like BLIP-2. The authors should clarify the advantages of their approach compared to existing image captioning models.**\n\nThank you for highlighting this connection to related work! By using pretrained text-to-image generative models (e.g., Stable Diffusion), our model can tackle image captioning tasks like BLIP-2. However, our approach is applicable to a broader range of scene understanding tasks other than image captioning. For example, by conditioning on object coordinates, our approach can perform object discovery tasks and even enable generalization to more complex scenes (many more objects) than seen at training. This flexibility and generalizability distinguishes our approach from traditional image captioning models. We appreciate your suggestion and will include a more detailed discussion in the next version.\n\n **5. In line 146, it would be clearer to use the full name, 'Energy-Based Model,' instead of the abbreviation when mentioning it for the first time.**\n\nThank you for pointing out the abbreviation issue! We will fix it in the next version.\n\n**6. In the zero-shot tasks, why does the diffusion classifier variant perform worse than the original version?**\n\nDiffusion Classifier is originally designed for single-label classification tasks and thus  performs more poorly  on the multi-object tasks we consider.  To apply Diffusion Classifier on our setting, we directly condition the generative model using compound prompts (e.g., “a photo of a dog and a cat”), as detailed in A.5, which we refer to as the Diffusion Classifier baseline in our experiments. We also further explore a single object variant of Diffusion Classifier for this compound setting, where we condition the generative model on individual concepts (e.g., “a photo of a dog”) and select the two classes with the highest score. Details of both methods are detailed in A.5. Since the Diffusion Classifier Variant uses individual concepts to fit multi-concept images, it has worse performance than Diffusion Classifier baseline.", "claims_and_evidence": "- In general, the claims of this paper are supported by clear and convincing evidence; however, two of the experiments are too trivial to fully validate the proposed method's claims.", "methods_and_evaluation_criteria": "- Evaluating the proposed methods on CelebA and the custom small-animal dataset may not be convincing, as the images contain only a limited number of attributes (typically 2–3).", "theoretical_claims": "- This paper does not present many theoretical claims. Furthermore, the proposed method is simply an extension of an existing approach", "experimental_designs_or_analyses": "- As mentioned above, the experiments on CelebA and the custom small-animal dataset are too trivial. For CelebA, it may be more appropriate to use the dataset from [1], which contains more attributes and is generated by StyleGAN. For the custom small-animal dataset, providing statistical characteristics such as distribution and ensuring a more consistent and diverse set of categories would make the evaluation more convincing.\n\n[1] SeqDeepFake: Detecting and Recovering Sequential DeepFake Manipulation", "relation_to_broader_scientific_literature": "- This method provides a framework for mining the structural properties of natural scene images. Unlike existing approaches that rely on text-to-image models, the proposed method leverages generative models with flexible conditioning, allowing it to be applied to a broader range of visual understanding tasks. This opens up a new direction for scene understanding.", "essential_references_not_discussed": "- In general, this paper adequately references related approaches.", "other_strengths_and_weaknesses": "- The implementation of the approach is methodical and straightforward, enhancing its practical applicability.\n- Comprehensive implementation details significantly improve the reproducibility of the research.\n- However, certain aspects lack sufficient discussion, such as the relationship between image captioning models like BLIP-2, which undermines the novelty and contribution of the work.", "other_comments_or_suggestions": "- In line 146, it would be clearer to use the full name, 'Energy-Based Model,' instead of the abbreviation when mentioning it for the first time.", "questions_for_authors": "- 1. How does the proposed method perform on tasks involving a greater number of attributes, as mentioned above?\n- 2. The authors should clarify the advantages of their approach compared to existing image captioning models.\n- 3. In the zero-shot tasks, why does the diffusion classifier variant perform worse than the original version? Can the authors explain this phenomenon and provide a theoretical rationale for its relationship with the proposed method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLh2RQGAAM", "reviewer": "Reviewer_6fxB", "summary": "This paper proposed an inverse generative modeling approach to understand attributes of a single image. This approach trains conditional generative models based on different conditions to construct a multi-conditional generative model through the composition of EBMs. The compositional modeling can generalize the generation ability to out-of-distribution images to generate combinations of conditions that have not been seen during the training phase. After the compositional generative model is trained, the inverse generative modeling algorithm is used to search for condition parameters that maximize the log-likelihood from a single input image as the inference results. For conditions with continuous attribute values, this paper proposed a gradient-based heuristic algorithm that can effectively search for optimal conditions in continuous space. The experiments show that the proposed method outperforms the comparative methods in tasks such as scene object number inference and object localization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments! Please see our response below about your concerns.\n\n**1. One concern is the fairness of comparison with unsupervised Slot Attention. I recommend to compare with more advanced DINOSAUR.**\n\nThank you for the insightful suggestion! We would like to clarify that the Slot Attention baseline we compare to in the paper has incorporated supervision on each head to the object coordinates in the scene (discussed in Appendix A.5), and thus has the same information as our approach. Furthermore, following the reviewer’s suggestion, we compared our model with DINOSAUR also with supervision on Clevr. As shown in the table below, DINOSAUR works better than Slot Attention, while our model outperforms DINOSAUR.\n\n||In-distribution|(3-5 objects)|Out-of-distribution|(6-8 objects)|\n|:---:|:---:|:---:|:---:|:---:|\n||Perception Rate|Estimation Error|Perception Rate|Estimation Error|\n|Slot Attention|80.4|0.00087|53.3|0.00130|\n|DINOSAUR|82.5|0.00084|59.0|0.00120|\n|Ours|94.7|0.00014|85.3|0.00035|\n\n**2. One concern is the efficiency of the algorithm. For discrete values, the size of the search space $M^K$ will increase exponentially.**\n\nThank you for the insightful question! Overall, we agree that our approach is substantially slower than existing feedforward methods, but we believe that the additional computational cost is justified by the ability of our method to generalize well to unseen complex scenes. It’s true that our naive algorithm has exponential growth of the search space ($M^K$) for discrete values. Beyond the early stopping strategy discussed in our paper, several additional approaches can potentially significantly mitigate this computational bottleneck. One approach could be to use heuristic search algorithms on discrete values – for instance we can run beam search with a beam width of K over each attribute sequentially, which can reduce time complexity to $O(M*K)$, making inference more efficient for large discrete spaces. Alternatively, continuous relaxation of discrete variables with gumbel-softmax/concrete relaxation could allow the use of gradient-based search and thus avoid enumerating all configurations. Finally, since our approach allows parallel processing across the $M^K$ configurations, inference time can be drastically reduced given sufficient computational resources, potentially approaching the time required for a single configuration evaluation. We appreciate your suggestion, and will include comparative experiments of the overall time complexity in our paper and discuss ways to speed up inference.\n\n**3. In Algorithm 3, should we need to update the original $c_r^k$?**\n\nYes, we should update $c_r^k$ with the gradient $\\Delta c_r^k$ in line 8. We will add an explicit update step ($c_r^k ← c_r^k - \\lambda\\Delta c_r^k)$  to Algorithm 3 in the next version.\n\n**4. In the reference, some formats are different from others, e.g., Lines 440-441.**\n\nThank you for pointing out the reference format issue! We will ensure reference format consistency in the next version.\n\n**5. In Table 3, why Diffusion Classifier Variant performs worse than the original Diffusion Classifier?**\n\nDiffusion Classifier is originally designed for single-label classification tasks and thus  performs more poorly  on the multi-object tasks we consider.  To apply Diffusion Classifier on our setting, we directly condition the generative model using compound prompts (e.g., “a photo of a dog and a cat”), which we refer to as the Diffusion Classifier baseline in our experiments. We also further explore a single object variant of Diffusion Classifier for this compound setting, where we condition the generative model on individual concepts (e.g., “a photo of a dog”) and select the two classes with the highest score. Details of both methods are detailed in A.5. Since the Diffusion Classifier Variant uses individual concepts to fit multi-concept images, it has worse performance than Diffusion Classifier baseline.\n\n**6. How to implement object discovery with Slot Attention? Is it possible to infer object number and locate objects only through masks extracted by Slot Attention.**\n\nThank you for the insightful question! To enable slot attention to predict object locations with supervision, we decode slot representations into object coordinates and enforce a coordinate prediction MSE loss by comparing the decoded object coordinates with ground truth, where we use Hungarian Algorithm to align decoded coordinates and ground truth. To infer object number, during training, we can enforce empty slots to predict a fixed coordinate [1,1] (the normalized rightmost corner of the image), while object slots predict the actual object coordinates. At inference time, if the inferred coordinates are greater than a threshold close to [1,1], we determine these slots as empty, while the rest are considered to contain objects. This approach allows Slot Attention to infer both object number and their locations.", "claims_and_evidence": "The claims made in this paper are clear, which are supported by the experiments.", "methods_and_evaluation_criteria": "The datasets and evaluation criteria of this paper make sense for the problem.", "theoretical_claims": "I checked the correctness of Equations 1-9. There is no proof in this paper.", "experimental_designs_or_analyses": "The experimental design and analysis are reasonable and effective.", "relation_to_broader_scientific_literature": "The main contribution of this paper is to design a novel inverse generative modeling algorithm, which infers attributes of a single image by searching optimal conditional parameters on a trained compositional generative model. The training of compositional generative models is based on several previous approaches.", "essential_references_not_discussed": "The introduction to related works is generally complete, but I am not sure if there are any important recent works that have not been discussed.", "other_strengths_and_weaknesses": "Strengths\n\nThe inverse generative modeling algorithm proposed in this paper does not require additional model training for image attribute inference. This makes the method have potential for direct application on existing compositional generative models. The experiments on multiple image understanding tasks also show the effectiveness and versatility of the approach.\n\nWeaknesses\n\nThe compositional generative models are trained based on given conditions and images, so the proposed approach requires annotations of the corresponding image attributes. But Slot Attention is trained by reconstruction loss, which does not require the annotations. Therefore, one concern is whether the fairness of model comparison. In addition, I recommend the authors to compare with more advanced object-centric representation methods. For example, DINOSAUR [1], which is an extension of Slot Attention that uses the representation from more powerful pre-trained visual models.\n\nOne concern is about the efficiency of the inverse generative modeling algorithm. If the number of conditions or the number of selectable discrete values of each condition are large, the size of the search space M^K will increase exponentially. It will be valuable to conduct comparative experiments on the inference efficiency of models.\n\n[1] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\", ICLR 2023", "other_comments_or_suggestions": "In Line 8 of Algorithm 3, after calculating the gradient, should we need to update the original c_r^k?\n\nIn the reference, some formats are different from other places, e.g., the paper link in Lines 440-441.", "questions_for_authors": "In Table 3, why does the Diffusion Classifer Variant perform worse than the original Diffusion Classifer?\n\nCould the authors explain how to implement object discovery based on Slot Attention? Although Appendix A.5 mentions that the supervised version of Slot Attention is used in the experiments, I wonder if it is possible to infer the number of objects and locate the objects only through the mask of each object extracted by Slot Attention.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVxxItfX85", "reviewer": "Reviewer_xCEx", "summary": "This paper casts scene understanding as an inverse generative modeling problem, where attributes of the scene are extracted by seeking parameters of the generative model that best fit a given image. To facilitate generalizability, the paper proposes to model visual scenes compositionally - by model sub components of the scene with smaller models, and compose them in the form of energy based functions. The paper proposes different optimization method to search for discrete and continuous attribute variables, and experimental results show that the proposed formulation can more accurately find attributes of images comparing to related methods, while being able to generalize to novel scenes that have different content distribution than training images.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments! Please see our response below about your concerns.\n\n**1. The tested datasets contain simple concepts (limited number of objects in CLEVR, limited set of attributes in face), so although performance evaluation positively supports the proposed idea, the method in its current state still cannot be extended to more complex real world scenes.**\n\nThank you for your insightful question! Our model is evaluated on widely adopted datasets (CLEVR and CelebA) that are commonly used for object discovery and multi-label classification. While these datasets are relatively simple, they serve as strong benchmarks for measuring effectiveness and generalization, as also acknowledged by reviewers 6fxB, Tv9z and s7kA. Nonetheless, we agree that testing our model on more complex scenes would be interesting. To take a first step towards that end, we conducted additional evaluations on ClevrTex [1], which features diverse object colors, textures, shapes, and complex backgrounds. As shown in the table below, our model outperforms all baselines in object discovery on ClevrTex, demonstrating its potential scalability to more complex scenarios. \n\n|                  | In-distribution |(ClevrTex 3-5 objects)| Out-of-distribution |( ClevrTex 6-8 objects)|\n|:---:|:---:|:---:|:---:|:---:|\n|                        | Perception Rate | Estimation Error | Perception Rate | Estimation Error |\n| **ResNet-50**          | 3.9             | 0.00196          | 1.8             | 0.00198          |\n| **Slot Attention**     | 41.9            | 0.00152          | 35.2            | 0.00161          |\n| **Generative Classifier** | 69.6         | 0.00098          | 52.9            | 0.00135          |\n| **Ours**              | 85.2            | 0.00051          | 72.4            | 0.00078          |\n\n[1] Karazija, et al., 2021 Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.\n\n**2. When inferencing a continuous concept, how to decide how many random trials are needed in order to escape a local optima solution?**\n\nThank you for raising this important point! We acknowledge that escaping local optima in continuous concept inference can be influenced by the number of random trials, as demonstrated by the ablation study (Table IV) of our submission. To determine the optimal number of trials, a principled approach is to monitor the reconstruction  error of the input image: if increasing the number of trials no longer leads to significant reconstruction improvements, further trials become unnecessary. In practice, our experiments show that a moderate number of trials (5 or 10) is sufficient for continuous inference tasks. \n\n**3. For the tested perception tasks, how is the inference time of the proposed method compared to the related baselines ?**\n\nThank you for suggesting the evaluation metric! We run our model and baselines on Nvidia H100 and report inference times in the table below, where “N/A” indicates that the approach is not applicable to the task. Our results show that generative approaches, including our approach and Diffusion Classifier, require more time than the discriminative approaches like ResNet-50 and Slot Attention, while our approach consumes more time than Diffusion Classifier. Specifically, our approach takes longer than Diffusion Classifier due to the need to evaluate a composition of multiple diffusion models. However, this increased inference time for our approach comes with significantly improved generalization performance. Additionally, inference time bottleneck could be further mitigated through parallel processing, which our algorithm supports, or using heuristic search algorithms such as beam search to speed up optimization, which we leave for future work. We appreciate your suggestion and will include the inference time metric in the next version of our paper and discuss how improving inference speed is an interesting direction of future work.\n\n|                           | Local Factor (Clevr) | Global Factor (CelebA) | Zero-Shot (Pretrained)|\n|:---:|:---:|:---:|:---:|\n| ResNet-50     | 0.011s  | 0.023s |   N/A  |\n| Slot Attention     | 0.009s    |   N/A   |   N/A  |\n| Generative Classifier     |   102.874s    |    28.485s   |   99.436s   |\n| Ours     |    146.854s   |    29.095s    |   179.957s  |", "claims_and_evidence": "The claim that compositional inverse generative modelling enables more effective scene understanding is supported empirically by local factor perception (object counting, position estimation on CLEVR, CLEVREX datasets), global factor perception (facial attributes prediction on CelebA dataset), and zero-shot multi-object perception, of which the proposed method show better results than related previous methods.  \nBut the datasets used in the experiments mainly contain simple concepts (CLEVR dataset with limited number of objects, or human face of a fixed set of attributes), as also mentioned in the limitation section of the paper, it's no practical to scale the proposed method (in its current state) to more complex, realistic real world scenes.", "methods_and_evaluation_criteria": "method:\n1. The compositional generative modeling formulation of the proposed method makes intuitive sense. However, in order to escape local optima, the success inference of continuous concepts relies on multiple random trials. The number of random trials may vary depending on the complexity of the concept, which may not guarantee that the method will succeed in problems that are significantly different than the tested problems in the paper without further tuning this hyperparameter.\n\ndatasets:\n1. The tested datasets contain simple concepts (limited number of objects in CLEVR, limited set of attributes in face), so although performance evaluation positively supports the proposed idea, the method in its current state still cannot be extended to more complex real world scenes.\n\nmetrics:\n1. As mentioned in the limitations in the paper, the searching process of the proposed method is long due to exhaustive search nature, so when comparing with related methods, in addition to the perception accuracy, it'd be better to add inference time comparison as well.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes, I checked all the subsections of the experiment section. No obvious issues.", "relation_to_broader_scientific_literature": "The proposed method approximates the conditional probability distribution of the scene under different concepts with product of probability under individual concept, this is related to [1]; It further uses the denoising function in diffusion model to estimate the gradient of energy function, which is inspired by [2]\n\n\n\n\n\n\n[1] Du, Yilun, and Leslie Kaelbling. \"Compositional generative modeling: A single model is not all you need.\" arXiv preprint arXiv:2402.01103 (2024).\n[2] Liu, Nan, et al. \"Compositional visual generation with composable diffusion models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "essential_references_not_discussed": "I haven't found missing essential references.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy the follow\n2. The description of experimental setups are is comprehensive", "other_comments_or_suggestions": "please see questions", "questions_for_authors": "1. When inferencing a continuous concept, how to decide how many random trials are needed in order to escape a local optima solution? \n\n2. For the tested perception tasks, how is the inference time of the proposed method compared to the related baselines ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jicBkqH04", "reviewer": "Reviewer_Tv9z", "summary": "The paper introduces an innovative inverse generative modeling framework designed to perform compositional scene understanding. Its primary innovation is to interpret visual scenes as compositions of smaller generative models, enabling effective generalization to complex or unseen scenes. Specifically, the authors propose training a generative model through diffusion processes and then using this model inversely, inferring conditional parameters (eg, object categories, locations, facial attributes) from images. Major contributions include the introduction of compositional modeling into generative inference, achieving improvement in generalization capability, and a demonstration of the method's broad applicability across various scene understanding tasks, including object discovery and zero-shot perception.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and comments! Please see our response below about your concerns.\n\n**1.How scalable is the compositional inverse generative modeling approach when dealing with a very large number of visual concepts or categories? Would an alternative optimization strategy reduce complexity without sacrificing accuracy? Clarify computational overhead or potential methods to reduce complexity in real-world deployments.**\n\nThank you for the insightful question! Overall, we agree that our approach is substantially slower than existing feedforward methods, but we believe that the additional computational cost is justified by the ability of our to generalize well to unseen complex scenes. When dealing with a large number of visual concepts and categories, our naive algorithm of enumeration has exponential growth of the search space ($M^K$) for discrete values. Beyond the early stopping strategy discussed in our submission, several additional approaches can potentially significantly mitigate this computational bottleneck. One approach could be to use heuristic search algorithms on discrete values – for instance we can run beam search with a beam width of K over each attribute sequentially  which can reduce time complexity from $M^K$ to $O(M*K)$, making inference more efficient for large discrete spaces. Alternatively, continuous relaxation of discrete variables with gumbel-softmax/concrete relaxation could allow the use of gradient-based search and thus avoid enumerating all configurations. Finally, since our approach allows parallel processing across the $M^K$ configurations, inference time can be drastically reduced given sufficient computational resources, potentially approaching the time required for a single configuration evaluation. We will discuss these possible extensions in the future work.\n\n\n**2. Consider discussing the limitations of compositional approximations explicitly in the paper.**\n\nThank you for the valuable suggestion! We will add a limitations section in the next version of our paper and discuss this. Our compositional generative modeling approach assumes object concept independence given the input image, enabling combinatorial generalization beyond the training distribution. However, one possible limitation of this full independence approximation is that it ignores the interaction between objects, which are crucial in many real-world scenarios. As a remedy, we could potentially learn additional models that model interactions between object components which can also be composed to represent more complex scenes.  We will explicitly discuss this limitation and potential remedies in the limitations section.\n\n**3. Could you further elaborate on how this compositional modeling approach would handle dynamic or interactive scenes (eg, videos)? Have you considered any temporal extensions or adaptations?**\n\nThank you for the insightful question! Extending our approach to dynamic or interactive scenes is an exciting direction for future work that we are also currently exploring. In the dynamic setting, we can change the reconstruction objective to not be reconstructing a given image but instead to reconstruct the entire video of the target interaction. We can condition each composed generative model on different aspects of the interactions such as the identity of an individual objects or agent in the environment. The inverse generative modeling procedure can then be used to persistently discover objects in the scene, even if they are occluded at different points in time, or to infer the individual behaviors of each individual agent in the interactive environment. We will update the paper and discuss this in future work.", "claims_and_evidence": "The claims made in the paper, primarily that compositional inverse generative modeling improves generalization on various visual tasks, are well supported by clear experimental evidence.\nThe experiments across diverse tasks (ie, object detection on CLEVR and CLEVRTex datasets, facial attribute classification on CelebA, and zero-shot object recognition using pretrained Stable Diffusion) demonstrate improved performance over multiple baselines.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are appropriate for the addressed problems.\nUsing established benchmarks (CLEVR, CLEVRTex, CelebA) and pretrained diffusion models (Stable Diffusion) provides relevant metrics, such as perception rate, estimation errors, and classification accuracy, all of which clearly measure method effectiveness and generalization.", "theoretical_claims": "No explicit theoretical proofs were presented.", "experimental_designs_or_analyses": "The experimental design and analyses presented in the paper are sound and robust.\nThe authors differentiate between in-distribution and out-of-distribution scenarios, clearly showcasing generalization.\nThe ablation studies, such as testing the effects of the multiple-initialization strategy, further strengthen the validity and thoroughness of their experiments.", "relation_to_broader_scientific_literature": "This paper effectively builds on prior work in generative modeling (diffusion models, generative classifiers) and compositional visual understanding.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n1. The novelty of applying compositionality explicitly within inverse generative modeling.\n2. Significant and demonstrable improvements over strong baseline models, especially in generalization scenarios.\n3. Effective demonstration of zero-shot capabilities using pretrained generative models, indicating high practical relevance.\n\nWeaknesses:\n1. The computational complexity involved when enumerating concept combinations might limit scalability for tasks involving numerous discrete concepts.", "other_comments_or_suggestions": "Minor suggestions include:\n1. Clarify computational overhead or potential methods to reduce complexity in real-world deployments.\n2. Consider discussing the limitations of compositional approximations explicitly in the paper.", "questions_for_authors": "1. How scalable is the compositional inverse generative modeling approach when dealing with a very large number of visual concepts or categories? Would an alternative optimization strategy reduce complexity without sacrificing accuracy?\n2. Could you further elaborate on how this compositional modeling approach would handle dynamic or interactive scenes (eg, videos)? Have you considered any temporal extensions or adaptations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "qotAyFhf85", "reviewer": "Reviewer_s7kA", "summary": "This work demonstrates how generative vision models can be composed to enable robust compositional scene understanding. The proposed solution enables different scene understanding tasks involving discrete or continuous factors, such as localization and multi-attribute classification. The method can also incorporate pretrained diffusion models to enable zero-shot classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and comments! Please see our response below about your concerns.\n\n**1. For the experiments on CelebA, I am wondering why the authors opt to use all male faces as the OOD set, which effectively removes the female/male attribute from classification. The more standard subdivision of the dataset in ID/OOD regions for compositional tasks would be to ensure that the ID set contains all values of all factors, but not all combinations, e.g., all women are only shown smiling, all men are only shown with a neutral face in the ID set.  See, e.g., [1,2].**\n\nThank you for your insightful question and the suggested references! Our primary goal is to evaluate the generalization ability of our compositional inference framework under challenging out-of-distribution (OOD) conditions by ensuring that the test data is substantially different from the training data. For example, in object discovery, the model is trained on Clevr while tested on a different dataset ClevrTex. Following the same spirit, we decided to train our model on CelebA female faces while testing it on CelebA male faces, as there are significant facial characteristic differences between the two groups. However, we agree that subdivision of the dataset in [1,2], where all attribute values are present in the training set but not all combinations, is also helpful for evaluating compositional generalization, and we will update the paper to discuss this evaluation procedure. Unfortunately, this type of combinational shift is not available to gather in a controlled manner in real-world datasets, leading us to use the CelebA dataset.\n\n[1] Schott et al., 2022, Visual Representation Learning Does Not Generalize Strongly Within the Same Domain\n\n[2] Wiedemer et al., 2023, Compositional Generalization from First Principles\n\n\n**2. What is the dataset used in Sec. 4.3 and Fig. 7/Tab. 3? Is this a dataset curated by the authors? A comparison on CelebA with pretrained models in zero-shot mode could have been equally insightful, or the authors could have opted for an established multi-label classification dataset.**\n\nThank you for the insightful question! Yes, the dataset used in Sec. 4.3, Fig. 7, and Table 3 was curated by the authors and consists of 20 images containing a cat and a dog, 22 images containing a cat and a rabbit, and 28 images containing a dog and a rabbit. Our primary focus in these experiments was to evaluate multi-object perception performance (local factor discovery) using compositional pretrained models compared against Diffusion Classifier. We agree that testing these models on CelebA to discover global attribute factors could provide additional insights and will do so in the final version of the paper.\n\n**3. Small typo L198: ignore --> ignoring**\n\nThank you for pointing out the typo! We will fix it in the next version.\n\n\n**4. How feasible would it be to extend this method to more \"involved\" types of compositions, such as part-whole relationships or hierarchical compositions? Could this method (in principle, given unlimited compute) be extended to infer, e.g., a complete scene-graph of an input?**\n\nThank you for the insightful question! We believe that our method would in principle be able to scale to more “involved” types of compositions, such as inferring a complete scene-graph. In the setting of a scene graph, each of the composed factors in our systems would correspond to an “edge” or relation between two objects. We then enumerate through different edges in a graph until we get a graph whose composed set of edges leads to a good reconstruction of the scene. We will update the paper and discuss this in the future work.", "claims_and_evidence": "1. The proposed method can accurately infer concepts.\n    - The prediction of continuous factors is demonstrated well by the results in Tab. 1 and Figs. 4&5, where the method achieves impressive performance.\n    - The prediction of discrete factors is demonstrated well in Tab. 5 and Fig. 6.\n2. The proposed method can accurately infer the number of concepts.\n    - This is supported well by Fig. 3.\n3. The proposed method can incorporate pretrained diffusion models.\n    - This is supported well by Fig. 7 and Tab. 3.", "methods_and_evaluation_criteria": "- The proposed optimization scheme for determining $K$ and the set of $c_k$s is sound and explained well.\n- The evaluation is sound and uses established standard metrics.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental design for object localization using CLEVR and multi-attribute classification using CelebA is sound and uses established datasets.\n\n- For the experiments on CelebA, I am wondering why the authors opt to use all male faces as the OOD set, which effectively removes the female/male attribute from classification. The more standard subdivision of the dataset in ID/OOD regions for compositional tasks would be to ensure that the ID set contains all values of all factors, but not all combinations, e.g., all women are only shown smiling, all men are only shown with a neutral face in the ID set. See, e.g., [1,2].\n- What is the dataset used in Sec. 4.3 and Fig. 7/Tab. 3? Is this a dataset curated by the authors? A comparison on CelebA with pretrained models in zero-shot mode could have been equally insightful, or the authors could have opted for an established multi-label classification dataset.\n\n[1] Schott et al., 2022, Visual Representation Learning Does Not Generalize Strongly Within the Same Domain\n\n[2] Wiedemer et al., 2023, Compositional Generalization from First Principles", "relation_to_broader_scientific_literature": "The basic compositional model follows prior work in the literature, as also noted by the authors. To the best of my knowledge, the inversion of this process is novel, as is the application to multi-target classification and localization.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper is overall clear, well-written and easy to follow. The method is an interesting and novel extension of prior works, and experiments are straightforward and demonstrate well the efficacy of the method.", "other_comments_or_suggestions": "- small typo L198: ignore --> ignoring", "questions_for_authors": "- How feasible would it be to extend this method to more \"involved\" types of compositions, such as part-whole relationships or hierarchical compositions? Could this method (in principle, given unlimited compute) be extended to infer, e.g., a complete scene-graph of an input?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an inverse generative modeling approach to understand attributes of a single image. After the compositional generative model is trained, the inverse generative modeling algorithm is used to search for condition parameters that maximize the log-likelihood from a single input image as the inference results. Some major concerns raised by the reviewers including:

1. Novelty: the proposed method is an extension of an existing approach (Reviewer 7Q2x, Reviewer 6fxB);

2. Complexity: the size of the search space will increase exponentially (Reviewer 6fxB); The computational complexity involved when enumerating concept combinations (Reviewer Tv9z);

3. Datasets: the custom small-animal dataset are too trivial (Reviewer 7Q2x); The datasets used in the experiments mainly contain simple concepts (Reviewer xCEx);

4. Practicality: It's no practical to scale (in its current state) to more complex, realistic real world scenes (Reviewer xCEx); How feasible would it be to extend this method to more "involved" types of compositions (Reviewer s7kA).

The authors tried to address most of the concerns and four reviewers are positive to this submission. After several rounds of discussion, Reviewer 6fxB is still not fully convinced and has major concerns on novelty and complexity/practicality:

> Reviewer 6fxB: I agree with Reviewer 7Q2x that this work is an extension of an existing approach. The idea of using generative models for classification is not unique to this paper—the Generative Classifier (GC) baseline already adopts a similar strategy. This paper extends that idea by using compositional generative models, but the current version does not adequately address the combinatorial explosion problem when scaling to a large number of concepts.

Despite the average positive rating, I agree with Reviewer 6fxB that the authors should clearly acknowledge the trade-off between computational efficiency and accuracy in the revision.
Decision: Accept (poster)