InCoDe: Interpretable Compressed Descriptions For Image Generation
Authors: Armand Comas
Aditya Chattopadhyay
Feliu Formosa
Changyu Liu
Octavia Camps
Rene Vidal
Venue: ICLR 2025 Poster (2025)
Keywords: Controllable Generation
Image Generation
Interpretability
Information Pursuit
Information Theory
Diffusion Models
Primary area: generative models
TL;DR: We present a framework to enable image generation through succinct, informative, and interpretable descriptions of images.

Abstract: Generative models have been successfully applied in diverse domains, from natural language processing to image synthesis. However, despite this success, a key challenge that remains is the ability to control the semantic content of the scene being generated. We argue that adequate control of the generation process requires a data representation that allows users to access and efficiently manipulate the semantic factors shaping the data distribution. This work advocates for the adoption of succinct, informative, and interpretable representations, quantified using information-theoretic principles. Through extensive experiments, we demonstrate the efficacy of our proposed framework both qualitatively and quantitatively. Our work contributes to the ongoing quest to enhance both controllability and interpretability in the generation process. Code available at github.com/ArmandCom/InCoDe.

Published as a conference paper at ICLR 2025
INCODE: INTERPRETABLE COMPRESSED DESCRIPTIONS FOR IMAGE GENERATION
Armand Comas'
Aditya Chattopadhyay Feliu Formosa'
Changyu Liu'
Octavia Camps'
René Vidal*
NNottheattrn University
JJohns Hopkins University
UUniversity of Pennsylvania
[comasmassaggu a, achattole jhu. edu
formosa.f, 1iu, changyu, o camps /eotteestttttteteenccoooo edu
ridalr@seat upenn.
ABSTRACT
Generative models have been successfully applied in diverse domains, from natural language processing to image synthesis. However, despite this success, a key chal- lenge that remains the ability to control the semantic content of the scene being generated. We argue that adequate control of the generation process requires a data representation that allows users to access and efficiently manipulate the semantic factors shaping the data distribution. This work advocates for the adoption of suc- einct, informative, and interpretable representations, quantified using information- theoretic principles. Through extensive experiments, demonstrate the efficacy of our proposed framework both qualitatively and quantitatively. Our work con- tributes the ongoing quest enhance both controllability and interpretability in the generation process. Code available at hwwpa//ooooommmmmoooo/eoeeee
INTRODUCTION
The ability to control the output of generative model is essential for practical applications various domains, ranging from natural language processing to image synthe- sis. For example, consider a scenario in which users may be interested in using online application to decorate their bedroom. This application could generate images that are semantically similar to an "inspiration" picture given by users, as depicted in Fi b), while allowing them to specify which aspects they want to preserve or change (Fig. [[c). Alternatively. if the user does not pro- vide an image, the application could ask them a series of questions about their preferences and generate set of im- age suggestions based on their answers, as seen in Fig. F d). To speed up this process and maximize user satisfaction, the app should prioritize asking the most relevant ques- tions upfront, reducing the number of interactions while ensuring the user's preferences are accurately captured.
Reference (A)
Re Re-generation
(b)
(C).
Are the hocshoets vea?? No - Yes
bunkbedk? No Does have window? =o
Figure 1: InCoDe Controllable Image Gen- cration and Editing: a user can provide a sample image (a) and request to generate new images that are semantically "similar" to the given (b). In addition, the user can also encode their preferences in the generated images by specifying certain attributes like color of bedsheets (c).
Designing such an application requires an image represen- tation that effectively captures the elements and properties most semantically significant the user for the specific task (c.g., bedroom decoration). The representation should be interpretable and easily modifiable based on the user's input. In addition, it should allow for measuring relevance and similarity within its domain. Using such representa- tion, the application could offer personalized suggestions and allow precise adjustments to the generated images, ultimately improving the user experience.
Work done prior joining Amazon.

---

Published as a conference paper at ICLR 2025
*hrtains
No
White wall
Yes
Dark wall
No
Window
Yes
Carpet"
Yes
Door
No
Ceiling ve
No
Bunk beds*
Yes
White curtains No
(a) SD V1-4
(b) SD XL
(c) Dall-B
(d) Ours (SD V1-4)
Figure 2: Text-to-image models fail when composing multiple concepts. Comparison with text-to-image
baselines. For a given subset of attributes: (a) Samples from Stable Diffusion v1-4 ( x0.95 params w.r.t InCoDe). (b) Samples from Stable Diffusion XL (+= x 10 params), (c) Samples from Dall-e 3 (xz x 15 params) (d) Samples from InCoDe (trained on top of Stable Diffusion v1-4). Red superscripts attributes indicate the models that fail to capture them in the presented samples.
Inspired by the above scenario, our goal is to develop generative models that can be controlled through interaction with high-level concepts. Arguably, plain textual scene descriptions provide an intuitive interface for controlling image generation.
However, controlling image generation through text is extremely difficult, because the semantic content of an image can be represented through diverse text captions, each providing a distinct perspective. Take, once again, photograph of a bedroom as an example. The scene could be described in terms of the objects it contains, the furniture style, the discernible colors, or the relationships among objects, since there is no canonical semantic representation.
Moreover, despite recent advances in text-conditioned image generation (Betker et al. 2023 Podell et al., 2023), constraining the generated image to a rich textual description that simul simultaneously en- compasses numerous semantic concepts remains challenge, illustrated in Fig, (...)c). Although this issue has been partially addressed by subsequent work (Feng et al. 2023), image generation still struggles with composition in the presence of negations and large conjunctions (Tbl. 2).
To address these difficulties, we propose to represent the semantic content of images by sets of predefined, user-oriented and task-specific questions and their answers. This allows for interpretable representations that thaat can be easily modified by the user, while specializing a generative model to the sub-distribution of their interest based the task (e.g. bedroom furniture distribution). Although this approach may limit the generality of the model, it prioritizes combining multiple concepts accurately.
Given an image and of questions to choose from, we propose to build interpretable representations by greedily selecting queries in order of information gain using an algorithm called Information Pursuit (Geman & Jedynak 1996: Jahangiri al., 2017 Chattopadhyay al. 2022). Specifically, every subsequent question is selected such that its answer has maximum mutual information (among the remaining questions in the set) with the image content conditioned on the history of question- answer pairs observed so far. Consequently, we specialize our generative model to synthesize images conditioned on these question-anse -aeserr based representations.
Our framework allows the user to efficiently and intuitively interact with the system and control generations by changing the answer to selected queries from the generated image's representation. Moreover, organizing the queries in order of information gain allows for natural information- theoretic definition of "semantic similarity", where two images are said to be k-similar they agree on the first & most-informative question -nnstion.answers..
Paper contributions. In this work, (i) We propose InCoDe, an information thcoretic framework based Information Pursuit (IP) for representing data in terms of concepts and using this repre- sentation to control image generation; (ii) We introduce novel adaptation method for conditioning text-to-image pre- trainedd diffusion models query-answer sets. (iii) We collected two new datasets along with sets of binary queries and answers about their content. (iv) We quantitatively and qualita- tively validate the effectiveness of InCoDe, showcasing its superiority over the selected baselines
Since they hased human intelligible questions and their answers.
2

---

Published as a conference paper at ICLR 2025
1s there TV? No
Querier Encoder
Append selection u [q(X))
QuerySetAnswerer Decoder
2(x)
dooc?
there TV? No
q
Next query
theee a window? Yes
Diffusion Model Generator
q(X)
X
VQA
there
add noise G
X
Figure 3: Overview: pipeline of InCoDe. consists of three main modules: Encoder that maps history of query-answer pairs S ((q..((X)) (x)) to the next asked query q, which is answered by providing q and reference image X to an off-the-shelff VQA module. Decoder that maps query- based representations to the complete query-answer set, and a diffusion basdd Generator that predicts the added noise e, to the corrupted image Xt, after t forward diffusion steps and conditioned the representations.
across a diverse set of scenarios, and provide additional results in the Appendix for query-types from different domain.
2
METHODS
In this section, we first formalize our choice for semantic data representation and introduce InCoDe, a new image generation framework, which can be controlled via compressed interpretable represen- tations. Then, we describe architecture designs to implement the proposed framework and discuss training strategics. Finally, we explain how to use InCoDe to generate and edit images.
2.1 INCoDE FRAMEWORK
We propose InCoDe, which uses interpretable compressed representations to provide control over image generation and semantic editing. An overview of InCoDe's 's training pipeline is depicted in Figure It consists of three main modules: an Encoder, which produces query-based representations by selecting elements from given query set and a reference image to answer the selected queries; Decoder, which maps these representations back to the complete query set with the corresponding answers for the reference image; and Generator, which uses reverse diffusion (i.e., gradual de- noising) to turn the representations into images. Next, we formally define the concept of query set procedure used to organize its elements.
and outline the information-theored
Notation. Let be a set of images, Q be a set of user-defined task-specific queries, and be the set of all possible answers. A query E Q is a function mapping a point in X E X' to point in A, ie. q: X + A. In the sequel, using a slight abuse of notation, we will denote a query-answer pair (g, a) for X simply (X). Finally, we denote by Q(X) the set of all query-answer pairs for X, Q).
Q(x) = (x(X)
Semantic Representation. We describe image X with sequence of
D(X)
(X) = 191 (X).... (X),. q, (X) E Q(X), L = D S IQ), where the queries are sorted in decreasing order of the information gain that they provide about all other query answers in Q(X).
Given an observation X = ro ob can be generated using the Information Pursuit (IP) algorithm (see Sec. Sec. for details), modified that each step it selects the most informative query towards recovering the complete set Qxx from the answers (2:bs)::
91 IP()) arg arg max (g(X)
QEQ
gx+1 IP (2 arg max (g(X): (1)b))
QEQ
where T(::)) is mutual information and 91:A (2x) denotes the history of far. The algorithm terminates once queries have been asked, where a hyperparameter that controls the compactness of the representation.
obtained so
Following Chattopadhyay et al. 2023). we use Variational IP (VIP) to efficiently implement IP. The main idea is learn function, called Querier, directly from data such that given any set of
3

---

Published as a conference paper at ICLR 2025
query-answer pairs it outputs the most informative next query, without the need to explicitly compute mutual information (which is challenging in high-dimensions). To do this, VIP employs a Querier 9. working as Encoder, and QuerySet Answerer, working as Decoder, as formally described next.
Encoder. Let (P))())) denote the power se of Q(X). A Querier function g P(Q(X)) Q greedily selects elements from Q that yield the most succinet representation of X for predicting Q(X). More specifically, given an observation X = rob the Querier takes as input a history of arbitrary length of queries and answers, S 91:A (2 = {qq (z) : qu (2.0b) } C Q(= and outputs the query [k+11 E Q that is the most informative about the complete set (() (equation [
To do this, we will seek query q" whose answer minimizes the KL divergence between the postenor (((()) X) and the conditional query set distribution p(Q(X) I q-
(X), where conditioning on should be read as conditioning on the event (a' E X (1=x(=')) = ))=0))
Since the above posterior depends on the data distribution and it is unknown, we will use a QuerySet Answerer f (x)) + (X)). which maps a set of query-answert to a distribution over Q(²)) and learns the posterior together with the Querier g. For this purpose, we adapt the VIP objective from Chay et al. (2023) to recover the set Q(X). More specifically, let DR. be the Kullback-Leibler divergence between two probability functions, and let S be a history of randomly chosen query-answe pairs. We find the querier g and the queryset answerer f by solving the problem:
Decoder.
min
Exx,sKL(p(((C) (X) ((Q()) S) f:g
where
q 9== E Q,
I (x),, I = s(((X) US.
In order, given the observed data point X z0 and an observed history S = qlsk (200b), the Querier [VQA] module in Figure3) = g(x:))) U 91.t (2:b)) the QuerySer Answerer, which outputs the distribution of answers to all the queries in Q given the partially observed set (1)
g selects a query g E Q, evaluates it on u (Visual and feeds the extended set 11:k+1(2
Chattopadhyay et al. 2023) prove that the optimal (f.g) obtained by VIP selects queries that maximize mutual information with respect to a target variable. In our appendix, we reformulate that proof with Q(X) as our variable of interest. In practice, since optimization over all possible functions f and g is challenging, we parametrize them as neural networks: fo and gé. respectively.
Generator. The role of the third module in InCoDe is generate image X, conditioned on representation D using a conditional generative model or Generator. Given representation D, the Generator produces samples X ~ p(X D). If the Querier was used to encode D from a data point X, then, an optimal system would generate X such that the agreement between Q(X) and ((X) is maximized. Notably, at inference- timee D can also be hand-crafted by the user by providing their desired answers to the queries sequentially asked by the Querier, as illustrated in Figs. T and 23.
Given the recent success in conditional generation, we propose to leverage Diffusion models for our generative module. In particular, utilize the Denoising Diffusion Probabilistic Method (DDPM) S. Next, we describe the optimization
(Ho et al.] 2020), conditioned on sets of procedure for training the Generator.
In generative modeling, we seek to maximize the marginal likelihood of the data Ezo~p(Xo) [ão (Xo) under our parametric model. This objective is often intractable. Instead, it is common to define and optimize the variational lower bound (VLB) of that quantity. In DDPM, simplified noise-matching objective is used, which is derived from the VLB (see Sec. A.2) and simplified as:
min Ex.. (
Xx, Xx) || (X X,) R min Ex...x..lll [Ile= - =x(
8
with ( Uniform(1,7). Xo X is the clean input data, and X, is the noisy data after forward diffusion steps. Here, ()(() )) is the DDPM's posterior, eg the parameterized noise estimator and er is the Gaussian noise added X₂ at time during the forward diffusion process. Please see the Appendix for the intermediate steps. As in Sec. A.2 we refer to the noisy version of Xo along the diffusion process as X,, with being the diffusion time and therefore the noise scale.
Here, we wish to sample from distribution conditioned on elements of (P))())), such as histories S (the sets of
pairs observed during IP) or representations D. We incorporate these
"A power set of a set is the set of all subsets of Q, including the empty and Q itself.
4

---

Published as a conference paper at ICLR 2025
conditions into objective by using the conditional distribution while preserving the objective function (Ho et al. 2020):
min E, x,s (xX, Xo) Po (x,,, min E, x,, [ll+ tx,
Using query-answer pairs conditioning variables has not been explored before. Even modestly- sized query set Q can lead to a combinatorial explosion in the possible values S can take. This makes optimizing equation challlengigg Next, in Section 2.2 we provide architectural and training details to address this challenge.
2.2 ARCHITECTURE DESIGN, TRAINING, AND SAMPLING
As noted above, the three main modules of InCoDe, the Querier, QuerySet Answerer and Generator, are parameterized as neural-networke In this section, we discuss our design choices for the scenario where Q consists of binary queries about the presencc/absence of semantic attributes image X.
Training the Querier and Duswerrr We parameterize both the Querier and the QuerySet Answerer as fully- fully-comnected networks, which are trained jointly using the objective function in equation 2 Note that this objective requires computing an expectation over all possible histories of pairs.
Since optimizing for all query-answer pair combinations is intractable, the networks must extrap- olate from limited examples. Following Chattopadhyay et al. (2023), we employ different query sampling strategies during training. At the initial stages of training, we use a Random Strategy where we randomly sample history of length M, with M ~ Uniform (0, IQ] - 1). Later, as the queried selections become more reliable, we refine them by employing Biased Strategy, where we sample a history of size M sequentially using the Querier's 's outputs. To answer queries at training time, either use the dataset ground-truth (if available) an off-the-shell VQA model
Tiiinable
SpC
LoRA
Conditioning the Generator with Queries. Current text-to-image models struggle to accu- rately generate images constrained by multiple concepts (as shown in Fig. 2 Therefore, we avoid relying natural language for this task. In our context, a Generator that can effectively combine multiple concepts within limited se- mantic space is preferable to one that attempts to generalize to all possible text prompts.
Yes
Init
Pre-trained Diffusion Model
Q(X
Embedder
Nuil Text
Sumpled
X,
X
Emihaldings
mask
Figure 4: Adapting pre-trained diffusion models to query-based conditions. Proposed scheme to fine- tune text-to-image diffusion models with LoRA Hu et al. (2022). We introduce small neural network (embedder plus ML. with zero-init for the last layer) that maps queries into null-text token features. Then, we use mask to train the model with a subset of queries.
Hence, we adapt the conditioning procedure of our diffusion-based generator to given query set Q. Given a query history S, we encode it into a set of feature vectors. Then, we employ cross- attention layers between these feature vectors and the image features at multiple hierarchies of the diffusion model backbone (UNet), usually done by text-to-image diffusion models (Zhang
al.
2023 Rombach et al. 2022).
Since our query-based conditioning setting is not typical in the literature, we lack access to large pre-trained diffusion models that be readily applied to our use case. Consequently, inspired by previous work (Zhang et al. 2023 Shi et al.) [024).. we propose a novel fine-tuning approach, illustrated Fig. As shown in the figure, our method uses module with two main blocks: (i) A LoRA Hu et al. (0022) system with rank 32 set as a fine-tuning framework for the frozen large diffusion model, and (ii) query embedder network, consisting of a feature embedding and an MLP with zero-initialized final projection. During training, we embed each query-answed pair ((X) QXX) individually, into the dimensionality a single text token using the embedder network. Then, the embedded query-answer pair is masked according to whether or not the query-answer is
Note that VQA model is used to obtain answers to query given input image, while the QuerySet Answerer predicts the answers all queries given the answers subset of queries from Q.
5

---

Published as a conference paper at ICLR 2025
present in S, summed to null-text token and fed to the UNet. Note that at the beginning of training, the UNet only sees null-text tokens and performs unconditional denoising. As the training advances, the network learns to adapt the given subset of data while utilizing the conditional information in S, which is sampled uniformly at random. By fine-tuning an existing pre-trained text-to-image diffusion model, such Stable Diffusion Rombach et al. (2022), our method can take advantage of the vast domain knowledge encoded in these models.
Beyond Ettriutee-Based Queries. InCoDe is not restricted binary queries. Our framework supports queries gathering different types of information about a datapoint, including non-binary answers that may even be continuous or spatially-awan queries. For example, in the Appendix we provide experiments conducted with location- based queries, where a query Q. requests the pixel values of an RGB image patch around the location (i.j) in X. Location- -asodd queries are different from attribute- based ones in two key aspects. Firstly, in this case, we can have Q(X) X, that is, asking all the queries in Q is the same as observing the image X. Secondly, since these questions are directly associated with image coordinates, they implicitly carry spatial structural information. These differences lead to a different architecture design. In this case, is convenient to parameterize a set image, where only the patches corresponding
S of location- basod queries and their answers to answered questions are visible. Thus, for location- queries we use a Querier consisting of convolutional blocks. A detailed explanation and results can be found in the Appendix (Sec. 1.1.1
The procedure for image generation is straightforward. The input to the Generator is representation of length L, either generated from reference image (Fig. [[), which can be modified by the user (Fig. [ or entirely handcrafted by the user by answering questions from the set Q selected by the Querier (Fig. 21). The Querier, starting from an empty history S == (. selects sequentially the most informative query from the query set Q, along with an answer produced by VQA model, or the user. Next, q(X) is appended to S and the process is repeated I. times until representation D is obtained. Finally, image samples are generated by using the Generator, conditioned on the representation D through a classifier-free sampling strategy (Ho & Salimans 2021).
Image Generation.
3 EXPERIMENTS
In this section we empirically evaluate the performance of InCoDe and provide analysis of its capabilities. In particular, study (i) its effectiveness in capturing the semantic content of an image by evaluating the Querier's ability to select queries that maximize information gain, as well as the faithfulness of the generated image the provided representations; and (ii) its editing and compositional capabilities by evaluating its ability to modify or generate an image consistent with a desired set of attributes. Please refer to the Appendix for additional results, including location-bas based queries and qualitative analyses that highlight the features and limitations of InCoDe.
Table I: Representations of the image size and qucry-set for different datasets.
Datuset (Image size)
Query Set Q
Size [Q]
CLEVR Colors (3 x 64 64)
Indicutor of presence of N objects of particular color
45
CelehA x 64 64)
Facial Attributes
40
LSUN Bedroom (3 x 512 512)
Room descriptor attributes
58
LSUN Churches (3 x 512 x 612)
Church and surroundings descriptor attributes
44
Datasets. We selected four datasets with increasing complexity, starting from simple cases with predictable outcomes based on human intuition and progressing towards more challenging ones. Following are the datasets considered:
A synthetic dataset (introduced Jo,oonso al. (2016)). where objects with different attributes are placed randomly in uniform background. For this set, we created an attribute- based query set with questions of the form: "Are there n or more objects of color Y?". This is asked for eight colors (including any color) and E [1, 5) for total of 45 queries. The only considered attribute for our purposes is the object's color, hence its name.
CLEVR Colors:
a special token that represents "no text".

---

Published as a conference paper at ICLR 2025
Reference
Regencration
Asphalt
Yes
People - Yes
Mountains -+ No
Figure 5: InCoDe offers intuitive tools adjust the semantic elements of generated images. InCoDe describes an image in the query-answer space and generates new, semantically aligned image. By changing the answer to specific queries, we can modify semantic attributes the new images.
CelebA: dataset with celebrity face images and 40 facial attributes provided with the images.
LSUN Bedroom and LSUN Churches: Datasets with 58 queries about the room layout and 44 queries about the elements around and belonging to each church, respectively. Both datasets were developed using off-the-shelf models, with a large language model (LLM) assisting in the creation of the query sets and visual question answering (VQA) model (BLIP, Li et al. (2022)) generating the answers. These datasets are key contribution of this work, filling gap where no existing datasets meet the specific requirements of our task, and have been made publicly available.
The types and number of queries used for each of these datasets are listed in Table can be found in the Appendix.
More details
Metries. We evaluate the effectiveness of the information acquisition strategy by predicting answers to the complete set of queries Q(X), using the representations gathered by each strategy. The set Q(X) is predicted with an attribute Classifier, trained to classify Q(X) from random histories S. Accuracy and F1 score are computed with a testset with 2k samples in all cases, comparing predicted answers with the ground truth for each acquired query.
To evaluate compositional generation, we compute alignment to the query-answen set by running BLIP to answer queries from Q on the generated images. We report accuracy and F1 score with respect to the ground-truth conditional signal, for the 10 queries with highest entropy.
Baselines. We compare our method with the following query -eleetsee strategies.
Random sampling baseline, where queries are sampled uniformly without replacement.
Decision tree with information gain by impurity criterion (DT--IC), where we choose the query with largest entropy (computed from the dataset), given the previous history of query-answer pairs. This bascline is the gold standard, as it seeks to maximize the entropy of the selected query given a history, which is our ultimate objective given (Xx: Q()) s) ((q(X) 8) (()())
Q(X), H (x(( s). However, since the entropy is estimated using empirical probabilities, DT-IC suffers from data fragmentation and long computation times for large data corpora.
Top K with impurity criterion (TopK-IC), where we select the top-k queries based on their answer's entropy in the training set.
Finally, for compositional generation experiments we compare against (i) Stable Diffusion V1-4 baseline text-to-image 1o-image model; and (ii) Structured Diffusion Feng et al. (2023), method to improve multi-concept conditioning in text-to-image models, specifically more accurate attribute binding and better image compositions. All methods are conditioned by text, and thus provide the attribute list as concatenated representations (e.g. "Photo of bedroom that does not have white walls, has window, ..."). Note that given the restricted length of text tokens accepted by the baselines (77 tokens) this experiment is performed with only 10 attributes from the query-set, selected by descending order of entropy in the dataset. All generative models (including ours) in our experiments have SD V1-4 as the base model for fair comparison. Note that our tine-tuning method can be plugged into other text-to-image diffusion models. More details are given in the Appendix.
3.1 QUALITATIVE EXPERIMENTAL RESULTS
Semantic Auto- Entoo-Encdding and Concept Manipulation. Figs. Tand 2 iillstrraee qualitative results for the LSUN Bedroom dataset. these examples, Figure shows four images generated using the complete description with I. 58 query-answer pairs, while Figure ((c) shows examples generated
7

---

Published as a conference paper at ICLR 2025
Generated Samples 8 with Learned Querier
[D]
5
10
20
30
44
Querier output:
Are thore or more abjects?
X
2. Are there more objects?
Are
oyan objects?
Are there more red objjects?
5. Are there or more green ofjects? =>-y6s
Are there
brown objects?
purple objects?
Are there more blue objects?
9. Are there more yellow objects?
10. Are there or objects? information)
Random Strategy
(AD
Figure Representation of InCoDe's performance CLEVR Colors dataset. We illustrate one example of generated samples by utilizing our model's Querier and the random baseline. InCoDe generates sequence of sensible queries and the Generator successfully generates an image that is semantically equivalent (as defined by our query set) to the reference image after only asking few queries.
Rare query
Contradictory
Curtains - yes
Collage yes (2.5%)
Can you window
Red curtains - yes (3.2%)
Can you plants from window yes
Figure 7: InCoDe behaves correctly in corner cases, Left: In case of generating concepts underrepresented the training set, images still capture them. Right: For apparently contradictory answers queries, the Generator tries to accomplish both the same time resulting in creative solutions.
by (left) changing one, and (right) two attributes. Fig. [ our method successfully generates samples based solely on a sequence of query-answers, while other methods fail.
Similarly, Fig. 5 presents an example LSUN Churches, where InCoDe regenerates a reference image by first representing it in Q(X), and then adjusts the answers to specific attributes.
In all cases, InCoDe results preserve quality while showcasing correct behavior. To appreciate similarity between generated and reference images, refer to the list of attributes for Bedroom and Churches found in Dataset details (Sec. A.3). More qualitative results can also be found in Sec. A.
Querier trajectory. In Fig. 6 see simple yer illustrative example of a trajectory generated by InCoDe, as well as one generated by a random baseline, with the corresponding generated images. The CLEVR Colors dataset is useful since it is easy for human to whether the output of the querier correct or not. For instance, in the depicted case, the Querier begins by asking a question about the number of objects and obtains answer 4. Next, it proceeds to ask about the colors by one and finds that there are or more objects of gray color and green color. Therefore, there is minimum of objects. Subsequently, iteration 10, In:oDe asks if there 3 or more objects, and receives negative response thereby completely capturing the semantics of the reference image as defined by our query set, which is only concerned with the number objects in the image and their color. As the result, the Generator manages to synthesis an image that matches our reference image in semantic content (again as defined by our query set).
Corner Cases. Figure illustrates the behavior of InCoDe when generating images from concepts that are underrepresented the training set, demonstrating that these concepts are successfully captured through the proposed fine-tuning process. Additionally, the figure presents example where InCoDe faced with seemingly contradictory answers to different queries and synthesis creative but implausible solution.
3.2 QUANTITATIVE EXPERIMENTAL RESULTS
Effectivvness of the Information acquisition strategy. The results for these experiments illustrated in Figure & where we plot Accuracy and Fl-score against description length, where higher values indicate better performance. Both DT-IC and TopK-IC compute the entropy and select the
8

---

Published as a conference paper at ICLR 2025
Clevr Colors
CelebA
030
0.5
0.-
0.50
042
0.0
020
20
number
LSSN Bedroom
LSSUN Churches
1.50
100
039
0.3
2.5
0.00
0.5
0.035
0.15
072
0.76
6.2
Gutymubbe
Cairy
Casryy
Ours
DT-IC
TopK-IC
Random
Figure 8: Performance evaluation of InCoDe using attribute- based queries. We report the Test Accuracy and Fl score of an attribute classifier on the predicted representations. InCoee outperforms the baselines in all cases, with increasing gains the datasets hecome more complex. DT-IC shows similar performance initially, but it suffers from data fragmentation, well long computation time.
query that maximizes The key difference that DT-IC takes into consideration each query-answe and fragments the data accordingly. Eventually, the method runs out of data points in the training set over which computing the entropy and thus it stops selecting queries (visible by the flattening of the curve). In other words, it suffers from data fragmentation. TopK-IC does not have this issue, but it is suboptimal as it does not account for history.
This experiment shows InCoDe consistently selects queries that yield better information acquisition strategy than the baselines, especially for more complex scenarios such as the LSUN datasets. In contrast, the CLEVR dataset simple and contains high degree of redundancy in Q(X). The latter favors entropy based methods while explaining the poor performance shown by the random strategy.
Composable generation. The Accuracy and Fl Score in Tbl. 2 for the LSUN datasets show the capacity of InCoDe combine multiple concepts into the same gen- erated sample. Note that the chosen metrics introduce a new source of етгог BLIP can produce bad predictions.
Table 2:
Our method generates images that respect the attributes more ofteen than baselines do. Quantitative evaluation in terms of Accuracy and F1-score for the top 10 queries, according to their entropy com- puted from their respective trainsct.
We can observe that InCoDe performs significantly better than Stable Diffusion (SD VI), which is the base method upon which InCoDe is built. It also outperforms SD XL. which a bigger, enhanced version of SD VI. Struc- tured Diffusion (eeng et al. (2023) is also based on SD; however, the modifications they propose negatively im- pact performance when dealing with the conjunction of numerous concepts.
LSUN Bedroom
LSUN Churches
Acc.
Fl
Acc.
Fl
SD V1
0.61
0.62
0.57
0.55
XI.
0.59
Stru. D
0.52
0.54
0.51
0.50
InCoDe
0.85
0.84
0.75
0.72
4 RELATED WORK
Interpretability in Machine Learning. A large number of papers are devoted to post-hoc inter- pretability. Recent research focuses on developing more principled frameworks where interpretability
part of the model's design ((Wu et al. 2021 Bohle et al., 2021 Alvarez Melis & Jaakkola, (018)). Another line of work learns latent semantic concepts or prototypes from data ((Sarkar et al, 2022
Nauta et al. 2021
Donnelly et al. 2022
al. 2018 Yeh al. 2020 and produces predictions

---

Published as a conference paper at ICLR 2025
by leveraging those concepts. Nevertheless, these learned concepts not always interpretable the end user. Instead. (Chattopadhyay et al.] (2022) introduced IP (with subsequent work in Gadgii et al. (2024), which produces predictions explained by interpretable query query-chains, allowing the user to define intermediate representations in the form of a query-set. This guarantees by construction that the resulting query-chain explanations will be interpretable.
Conditioning, Control and Interpretability In Diffusion Models. Diffusion models have been used for conditional generation with great success. While early conditioning diffusion models studied class-conditional generation (Dhariwal & Nichol 2021); [Ho & Salimans (202T)) recent focus has shifted text-to-image generation, with high-qualing results (Rombach al. (2022); |Saharia et al.) Nichol et al. (202T). Consequently, many works on controllability rely on the direct intervention of embedded text representations (Kawar et al. 2023); Ramesh et al. (2022) :
(2022); Ramesh et al. [
Avrahami et al. (2022); Kim et al. (2022); Feng et al. (2023)) In this case, image manipulation still relies on the ability of text-to-image models to compose multiple concepts, which often fail (Fig. 2. Tbl. 2 Alternative methods, such as those in Mokady et al. (2022); Hertz et al. (2023); Epstein ct al. (2023), leverage the analysis and manipulation of cross-altention maps between text tokens and U-Net features for control. Regardless of the text-based method, when control does not directly involve text prompts, interpretability and subsequent manipulation typically rely on post-hoc analysis of the network's features.
Beyond text-based conditioning, several efforts have focused on using alternative control signals to guide the generative process. Works such as Zhang et al. (2023): /Zheng et al. (2023); : Du et al. (2023); Meng et al. (2022); Li et al. (2023); Yang et al. (2023); Huang et al. (2023) are representative examples of approaches for controlling image generation by means of spatially grounded inputs. These signals include contours, bounding boxes, masked images, depth maps, and sketches, often integrated with textual inputs. Interpretability relies on the user's ability to understand these control signals, which is not guaranteed. Control of the generated outputs has also been engineered directly on U-net features [won et al. (2023), or on abstract latent codes |Preechakul et al.. (2022), both requiring post hoc interpretation.
Our work presents principled framework reminiscent of model inversion Ramesh et al. (2022); y Gal let al. (2023); Mokady et al. (2022); Kwon et al. (2023) or semantic compression Preechakul et al. (2022): Koh et al. (2020); Kodirov et al. (2017). However, while these works find representations of the semantic content of data point, they often not interpretable by design, and when they are Koh et al. 2020): Kodirov et al. (2017). they are not compressed (they have no measure of relevance) or principled. We propose a general method that greedily selects elements from an interpretable- by-design query-answed set that are most informative, and generates new data conditioned on these representations while allowing for targeted semantic modifications.
5 CONCLUSION
This work introduces Interpretable Compressed Descriptions for Image Generation (InCoDe), novel framework that leverages Information Pursuit (IP) to effectively represent data and guide image generation based user preferences. InCoDe generates images consistent with succinet and meaningful representation D, given by a sequence of user-defined questions and answers, which are chosen sequentially to maximize mutual information between D) and the complete Query-Answer set Q(X). In this way, InCoDe prioritizes the most relevant queries and provides an intuitive, interactive interface for generating customized images, while addressing challenges of current generative models by building upon them. Through validation across multiple scenarios, InCoDe demonstrates superior performance over existing methods, offering an efficient solution for fomain-spectie applications such as bedroom decoration. This work paves the way for more personalized and interpretable-by- design generative models, enhancing both user experience and practical utility.
The proposed framework requires having query set. This can be perceived as both a strength and a limitation. It is a strength, since the designer is free to select the set of semantic concepts that are important for the intended generation task. Yet, building this set can be tedious. Modern large language models can alleviate this task. We have used this approach to create the queries for the LSUN Bedroom and Churches datasets. However, human supervision and prompting is still necessary. This raises the research question of designing a framework that allows users to define the semantic domain of query set without the need for direct supervision of the query set itself.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS STATEMENT
The development of generation methods such as InCoDe offers significant benefits, such as enhanced personalization and domain-specific applications, but also presents risks. Without proper safeguards, generative models can be misused in harmful ways, such as spreading disinformation or creating offensive content. As they become more accessible, it important to balance their innovative potential with ethical considerations, ensuring that the technology is used responsibly and transparently.
In particular, one major issue is the potential for bias. If the query set used by InCoDe not diverse or representative, it may result in the reinforcement of stereotypes or exclusion of certain groups. Thus, adequate oversight ensuring that the query set is ethically designed is key to mitigating this risk and avoid harmful or misleading content.
7 ACKNOWLEDGMENTS
This work was supported in part by NSF grant CNS-2038493, ONR grant N00014-21-1-2431, ONR MURI contract 503405-78051, ARO MURI contract W91 INF- 17-1-0044, and the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (NSF grant 203 1985, Simons grant 114201).
REFERENCES
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018. pages
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022. pages 10
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. openai.com, 2023. URL
hittps:/een //t::///dd..eee openai wwwp//pppps///c/cppooo 1alalll-33. pdff pages
Moritz Bohle, Mario Fritz, and Bernt Schiele. Convolutional dynamic alignment networks for interpretable classifications. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10029-10038, 2021. pages 9
Aditya Chattopadhyay, Stewart Slocum, Benjamin Haeffele, Rene Vidal, and Donald Geman. In- terpretable by design: Learning predictors by composing interpretable queries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. pages 2, 10. 16, 23
Aditya Chattopadhyay, Kwan Ho Ryan Chan, Benjamin D. Haeffele, Donald Geman, and René Vidal. Variational information pursuit for interpretable predictions. In International Conference on Learning Representations, 2023. pages 3, 4, 5, 23, 32
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 -87944 2021. pages 10
Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10265-10275, 2022. pages
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Composi- tional generation with energy-based diffusion models and meme. In International conference on machine learning, pp. 8489-8510. PMLR, 2023. pages 10
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. Advances in Neural Information Processing Systems, 36: 36:16222-16239, 2023. pages 10
11

---

Published as a conference paper at ICLR 2025
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training free structured diffusion guidance for compositional eet-to-imaae symthesis. In The Eleventh International Conference Learn- ing Representations, 2023. httphhttsss//ppeeevieeeee net/ ott. pages 2, 7, 10, 26
Soham Gadgil, Ian Connick Covert, and Su-In Lec. Estimating conditional mutual information for dynamic feature selection. In The Twelfth International Conference Learning Representations, 2024. URL CttLhhtps:////
vm
pages 10
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and
Daniel Cohen-or. An image is worth onc word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
URL chttps://opente 2ttpeeneeeiiieeettttoooaeaaeeae (errrumiiidNNAQOOOTTTYY pages 10
net/
Donald Geman and Bruno Jedynak. An active testing model for tracking roads in satellite images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(1):1-14, 1996. pages
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt- to-prompt image editing with cross-attention control. In The Eleventh International Confer- ence Learning Representations, 2023. URL
CDixzkzeyb pages 10
Jonathan Ho and Tim Salimans, Classifier-free diffusion guidance. In NeurIPS 202/ Workshop Deep Generative Models and Downstream Applications, 2021. URL http:tts//ppeeeeeeeeee net Eorum?
met/Eooum?2 wwwpaumddddwwwAxxEE--1
pages 6, 10, 30
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. pages 4, 5, 23, 25, 30
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
hwwp///wwww...........ee.e forum?
id=nževKeeFYfF pages
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. /CML, 2023. pages 10
Ehsan Jahangiri, Erdem Yoruk, Rene Vidal, Laurent Younes, and Donald Geman. Information pursuit: A bayesian framework for sequential scene parsing, arXiv preprint arXiv:1701.023:: 2017. pages
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988- 1997, 2016. pages 6, 16, 17
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Conference on Computer Vision and Pattern Recognition 2023, 2023. pages 10
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2426-2435, June 2022. pages 10
Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. 2017 IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 4447-4456, 2017.
/ semant www.aatttsschhhooo cscho ar .ogg Corpus IuppssDS 2.331113333 pages 10
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning. volume 119, pp. 5-385333444 5348, 2020. pages 10
12

---

Published as a conference paper at ICLR 2025
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. The Eleventh International Conference on Learning Representations, 2023. URL
net/ http://wwww...t/ooeeeeeeeeeeee /ttpr/oruummdddddeeeeeeee
pages 10
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278222) -22244, 1998. pages 16, 17
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre- pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), 2022. pages 7, 16, 27
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. pages
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. CVPR, 2023. pages 10
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. pages 10
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6038-6047, 2022. pages 10
Meike Nauta, Ron van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14933-14943, 2021. pages 9
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2021. pages 10
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2023. URL https::/ openreview n
52 221522822xxx2
pages 2
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffu- sion autoencoders: Toward meaningful and decodable representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. pages 10
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, abs/2204.06 02155, 2022. pages 10
Samrudhdhi B. Rangrej, Chetan L. Srinidhi, and James J. Clark. Consistency driven sequen- tial transformers attention model for partially observable scenes. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2508-25111 2022. URL https pages 17
7/api. www.stmattttcchhceeecoe
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10695, 2022. pages 5, 5,6, 10
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 35:36479-3644 2022. pages 10, 26, 29
Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A framework for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10286-10295, 2022. pages
13

---

Published as a conference paper at ICLR 2025
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu S. Advani, Artemy Kolchinsky, Bren- dan D. Tracey, and David D. Cox. On the information bottleneck theory of deep learn ing. Journal of Statistical Mechanics: Theory and Experiment, 2019, 2018. URL https: //api. se..mmaannn nt ott.tiiaa.coo..ooooooooorr :::pr/// pages 16
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai. Kejie Li, and Xiao Yang. MVDream: Multi-view
diffusion for 3d generation. In The Twelfth International Conference on Learning Representations, 2024. URL
pages
Mike Wu, Sonali Parbhoo, Michael C Hughes, Volker Roth, and Finale Doshi-Velez. Optimizing for interpretability in deep neural networks with tree regularization. Journal of Artificial Intelligence Research, 72:1-37, 2021. pages
Binxin Yang. Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. CVPR, 2023. pages 10
Chih-Kuan Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-basee based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554-20565, 2020. pages
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop, 2016. pages 27
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023. pages 5, 10
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018. pages 16
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22490-22499, 2023. pages 10
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
In this appendix, we will provide additional information to complement the main paper. We structure it follows: After brief summary of the notation, Section A.I we provide and analyse additional results for all datasets. In particular, in Section A.I.I] we discuss results on different type of queries that are Docation-centered. An.2.1.2 we present results that compliment those in the main body the paper. Next, in Section A.2 we summarize background material on Information Pursuit and Diffusion Denoising Models. In Section A.33 we provide additional details for each of our experiments; Section A.4 is used to proof Proposition 2.1 of the location-based experiments described in Sec. A. of this appendix and a justification of the training objective. Finally, Section A.5 and ethical impact of our work.
we discuss the broader
Notation
We summarize the notation for the main concepts used along this appendix.
p Xo): Real data distribution
p Xo): Our model of the data distribution.
Xg: Data-point sampled from real data distribution.
t: Time step in our forward diffusion process. It represents the noise level. L.e. we add small amount of Gaussian noise to the sample T steps. This produces a set of noisy samples (X1,... .Y)
Xy: Noisy sample at noise level
p(Xe Xx-1): Forward diffusion process distribution.
p(X,-1 Xx, Xo, s): Reverse diffusion process true posterior distribution.
p 1 Xx,, s): Our model the reverse diffuuson process posterior distribution. á also used as the denoising mapping function. The subscript 0 is used to indicate the optimizable parameters of the model.
3g Variance schedule.
Arbitrary condition
{g. {(x)): Query and Queryset Answer respectively.
S: Query-Answer history random variable. With as a Query-Answer history realization.
Eg Gaussian noise term added to the clean sample Xo order to obtain X₁-
co(X,, t, c): Estimated noise at step Output of one forward pass in our diffusion model (with parameters 0). The network is conditioned to the current noisy image X,, the current noise level f, and potentially another condition c, that should steer our generative model towards p( Xolc).
15

---

Published as a conference paper at ICLR 2025
A.1
MORE RESULTS
In this section we analyse additional results from the remaining datasets, including those with location-based query sets.
A.1.1 LOCATION-BASED EXPERIMENTS
As noted in the main body, InCoDe is not restricted to textual queries. A query gather different types of information about a datapoint, such as location-based queries, where query Ki,j requests an RGB image patch around the location (i. in X. In this case, to ask enough queries to retrieve every image pixel. Since these questions carry spatial structural information, we can parameterize set S an image where only the patches corresponding to answered questions are visible and we use a Querier consisting of a set of convolutional blocks.
have QX) X by allowing
In the case of Q(X)= X, where the answers queries portions of the image itself, the denoising objective of DDPM can be used to optimize our Querier, avoiding the need for a QuerySet Answerer, jointly training the Querier and Generator in an end-to-end fashion.
Equation can be rewritten as:
min
Ex.Xo.DDLLb(XI-1 Xx, Xo) 11 pe 9.f
(A1)
where
q g=) EQ,
(X Xx, (((())
We formalize the above procedure as follows: Proposition A.l. Let (f",g") be an optimal solution to
A1 We define the optimization problem as:
max T(a(Xo); X1 (8) Xo. DKI.p((X- Xx. PEPx
11 (1x (x,g 8)]
(A2)
Then, there exists an optimal solution (p% g.) ) to the above objective for any realization S = s such that pSS s) > such that q.- = g" and p.= (X,, {g%, , q. (Xo)} U.).
The proof of proposition A.1 be found in A.4 and it draws inspiration from Chattopadhyay et al. 2002)). Note that for an optimal solution (p 1h) ). the KL divergence term would be 0, and the Querier would choose the query that maximizes the mutual information term between the first latent variable X₁ of the diffusion process and any given subset S = of Q(X). While the objective is to maximize I(g(X), ): s). this quantity is undefined for continuous X. A common strategy make mutual information well defined is to add a small Gaussian noise to X, Saxe et al. (2018). Hence, we seek to maximize (x))))); X1 s) instead. In section 2.2 we provide a practical methodology to solving the presented optimization problems, by means of neural-network architectures.
Datasets. We selected datasets with increasing complexity to assess a range of scenarios, starting from simple cases with predictable outcomes based human intuition and progressing towards more challenging scenarios. We include results for MNIST (LeCun et al.) 1998): database of images of hand-written digits; CLEVR Johnson et al. (2016): A synthetic dataset, where objects with different attributes are placed randomly in a uniform background. For CLEVR, we create an attribute set that answers the following query-set format: "Are there n or more objects of color Y ?". This is asked for eight colors (including any color) and n E [1, 5) for a total of 45 queries; CelebA: A dataset with celebrity face images and 40 facial attributes; and LSUN Bedroom Attributes, a large dataset with bedroom images and their descriptions
set of 58 binary attributes consisting on answers provided by BLIP Li al. (2022) to human-crafted queries. The latter a contribution of this work, given the lack of existing datasets that meet our task's 'ss requirements, and it will be released for public use. More details can be found in the Appendix. The types of queries used for each of these datasets are listed in Table 3
We assess performance using different metrics, depending on the type of queries utilized. For cecatiionbbsed queries, aiming impute missing values in image, we employ MSE and LPIPS Zhang et al. (2018) (measuring perceptual similarity through VGG network features), computed between the ground-truth and 200 generated samples, one per example, for description lengths
Metrics.
16

---

Published as a conference paper at ICLR 2025
Table 3: Descriptions of the image size and location based query-ser for different datasets.
Dataset (Image size)
Query Set Q
Size [Q]
MNIST [LeCun et al.
1998
(1 x 32 32),
pixel intensities in 3 stride
900
CelebA Face (3 64 x 64)
pixel intensities in 8 8 patches with stride 4
225
Clevr Johnson et al. 2016 (3 x 64 x 64).
pixel intensities in 8 x 8 patches with stride
225
of |D {0, 1, 2, 10, 20, 40, 60} (for CLEVR and CelebA) and [D] = {0, 2,5, 10, 20) (MNIST). The Generator used for evaluation was trained on random histories S for fairness across baselines.
For experiments using attribute-basee queries, we predict answers to the full set of queries Q(X) with a QuerySer Answerer, trained classify (X) from random histories S. Accuracy and Fi score are computed with a testset with 2k samples in all cases, comparing predicted answers with the ground truth.
Baselines For location-based query sets, where answers are not binary and impurity computation intractable, we employ two hand-crafted patterns that emulate human intuition, as introduced in Rangrej al. (2022: (iv) Spiral, where patches selected in spiral starting from the center of the image, and (v) Cross, with patches selected with a cross pattern. Baselines are described in higher detail in the Appendix.
Quantiative results: In Fig. For these ex- periments we report reconstruction metries in Figure [9] where smaller values indicate bet- ter performance. The horizontal axes indicate the number of asked queries. InCoDe clearly outperforms the baselines both MNIST and CLEVR datasets, where there is abundant re- dundancy to be exploited. For instance, both datasets have uniform backgrounds. As the main content located in the center of the image, spiral and cross baselines have relatively good performance the initial steps. However, explo- ration is needed disambiguate the remaining appearance and thus the random baseline gets better as more queries are asked. In the case of CelebA, we observe that the random baseline slightly surpasses InCoDe for fewer than 20 queries. We attribute this behavior to two main factors. First, this baseline selects queries with the same random strategy that was used to pre- train the Generator. Second, CelebA samples contain useful information in most of the image, and thus a highly exploratory method can be especially effective.
Clevr
0000
040
0012
0.08
21
CelchA
MNIST
4.75
Qualitative results: We can see in the side (a) of Fig. [0] sequence of patch queries se- lected by our Querier for sample of Clevr. The images this dataset consist of uniform back- ground with set of objects of different sizes located around the center. We can see how the Querier initially selects patch in the center of the image and then explores around it as more objects are glimpsed. It is interesting to see that for L = 5, the Querier has the objects. However, the generator creates 4th object, which is does not exist. This example illustrates the potential of the Generator to create diverse samples, while respecting the semantic content of description D, and the overall data distribution. In the right side
Quily
Ours
Spiral
Cross
Random
Figure 9: Performance evaluation of InCoDe using location-based queries. We measure the reconstruction and perceptual similarity (LPIPS). InCoDe out- performs the baselines both datasets.
17

---

Published as a conference paper at ICLR 2025
(b) of the figure, we the same illustration for CelebA example. Here, the Querier selects patches in the main features of the face, the clothes and the contour of the head, which defines the shape and provides information about both the subject and the background.
Generated Samples
X
Generated Samples
X
x
Selected Queries
Selected Queries
(b)
Figure 10: InCoDe selects meaningful patches according to the data distribution Illustrated in the left side (a) we see sequence patch queries selected by our Querier with their associated generated samples (same color border). The Querier initially selects a patch in the center of the image and then explores around it as more objects are glimpsed. the right side (b) of the figure, we show an example for CelchA. Here, the Querier selects patches in the main features of the face, the clothes and the contour of the head.
Fig illustrates InCoDe's performance when using location-based queries. The querier se- lects patches that contain most information about the digit. The identity of the digit is quickly captured after [D] = patches, and further steps refine the appearance match the reference image.
Querier Output
10
20
D
GT
X
T
q
a
a
a
Generated Images
A.1.2 ATTRIBUTE BASED EXPERIMENTS
Figure 11: InCoDe selects the most informative im- age patches towards reconstructing reference. Sam- ples generated using descriptions with Docationbbssed queries of length 2, 5, and 20, with MNIST dataset. The Querier selects highly informative patches according to the previous history, quickly predicts the digit's identity, and refines its appearance.
The following results illustrate qualitatively the behavior of our model, including corner cases.
LSUN Bedrom more generated samples: In Fig. [7. we display two examples of given set of attributes (query-answer pairs) and their corresponding image examples in the dataset. For visualization purposes we choose combinations of attributes that have exactly 4 corresponding examples in the dataset. Note that in most cases samples have a unique combination of attributes. We observe how the generated samples by our dataset are semantically similar, while appearance at the pixel level varies widely.
Fig. 12 showcases how generated samples of L.SUN Bedroom dataset using attribute-hased queries align to the reference the number of selected queries increases. After 30 queries, the generated sample resembles image the reference just as much as when asking all queries in the query set.
LSUN Bedrom more editing results: We provide additional editing results in Fig. 16 We can see how modifying one attribute does not change the overall structure of the generated sample, while it does add subtract that particular attribute from the image.
13 we present a trajectory generated by our Querier, along with the corresponding InCoDe outputs description lengths of 5, 10, and 15. The results demonstrate alignment with the majority of answers while producing images of high quality.
Trajectories: In Fig.
Contradictory Answers: in Fig. 15 we illustrate one particular case. How does the model
interpret the condition in case two queries have contradictory answers. This should rarely happen when answers represent a real datapoint. However, user could input contradictory answers, or the VQA model could have defective output. If contradiction occurs, or the other attribute is disregarded. This is made obvious in the figure, for queries "Is a window present the image? No" and "Are plants visible from the window? Yes".
18

---

Published as a conference paper at ICLR 2025
Reference
10
CIUAT Loodon
20
30
All (58)
Figure 12: InCoDe generates samples according semantic deseription given at different levels of detail. This figure illustrates how descriptions chosen by our model generate samples that increasingly mirror reference (top-left).
D
D|=10
D5=15
Querier output
Is the building in rand sma" Ane there chouts in the sky? -
Daos any peweon sppcer the image" - If here an tres the image, they large trres? Does the benddng have bell on the roct? yes Is the building adomned with sculptures? yas the building petnned?
the roof of building tiled?
there path leading to the huilding? yos
cxplolt viebli
imsge?
the wall of the building made visible brickal
Doss than one huilding image?
thene bench beclding
buildig sumounded fence!
Figure 13: InCoDe Example of trajectory created by our Querier. The corresponding images generated
with different description lengths respect most attrihutes.
Underrepresented attributes: Fig. 14 illustrates the behavior of InCoDe when individually asked about queries with an underrepresented answer in the dataset. We observe that our approach can still capture them. However, this experiment also brings to light the limitations of our pipeline. First, we observe that the presence a Hunting trophy rare and disregarded by our model. If we look deeper, we conclude that the BLIP model used to answer the queries hallucinates this particular attribute, and thus the model cannot properly capture it. We also observes that the image quality suffers when only or two queries are visible to the generative model.
CelebA with Facial attributes: We illustrate in Fig. 18 the performance of InCoDe for the CelebA dataset. Similarly to the previous example, display the list of queries selected by our trained Querier. In this case, it is harder to judge the correctness of the sequence intuitively. However, the first three questions likely have very high entropy the dataset. Moreover, we see that the generated images using our strategy resemble the reference at faster pace than when using random selection strategy.
19

---

Published as a conference paper at ICLR 2025
(a)
Curisins -r y43 Red curtains - you (3.2%)
(b) Collage - yob (2.5%)
(c) Hunting troply. 'yes (2.75%)
(d) Training bunting tophy BLIP may hollucinate the attes
Figure 14: Attributes in the dataset may be respected some cases they are disregarded. We illustrate this behavior with the following examples: (a) Generated images with active attributes indicating presence of "red courtains are present the room", (b) "Image collage" and (c) "hunting trophy is present the room". All of them with rare presence in the training set. We observe that in some cases the attribute may be respected, with the presence of failure cases. For instance, in (c) Hunting trophys
when unfrequent in the dataset, however in
not seen in the scene. We argue that this also be effect of (d), where the BLIP model used answer the queries may sometimes hallucinate aftributes and give wrong responses, as seen by retrieving examples for "hunting trophy". We also observe that the quality of the generated images suffers slightly when the Generator is conditioned on one or two attributes only, with the rest masked out. Human figures also have bad quality but this is common problem with the version of Difl--Dif that we are using base model.
Can you see window - no Can you see plants from the window yes
Figure 15: Contradictory attributes lead to valid images, but either one attribute or the other are disre-
garded. In this particular case, we how indicate that window should be visible, while plants should appear through window. The result often shows plants an environment without windows, conjunction of both attributes is not exactly present, as it is impossible. Note that none of the training examples (assuming VQA method being flawless) should not have such combination present.
obviously the
20

---

Published as a conference paper at ICLR 2025
ROE IEE
Original set of attributes
Light in nightstand - No
Plants outside - Yes
Figure 16: Examples of targeted modifications. (left) show two samples generated from same of query-answers, (middle) We edit the answer of "Is there nightstand light in the table?" "No". (right) We edit the answer of "Can you see plants from the window?" to "Yes".
Dataset samples with shared attribute list
Generated sumples with the attribute list
Dataset samples with shared attribute list
Generated samples with the attribute list
Figure Our method generates high-quality samples, substantially different from the training examples. We show examples in our training and testing set particular set attribute responses. We then use these attributes to generate samples and we conclude that they are semantically similar while varying widely in appearance. We include one failure case (right figure, bottom-left).
21

---

Published as a conference paper at ICLR 2025
Samples
Random Striteyy
Learned Querier
Selected
Queries:
1. Male yes
2. Smillng yes
3. Young yes
4. No. Beard -x
5. Black Har
6. Mouth_Slghtt Siihhyy_ppppp
7. Weaning Hat -2i no
8. Weoning Nec ->
0. Bueity Eyebrowi--o
Brown_ -ox no
GT
X
Attractive -or yes
Eyegianees -cx no
Blurry -co ng
Very
Range
High Cheekbonie yos
Hanvy Makaap
Arched, Byebrows
Straight
Narrow Eyes
Bags. Under Eyes yes
Rosy
Chubby
Wearing Wewrth__aaingg
Eanings
Clocklock Clockk_hhhoowwwrrr.cocc
Reseding, Hailine
Blond, Hair
30
Oval. Face
Gostee
30. Pointy Nose -o
33. Sidabums -2
34. Paee. Skin no
35. Wearing, Lipstick
38. Weering Necklace - n0
All
37. Doublle Chin -a nd
38. Muatsche -as no
30, Gray, Hail ->-
40, Bald -2-no
Figure 18: InCoDe selects meaningful queries to reconstruct an image in terms of the query set. On the right, we see a list of queries selected by our trained Querier. The first three questions likely have very high entropy in the dataset. The generated images (center) using strategy converge faster towards the reference appearance (left) than the random selection strategy.
22

---

Published as a conference paper at ICLR 2025
A.2 BACKGROUND
A.2.1 INFORMATION PURSUIT AND ITS VARIATIONAL CHARACTERIZATION
Information Pursuit (IP) was proposed as framework for explainable prediction by choosing explainable queries Chat et al. (0022). Its main idea is to make a prediction of a target variable Y from input data X E X, by sequentially asking and answering queries about X.
In practice, only a small number of question-answer pairs might be enough to make a prediction. Thus, IP aims to construct the shortest sequence of questions that is most informative to predict Y, by selecting the next query to maximize the mutual information with X, building on the information gained from previous queries. Given set of possible queries Q, and an observation X = the IP algorithm is as follows:
Q1
(P(0) arg max I (x(x);
QEQ
gh+1 IP( )
(A3)
arg max [(q(X); Y (xxh)))
QEQ
where ((r), with slight abuse of notation, denotes that the answer to query for the data X x is g(x), and qk-1 E Q refers the new query selected by IP step 1, based on the query history (denoted as 91:* (2=h))))), )) and I denotes mutual information.
IP can be carried out by learning the distribution p p(Q((),, from the data by using generative models and MCMC sampling to estimate the mutual information terms Chattopadhyay et al. (2022). However, MCMC sampling is computationally expensive. To overcome this challenge, Chantooaaaay et al. (2023) proposed variational characterization of IP, noting that generative models are only means to an end. Thus, they introduced a function called querier, mapping the observed histories, (1:a(1:00) ), to the most informative next query qk++1 E Q. They showed that this most informative query is exactly the query q* whose answer will minimize the KL divergence between the conditional label distribution p(Y YY| () and the posterior p(Y q² (z*)))))
Based on this insight, an optimization problem is designed to perform IP as follows. Let Q(x) = fg(=) q Q} be the
set containing all inquiries about the data and their corresponding answers. Let K(x) = P()(z)), be the power se of Q(x) with all possible query histories. Define K = Uzr)) let classifier ʃ K Py be a function mapping arbitrary query-answer sequences to a distribution over y and the querier y K + Q be a function mapping arbitrary query-answer sequences to query q E Q. Then, the variational objective for IP is given by the optimization problem:
min 0.
Ex.s [DRL. (p(Y) X) ll P6 [x(X)
subject to
q = (0) EQ ((x(X)
(A4)
P(Y
where DxL. indicates the Kullback- divergence between two probability functions and S' is a random set of query-answer pairs taking values in KK, and conditioning on (X) should be read as conditioning on {xr E X' g(x) g(X).. Given S = s and X b the querier (-(-) chooses a query E Q, evaluates it on ob and passes (b=a to the classifier. Then, the classifier makes a prediction based on appended with this additional query-answer (== This implemented by parameterizing the querier and classifier by neural networks, with parameters 0 and Φ. respectively, and a random set of query-answa pairs S. For futher details refer to Chattopadhyay al. (2023).
A.2.2 DIFFUSION DENOISING MODELS
Here, we provide a detailed overview of the formulation of Gaussian diffusion models from Ho et al. (2020).
There are two main stages in diffusion-based generative modeling. The first stage is the forward diffusion process. It gradually adds small amount of Gaussian noise to clean sample from the
'A power set of set is the set of all subsets of Q, including the empty and Q itself.
23

---

Published as a conference paper at ICLR 2025
source distribution Xu pXX) to create sequence of noisy samples Xo. ,,,,,,,,,..... XI....Xp. The amount of added Gaussian noise is defined by a variance schedule noted by Be:
p p(X+|x,-)) = N 3
(A5)
As seen the transition probability is parametrized as a Gaussian distribution. The Gaussian Markov proces has good properties. We can easily obtain noisy sample arbitrary step of the diffusion process given a clean sample Xo. Instead of repeatedly applying p until the desired t, we simply need accumulate the noise scales given our schedule:
X(X = (Xx: (1 T.)
(A6)
=ā 1-1-āu N(O.I)
(A7)
with 0r = 1 3r and
II 0x2
(A8)
with - ag indicating the noise variance at an arbitrary step f.
This is followed by the reverse diffusion process, which starts with sample from isotropic Gaussian noise, X7, and incrementally removes noise towards generating true sample from the source distribution, thus reversing the forward process. We are thus interested in approximating the true posterior the reverse diffusion process. By means of the Bayes theorem, we find that the true posterior conditioned on Xo is also Gaussian with mean (X,, Xo) and variance Be:
(X,, Xo) = V
Varl - ār-1) Xt
Xo
(A9)
Utt
Ot
B. = 1- Ōz-1 Br
(A10)
1 Ctt
PXX,-IXX,,X0 Xo) (X is (x(x,, Xo), 3.1)
(A11)
However, we wish to approximate the posterior p(X;-1 Xx), without access to the sample Xg. This term cannot be computed exactly but it is assumed to have a Gaussian form when the step-size 3r is sufficiently small. We approximate the posterior with a model po(X, (0xx-1XX (x,) parametrized by a neural network with parameters 0. In particular, the model estimates the mean μs and diagonal covariance matrix Eg Gaussian distribution:
= N Eo(,,
(A12)
In order to learn the true distribution p(Xo), we can optimize the following variational lower-bound Lyb for po (Xo), instead of directly approximating p(Xo):
- llgg po (Xo)= < -log (Xo) + DKL (X)) (Xo_I((XX (oo)
(A13)
log p(X1.T (Xo)
log po + -x (Xo)
(A14)
Do ((xT)//Pi /a (Xo)
log (Xo) Ep log x(
log po log (Xo)
(A15)
po(Xo:))
I(X1.rl
Ep [log
(A16)
po (o(Xo.)
log p(XirT|Xo)
Let LYLE Ex(Xo:))
> -Ep(X)) log do (Xo)
(A17)
Do (Xorr)
24

---

Published as a conference paper at ICLR 2025
Lvbb can be decomposed into three main terms:
log p( p(XITT12 ( Xo)
LVLB = Bp(Xor)
(A18)
Po( (Xo:))
II 1=1 p( (
=Ep log
(A19)
Po (X()) II Po (Xr (x,)
log p( ((Xx(XX-11)
Ep
log Do
(A20)
po (x-1 (X+)
p(Xx (X,11 (x,) + log p(X)
Ep
log Po (XT)
log
(A21)
Pa (
T
Po(Xx-1 p(X(-1lXo) + log p(X,
1 p(X,(Xoo)
Bp
- log 2ox
log
Po(
1=2
(A22)
T
pXXr1 Xx,
T
((,((x)) + log
p(x, (Xo)
E,
-
log (XT)
log
po ( (Xt)
log
p(X, p(X,-1(Xo)
Do (Xo (X1)
2=
t=2
(A23)
T
p(Xr(Xo) ((X)(Xo)
+ log p
Bp
log Dog
+
log
+ log
Po (x
(X X1)
1=2
(A24)
log p(Xr)
p(X, - |x Xo)
Ep
Σ
log po (x
log 6o (XolXX1)
(A25)
po (XT)
2=2
Ex
DKL. (x()) || (XT)
Lr
T
Σ
DKL(p(X,-1 X2, Xo) || Po (X,
t=2
Le-1
(Xolx((
(A26)
From this point on, it suffices with optimizing the intermediate term L,_1 for all values of f. Note that we have an analytical expression for p(X,-1 X,, Xo) which has Gaussian form. Given that our model (x X2)) also parametrizes Gaussian, we can exploit the closed-form solution for the KL divergence between Gaussian distributions, and leverage simple change of variables (described in Ho et al. (2020)) order to optimize a simpler expression:
Laimlee ==
|lle co (x,,
(A27)
Here e: is the added noise time I, and train model en (Xx,,
K,,() () predict e, from Equation A7
Exximple does not provide any learning signal for Eo( (X,, t). This happens because instead of learning Eg (X,, f), it is fixed constant [3,I as proposed by Ho al. (020).
H4 (X,, f) can be derived from cp (X,, t) as follows:
1 0z
po(X) (o,X,,1 t)
v1 - āi co (Xt,t))
(A28)
Vaz
This sustitution is used both to derive Equation A27 and to sample. The sampling algorithm can be found in Ho et al. 2020). not the only possible sampler that has been proposed, but we use it for method.
25

---

Published as a conference paper at ICLR 2025
A.3 EXPERIMENTAL DETAILS
In this section we describe the details for different aspects of the experiments: the overall set- ting and architecture, the datasets, details for LAION-400M, the sampling procedure and training regularizations.
Architecture and Setting Details. We design different architectures for different query-set types. Tables [5] depict the designs for the Querier, and Table 66 for the QuerySet Answerer. For different datasets, The blocks may have different depths. For the patch-based querier, the number of channels is the sum of channels for the ground-truth reference image (if any); channels of the history and an extra channel that is a binary mask indicating what regions of the image are visible C = Cgt + Cs + CM- The input thus the naive concatenation of these three sources. For attribute-based query-sets, the input data is formatted differently. We have a two-channel one-dimensional vector that is the concatenation of an indicator of the answers to the visible attributes with values (-1, 0, 1} and binary mask indicating visibility.
We use Imagen Saharia et al. (2022) as the main generative pipeline. We slightly modify the conditioning methodology to adapt to our requirements. We describe the main modifications:
Remove LayerNorm from fully connected layers that process attribute-like conditions.
Attribute query-sets condition the U-Net through masked cross-attention in all layers, an- nulling the effect of queries that have yet to be asked. Diffusion time ć interacts with the U-Net solely through feature-wiss modulation.
Image-like conditions are concatenated to the noisy input image of the U-Net.
Next, we describe the main hyperparameters used for Imagen's U-Net. Learning rate: LR le with a cosine decay; Base dimension: 32; Dimensionality multiplyers: (1, 4, 8), Self-attention resolutions: (x 4+ ) Query embedding size: 16 2: Condition size: 256; Number of steps
for training and sampling: 256; Condition drop probability: p 0.1. The reference code-base
and other details can be found in /tttps://gi thtbb com wwtp:/ugiiiee dra ins www.daaiinnnmmmeepttttoo More details can be found in our codebase.
Table Architecture the querier used the patch-based experiments.
Table 5: Architecture for the querier used for the attribute- based experiments.
Table 6: Architecture for classi- fier used for the attribute-based experiments.
ConvBlock (C - 512)
LinearBlock (IQI 2000)
LinewBllck (QQI 2000)
DeConvBlock (512 - 32)
LinearBlock (2000 + 500)
LinearBlock (2000- + 500)
Channcl-conaal
Dense (32 + 1): Attention logits
Dense (1000 - IQ): Attention logits
Channel-concat
Hard Softmax: Selected Query
Hard Softmax: Selected Query
Dense (1000 IQU: Attribute logits
Our method for LSUN Bedroom experiments has been trained as a wraper to Stable Diffusion V1-4: We the same version for the results displayed in
TugginngfaaccooCCommssssittttetd Fig. 2.
When showing results for Stable Diffusion XL, we use the model in buegginhiic.costbbillyyassbbll diflusion-xI-base-100
Quantitative experiments for the generative approach compositionality capability. We choose as baselines the Stable Diffusion V1-4 baseline text-to- text-to-image model and Structured Diffusion Feng et al. (2023), method to improve multi-concept conditioning in text-to-image models, specifically more accurate attribute binding and better image compositions. Both methods are conditioned by text, and thus provide the attributes in the following format:
"Photo of a bedroom with curtains that are light-colored, with white walls, with floor made of wood, with visible door, without ensuite bathroom, with white bedsheets, and without watermark."
Note that we select 10 attributes from the query set of length 58. They correspond to the attributes with top entropy in the training dataset.
26

---

Published as a conference paper at ICLR 2025
The reason why we do not provide the full list of attributes for the LSUN Bedroom dataset is that the baselines only accept prompts with a maximum length of 77 tokens, and thus we cannot concatenate all attributes in natural language.
Datasets details.
Here, we describe the dataset details.
MNIST: Training corpus consists 60k 1 32 x 32 greyscale images of handwritten single digits. In inference time generate samples for each example of the test set, consisting of 120 images. We generate samples for multiple description lengths, order to generate the The test setting is the same for all experiments, except indicated otherwise.
curves in Fig. g
CelebA: It consists of 50k 3 x 64 x 64 images of celebrity faces, divided into 34-1k- 15k for training, validation and testing. The publicly released dataset can be found in https://www.tenssorlowww wwt.:o///ddttssss//taaao////////
Clevr: It consists of 8k (partitioned as 7k- kk-kk for training), validation and test. 3 x 128 128 images generated by randomly placing objects flat background with different lightings. The objects have different controllable discrete attributes such color, shape and material, as well as continuous attributes such as rotation or size. We choose color for our attribute-based experiments.
LSUN Bedroom and Churches with Queries: Given the lack of existing datasets that met our task's requirements adequate number of samples, high image quality, binary attribute descriptions and images belonging we created our own dataset. To do so, we selected a set of descriptive binary queries, some of which redundant with others, to categorize certain image distribution. Bedroom dataset consists of 316k images, filtered to 60k and resized to x 512 x 512 belonging LSUN (Large-scale Scene Understanding) dataset Yu et al. (2016) under the category 'bedroom'. Churches dataset consists of 70k images, filtered to 11k and split as 90% 10%, for training and validation, reserving 2k images for test. They belong to category "churches' of LSUN and are also resized to 3 x 512 512. Link to datasets provided in https://gttub..omm/ http://tw..oooommmmmmmooo/mnooaa
concrete and well-defined distribution,
Filtering includes discarding images with width/length ratios exceeding 1.3 or falling below 0.7, with additional rule-based filtering. The images were then passed through BLIP Li et al. (2022), visual question-answering engine with a set of queries designed describe the scene with binary answers. The official LSUN dataset can be found in:
https://www.eensorlteh www.oo//daaaasss//caaaooooooooo
The list of binary attributes were the following (asterisk marks some level of redundancy):
Bedroom:
Presence of curtains
Presence of red curtains*
Presence of brown curtains*
Presence of blue curtains*
Presence of light colored curtains
Presence of a luggage
Presence of at least one person
Presence of an adult*
Presence of a child*
Presence of two children*
Presence of a blue wall
Presence of red wall
Presence of white wall
Presence of a dark wall
Presence of a window
Presence of sunlight coming out from *
Presence of plants visible from window*
Presence of city buildings visible from window*
27

---

Published as a conference paper at ICLR 2025
Visible floor
Presence of carpet*
Wooden floor visible*
Presence of a door
Presence of an open door*
Presence of door to en-suite bathroom*
Visible ceiling
Presence of hanging lights from the visible ceiling*
Presence of more than one bed
Whether all beds are from the same size*
Presence of bunk beds
Whether the bed is big enough for two people
Presence of mosquito
Presence of a TVV monitor
Presence of a radio
Presence of a radiator
Presence of a bedside table
Presence of nightstand light in the bedside table*
Presence of more than one bedside tables*
Presence of photo frames in the bedside table*
Presence of white bedsheets
Presence of dark bedsheets
Presence of blue bedsheets
Presence of red bedsheets
Presence of green bedsheets
Whether the image collage
Presence of an animal
Presence of a dog*
Presence of cat
Presence of a bird*
Presence of a hunting trophy wall*
Presence of a telephone
Presence of plants inside the room
Whether the image is from hotel room
Presence of paintings in wall
Whether the lights are on
Whether the bed made
Presence of a closet
Presence of a wooden closet*
Presence of an image watermark
Churches:
Building made of stone
If building is made of stone, is it light-colored stone**
Presence ofgreen grass in the image
Presence of trees in the image
If trees the image, are they large trees*
Presence of person in the image
If person appears in the image, are they standing near the building
Presence of a chimney on the roof of the building
Presence of windows in the building
28

---

Published as a conference paper at ICLR 2025
If building has windows, are they large windows*
Wall of the building made of visible bricks
If wall of the building is made of visible bricks, are the bricks red*
Sky clear
- sky is clear, is the sun visible in the image*
Clouds in the sky
Presence of more than one building in the image
Is asphalt visible in the image
Does the image have watermark
Does the building have a bell on the roof
Presence of garden in the building
Building rural area
Presence of a mountain in the image
Nighttime
Building in the image from the Romanesque architectural style Building in the image from the Gothic architectural style
Building in the image from the Renaissance architectural style
Building in the image from the Baroque architectural style
Building located near body of water
Presence of a path leading to the building
Presence of bell tower in the building
Presence of a bench near the building
Building surrounded by fence Building made of wood
Presence of columns in the building
Presence of large
window in the building
Does the building have cross on the roof
Presence of a bell tower separate from the main building
Building adorned with sculptures
building is adorned with sculptures, are they religious figures*
Presence of arches vaulted ceilings inside the building
Roof of the building tiled
If the roof is tiled, are the tiles red*
Presence of a statue of a saint or angel on the building
Building painted
Image Sampling. Image sampling is done with the same algorithm as described in Saharia et al. (2022), the main difference being that the diffusion model conditioned on the selected description D.
Hardware InCoDe has been trained in two NVIDIA GeForce RTX 2080 Ti GPUs. For images of resolution 64 x 64, takes ~ 1 day to train. There are slight fluctuations due to model variations. The binary attribute image classifier has been trained in two NVIDIA RTX A6000 GPUs during days.
A.4 CONDITIONAL DIFFUSION FROM THE PERSPECTIVE INFORMATION THEORY: PROOF OF PROPOSITION 2.1
Initial remarks. We wish to approximate p(X)) with our generative model p(X,). Next, we provide derivation of the diffusion objective by means of the variational lower bound. Steps
29

---

Published as a conference paper at ICLR 2025
skipped for brevity. An expansion can be found in Ho et al. (2020) and Section A.2.2 of this appendix.
(o0)))))))))) 2 Bx(X))
-- ((Xo) + DRL(p(XLTT) Xo) (1 Xo))]
p(X1.r Xo)
-Bp(Xor)
log p(Xo:r)
= =Ep(Xor)
(KL(p((T)) Xo) || ((Xr))
(A29)
T
PRI(p(Xx-11 Xo, Xx) (Xx Xx))
2=
log ((x X1)
Effectively optimize the following objective, which is simplified.
min E Bexxooeeel..//PuxxXXXXX Xo. X1) 11 ((xx x,)) +C
pEPx min E (x.,,, pEPx
(A30)
[ller co +)] Liimplo
where a constant not depending on the parameters 9 of P and eg (Xx,)) as our noise estimation
model. Instead of summation take the expectation for E [1. Minimizing Simple is
empirically shown to also minimize the KI. divergence term in Equation A29 (Ho et al. (220
Here, interested in generating X conditioned set of conditions (c,--.., CN Ho &
Salimans 2021) makes the following connection:
min (x (No c)]
pEPx
< min Ep (mxxoeeeee Xo. Xe,c) p(x.-1 X,,e)] pEPx
(A31)
min Ep(Xole)..tt.p..)) [lle: - (Xer,,,,)))
pEPx
Ho ct al. 2020) only considers the unconditional distribution. The above is true we consider
p(Xo c) being separate distribution, and model (XX, c) equivalent having family of unconditional models with different for each condition c. Therefore: p(Xo c) = Pe(Xo) and
(Xo c) (Xo).
Next, we prove the following lemma:
Lemma A.2. Let Q be user-defined query set and IP(),) all possible distributions on Xt with t being an arbitrary noise level of the diffusion process. Then, for any realization S the following holds true:
min Ex(Xo pEPx.qEQ
DkL(p(X.-11 Xo. P(X,-1 (e. s)]
max AEPx.0EQ
X(
(A32)
T Ep(Xo|Xe.)) Σ DKLlP((X111 Xx, (()), p(x,11 (Xo,
(A33)
30

---

Published as a conference paper at ICLR 2025
Proof of Lemma A.2 mutual information.
With c (q(Xu), s), express objective in Equation (1l in terms of
min PEPx.
Bp(XolX...)
DKL Xo. g(Xo), p(X.11 Xx. (XXo), s)]
p(Xi-1 Xo, ((Xo), (()o)))(()))))))
min pEPx.
Bp(XolX..s)
Σ p(Xe-1 Xo, (Xx log
p(X-11 Xr,9(Xin
t=
X1-1
T
(X X,, X,, (Xo), 8)
min PEPx.9EQ
Bp(XolX..s)
Σ
Σ p( (X Xo, X2, (Xo) log
p(X(-1
Xx. ((x) s)p( Xo Xe, (()),s)
2=
Xi-1
T
p(X,, x, X, ((x),))
Ep Xi PEPx.
min
log
p p(X
Xx, a( ((Xo) ()
Xe, (()))
1=2
T
p(X2-1
Xe, (XXo,,))
+ Xp(Xx X, (2(8)
log
p(X,-1
Xxs ((o),
1=
min QEQ
[[ X1-1
Xe, ((Xo).))
+ E Ex(Xx
1 (x. |1 (x, (X,)(X)),
(A34)
with = Xo. Now observe that for any fixed = s and any 9 E Q,
(Xo) X,, I(Xu; X,-1 X), - (g(( X1,
(A35)
= I(xo; X.-1 8t,
Decomposing 1 ((Xo) S),
(Xo); Xt, = [(g(Xo); X X,,8) 1(X; (x,)(X)),
(A36)
We find that:
min (X N2, (Xo), qEQ
min qEQ
- X,-1 Xt, 8).
(A37)
Then, we apply the equality in A37 to the first term of the last expression in Equation in
A34
T Σ 2
T
X), = Σ [l((Xo); [g((Xo);
XT},ss)
(A38)
t=
= I([X -I( s) = =(Xx:q(Xo))
X71:9(Xo)
We casily see that Le2ma proven by applying the resulting expression in the mutual information
term Equation A34
Proof of Proposition 2.1 Once again, restate the proposed objective in Equation Al the parameters subscript for simplicity):
(ommitting
min 0
PKI (p(X--1 | X,, Xo) [1 Xe.g(Xo),
(A39)
where q 9= g5) Q
31

---

Published as a conference paper at ICLR 2025
We then proceed as follows:
min E (Xoll)))
DEL (p(Xe-1 Xo) 11 (X (xo)(Xx))
(A40)
0.0
T
-Bo(X))e)
[KK.. (p(Xe-1
Xx, (x, (xo))
T
-Ep(Xole)
DRL. (p(Xe-1 Xo) p(x=11 Xx, ((X),,
t=2
p X1, .))
Σ
Σ p(X-11 Xo) log P/ (X+- t-1
+ Bp(X)ls)
X 95 (Xo),
t=
-
r
[Ep(X))>)
DKL (p(Xe-1 Xo) || M(X,- 1 X2, ((x),
Σ p(X Xo) log 1x X1, (Xo),
- ()()))
p x-1 Xt,
2=
-
T
=Ep(Xols)
DKL (p(X(-1 Xo) || p(X,-1
Xx, ((Xo),
1=2
T
- Xols)
DKL Xx. 8) (1x (Xo)
1=2
T
<Pp(X))))
DRL (p(X X1, Xo) || p(X.- ((xo,
(A41)
for any realization S with p(S s) 0. The optimal denoiser 3 and query of the solution to the minimzation problem. We denote any denoiser as p., and q - g(s) () is the output of any querier. We make of Lemma A.2 in the fourth equality. We appeal to the KI. divergence non-negativity for the inequality. This inequality holds for VS = s. We conclude that gf = g (s) and 02 = (X,, {9- of [xo)] Us) for any given s. Note than in slight abuse of notation denote both the denoising mapping and the posterior probability as p. The Theorem A2 is proved by using Lemma A.2 to characterize q and p..
Note that parts of this proof are structurally equivalent to that of Chattoopadyyy et al. (223)). However, we apply to our particular case of image generation by denoising.
Training Objective Function. Finally, applying the empirical equivalency in A30 we substitute the KL divergence term in our objective function, obtaining the following expression:
min 0,0
E... [lle+ =x(,,, SU (X
(A42)
where == (S) EQ
with as the noise estimation model with parameters and go as the querier, with trainable parameters ø.
A.5 IMPACT
This paper presents work whose goal is to advance the field of Machine Learning and provide useful tools for artists to create and edit image content. The ethics of creating realistic images involve a complex interplay of factors such intent, consent, impact, and cultural sensitivity. While realistic images can serve legitimate purposes, responsible creation and use, transparency. and respect for individuals' rights essential to navigate the ethical considerations involved.
32


Reviews:
[{"review_id": "78QQRBfXJX", "reviewer": "Reviewer_DhcA", "summary": "The authors propose a new method for selecting human-interpretable semantic descriptions that allow for precise and reliable control of diffusion image generative models. The authors find these descriptions via the Information Pursuit optimization with neural networks, collect datasets to support and evaluate their approach, and quantitatively and qualitatively validate the effectiveness of InCoDe (the proposed method) on four different datasets. Specifically, these datasets consist of image and query-answer pairs for each image that describe its semantic content. Out of the four datasets, two are new and created by the authors, where existing LSUN Bedroom and LSUN Church images are given relevant query-answer pairs to describe the images.", "questions": "*minor questions and concerns from the reviewer*\nSome figures, e.g., Figures 8 and 9 appear to be raster rather than vector graphics? They look a bit blurry when I zoom in.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I find the paper generally well-written and polished. As a result, it was easy to read and understand. For instance, the mathematical equations are clear and notions used in this paper are explained in the appendix. The diagrams in Fig. 3 and Fig. 4 clearly show the how InCoDE is trained. \n\nMaking image generation more controllable via succinct, human-interpretable descriptions is of great value to the image generation creative community. \n\nInCoDe is compared against solid baselines (Figure 8, Figure 9), where it achieves significant improvements in image quality. \n\nThe authors promise the newly created datasets will \"be released for public use\", which is a big plus given the lack of similar datasets at the moment. \n\nThe authors do a good job explaining the model architecture, datasets, and the hardware setup in A.3 EXPERIMENTAL DETAILS, which I believe makes the results reproducible.", "weaknesses": "The related works section is somewhat oversimplified -- I think the authors could do a better job offering more context on the current state of interpretability and controllability in image diffusion models.\n\nDespite the effectiveness the authors have discovered with using InCoDe on the four dataset which they tested, it is not clear whether InCoDe can generalize to more \"free-form\" types of controllability. All four datasets include a fixed set of 40-58 queries that are hand-selected and deemed relevant, but in the real world, users use image diffusion models to generate highly diverse images. It would seem that they are out of luck if these images do not comply with one of the pre-defined set of queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nVrydYvOsd", "reviewer": "Reviewer_K8Cj", "summary": "The paper introduces a framework InCoDe designed to improve the interpretability and control of image generation models. InCoDe provides a user-friendly, interpretable interface for custom image generation, enhancing both semantic control and user experience. The authors use the compressed, interpretable descriptions based on query-answer pairs to guide image generation, addressing limitations in current text-to-image models that struggle with composing multiple concepts accurately. And they also collected two new datasets along with sets of binary queries and answers about their content.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **A Novel User Friendly Interface:** InCoDe’s query-answer framework allows users to control image generation through a sequence of intuitive questions, making it more approachable and interactive than text-based inputs. This structure lets users directly specify attributes without needing technical prompts.\n2. **Enhanced Interpretability and Control:** By using interpretable, information-rich query chains, InCoDe enables users to understand and shape each step of the image generation process, offering a clear view of how specific attributes impact the final image. This makes image manipulation more transparent and precise.\n3. **Comprehensive Experimental Validation:** The framework’s effectiveness is demonstrated across diverse datasets and scenarios, with strong performance in generating accurate, attribute-aligned images.", "weaknesses": "1. **Limited Flexibility Compared to Text Input:** While the query-answer format improves control, it may restrict expressiveness compared to free-text descriptions. Users can only manipulate images within the bounds of available queries, which might not capture all desired nuances or novel concepts.\n\n2. **Potential Challenges in Real-World Scenarios:** InCoDe has been tested on curated datasets, but real-world image generation tasks may introduce more variability and ambiguity. Handling this with a fixed query-answer system may be challenging and could limit the framework’s effectiveness in broader, unstructured contexts.\n \n3. **Reliance on Visual Question Answering (VQA) Accuracy:** The framework depends on accurate VQA responses to generate coherent descriptions and images. Errors in VQA performance could affect the quality and alignment of generated images, especially in complex or nuanced queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sinvGiWktm", "reviewer": "Reviewer_vMKV", "summary": "The paper proposes a framework called InCoDe, designed to improve the control and interpretability of image synthesis models. InCoDe leverages information-theoretic principles and compressed, interpretable descriptions, enabling users to generate images through a sequence of query-answer pairs. The model uses an Information Pursuit technique to select the most informative queries for defining images and then generates images using a diffusion model based on these descriptions. InCoDe’s primary modules include a Querier, Decoder, and Generator, which work together to iteratively refine image generation given user's attributes.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and the authors have done a good hob in providing the motivation for the paper. In addition, the given examples throughout the paper help providing intuition and a more clear understanding.\n\n+ Information-Theoretic Foundation: The method relies on information-theoretic principles to determine query relevance and hence, prioritizes the most informative attributes.\n\n+ Conditioning pre-existing diffusion models to harness their prior while \"smoothing\" the training procedure is interesting.", "weaknesses": "- The theoretical foundations are mainly taken from [1], and adapted into a new domain (text-to-image generation)\n\n- Experimental part - the used models are relatively obsolete - the method is demonstrated on SD 1.4 while there are newer and stronger model, that perhaps more capable of following user-specified instructions. Moreover, utilizing BLIP as an evaluation does not express the generated images quality. Perhaps inCode is capable of following the user specified attributes better but while decreasing images quality? I believe that images quality evaluation (e.g., FID) should be considered.\n\n- The method requires pre-defined set of attributes, which is feasible in simple datasets, but significantly less in real-world datasets such as ImageNet or COCO.\n\n- Minor - error in the axis names in Fig 8? \n\n\n[1] VARIATIONAL INFORMATION PURSUIT FOR INTERPRETABLE PREDICTIONS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "li5Y3iopKu", "reviewer": "Reviewer_DRti", "summary": "To improve generative modelling for multi-concept inputs, the paper proposes the Interpretable 'Compressed Descriptions for Image Generation' (InCoDe) Framework, which consists of a query encoder, an answer decoder, and a diffusion model. \n\nThe proposed framework sidesteps natural language to condition the generator. If I understand correctly, concept vectors replace natural language embeddings. The concept vector is constructed using a visual question answerer in tandem with an Embedder + MLP combination. The resulting embedding is fed into a pre-trained diffusion model.\n\nThe methods section builds upon the work of Chattopadhyay et al. The paper's key innovation is that we can modify the answers from the visual question answerer, which leads to changed images.\n\nThe proposed framework is experimentally tested on the CLEVER, LSUN Bedroom, and LSUN Churches Datasets.", "questions": "Figure 3:\n\n- Why does the QueryAnswerer in Figure 3 compute $\\hat{Q}(X)$ instead of $\\hat{Q}({\\hat{X}})$? Arent we minimizing the a \ncost term with $\\hat{Q}({\\hat{X}})$?\n\nEquation two:\n\n- How is the constraint in equation two enforced in practice?\n- Do we need the decoder if we are mainly interested in the modified image?\n\nFigure 8:\n\n- What does DT-IC refer to?\n- What does TopK-IC refer to?\n- Are these related to lines 342 to 351?\n\nTable 2:\n- What do Stab. D and Stu. D refer to?\n- Are these related to lines 342 to 351?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The work focuses on interpretability and is relevant to the community. The paper presents a solid extension of Chattopadhyay et al.'s (2023) work in an image generation setting.", "weaknesses": "Evaluation:\n\nIt would have been nice to compare it to an off-the-shelf diffusion model (https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf) conditioned on, for example, the facial attributes of the CelebA dataset.\nThe paper's introduction claims that generative models struggle when asked to compose images with multiple concepts. While Figure 2 illustrates the situation qualitatively, it would have been nice if the paper had followed up with a quantitative analysis of the situation, including a comparison to established work. Table 2 might be doing this. I am not sure if lines 342 to 351 refer to Table 2. Please clarify in the rebuttal.\n\n\nMinor points:\n\n- Links to the supplementary material are broken. Please consider submitting single files in the future. The author guide specifically encourages single file submission ( https://iclr.cc/Conferences/2025/AuthorGuide ).\n -  The writing was hard to follow at times. Consider lines 342 to 351, for example. It would have been nice to mention and link to a table where readers can find the numerical results right there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)