Interpretable Diffusion via Information Decomposition
Authors: Xianghao Kong
Ollie Liu
Han Li
Dani Yogatama
Greg Ver Steeg
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Models
Information Theory
Interpretable Machine Learning
Primary area: visualization or interpretation of learned representations
TL;DR: The study explores how denoising diffusion models understand data, revealing ways to measure relationships between images and text, and introducing techniques for object localization and directed image modifications.

Abstract: Denoising diffusion models enable conditional generation and density modeling of complex relationships like images and text. 
However, the nature of the learned relationships is opaque making it difficult to understand precisely what relationships between words and parts of an image are captured, or to predict the effect of an intervention. We illuminate the fine-grained relationships learned by diffusion models by noticing a precise relationship between diffusion and information decomposition. Exact expressions for mutual information and conditional mutual information can be written in terms of the denoising model. Furthermore, ${pointwise}$ estimates can be easily estimated as well, allowing us to ask questions about the relationships between specific images and captions. Decomposing information even further to understand which variables in a high-dimensional space carry information is a long-standing problem. For diffusion models, we show that a natural non-negative decomposition of mutual information emerges, allowing us to quantify informative relationships between words and pixels in an image. We exploit these new relations to measure the compositional understanding of diffusion models, to do unsupervised localization of objects in images, and to measure effects when selectively editing images through prompt interventions.

Published as a conference paper at ICLR 2024
INTERPRETABLE DIFFUSION VIA INFORMATION DECOMPOSITION
Xianghao Kong",, , Ollie Liu²*, Han Li', Dani Yogatama', Greg Ver Steeg! 'niiersstty of Califormia Riverside, UUniversity of Southern California
(xkong016, h11358, gregoryv/Juur edu, ( {zliu2898, yogatama]@usc.ce
ABSTRACT
Denoising diffusion models enable conditional generation and density modeling of complex relationships like images and text. However, the nature of the learned relationships is opaque making it difficult to understand precisely what relation- ships between words and parts of an image are captured, or predict the effect of an intervention. We illuminate the fine-gras ane-grained relationships learned by diffu- sion models by noticing precise relationship between diffusion and information decomposition. Exact expressions for mutual information and conditional mutual information can be written in terms the denoising model. Furthermore, point- wise estimates can be easily estimated well, allowing us ask questions about the relationships between specific images and captions. Decomposing informa- tion even further to understand which variables in high-dimensional space carry information long-standing problem. For diffusion models, we show that a natural non-negative decomposition of mutual information emerges, allowing to quantify informative relationships between words and pixels in an image. We exploit these new relations to measure the compositional understanding of diffu- sion models, to do unsupervised localization of objects in images, and to measure effects when selectively editing images through prompt interventions.
INTRODUCTION
Denoising diffusion models the state-of-the-ee tfr----rrttttttte- for modeling relationships between complex data like images and text. While diffusion models exhibit impressive generative abilities, we have little insight into precisely what relationships are learned (or neglected). Often, models have limited value without the ability to dissect their contents. For instance, in biology specifying which variables have an effect health outcomes is critical. As Al advances, more principled ways to probe learned relationships are needed to reveal and correct gaps between human and Al perspectives.
image
Intervention on
Ho: y|c) ((GG
Attention
A replica of a and her cub in glass
y" elephant
62.8 bits total
A turboprop that is the hanger For repair.
a airplane
y" w
1.8 bits tolal
Figure 1: We start (left) with real image from the COCO dataset. do a "prompt intervention" (3.3) generate new image. Next show conditional mutual information, illustrated using our pixel-wise decomposition, and attention maps for the modified word. Top row shows an image where prompt intervention has effect, while in the bottom it has little effect. Conditional mutual information reflects the effect intervention while attention does not.

---

Published as a conference paper at ICLR 2024
Quantifying the relationships learned in complex space like text and images is difficult. Informa- tion theory offers a black-box method to gauge how much information flows from inputs to outputs. This work proceeds from the novel observation that diffusion models naturally admit simple and versatile information decompasition that allows us to pinpoint information flows in fine detail which allows us understand and exploit these models in new ways.
For denoising diffusion models, recent work has explored how attention can highlight how models
depends on different words during generation (Tang al. 2022 Liu et al. 2023 Zhao al. 2023
Tian et al. 2023; Wang et al. 2023 Zhang et al.) 2023 Ma et al. 2023 He et al. 2023 Our information theoretic approach diverges from attention-
-eemmatioo-teoorticc
methods in three sigmificant ways.
First, attention requires not just white-box access but also dictates a particular network design. Our approach abstracts away from architecture details, and may be useful in the increasingly com- mon scenario where we interact with large generative models only through black-box API ac- cess (Ramesh et al.] 2022). Second, while attention is engineered toward specific tasks such as segmentation Tang et al. (2022) and image-text matching He et al. (2023). our information estima- tors can adapt to diverse applications. As an illustrative example, in $3. we automate the evaluation of compositional understanding for Stable Diffusion Rombach et al. (2022). Third, information flow as a dependence measure better captures the effects of interventions. Attention within a neural net- work does necessarily imply that the final output depends the attended input, Our Conditional Mutual Information (CMI) estimator correctly reflects that a word with small CMI will not affect the output (Fig. D We summarize our main contributions below.
We show that denoising diffusion models directly provide a natural and tractable way to decompose information fine-grained way, distinguishing relevant information at a per- sample (image) and per-variable (pixel) level. The utility of information decomposition is validated on variety of tasks below.
We provide a better quantification of compositional understanding capabilities of diffusion models. We find that on the ARO benchmark Yuksekgonul et al. (2022). diffusion models are significantly underestimated due to sub-optimal alignment scores.
We examine how attention and information in diffusion models localize specific text in images. While neither exactly align with the goal of object segmentation, information measures more effectively localize abstract words like adjectives, adverbs, and verbs.
How does a prompt intervention modify generated image? It is often possible to surgi- cally modify real images using prompt intervention techniques, but sometimes these in- terventions are completely ignored. We show that CMI is more effective at capturing the effects of intervention, due to the ability to take contextual information into account.
2 METHODS: DIFFUSION IS INFORMATION DECOMPOSITION
2.1 INFORMATION- THEORETIC PERSPECTIVE ON DIFFUSION MODELS
A diffusion model can be seen as noisy channel that takes samples from the data distribution, p(X x). and progressively adds Gaussian noise, Ta = vo(ajx Vo(-aJe, with e <~ <~N(0,,) I) (a variance preserving Gaussian channel with log SNR, a, using the standard sigmoid function). By learning to reverse denoise this noisy channel, we can generate samples from the original distribution (Sohl-Dickstein et al., 2013). a result with remarkable applications (Ramesh) al. (022)). The Gaussian noise channel has been studied in information theory since its incep- tion Shannnn 1948). A decade before diffusion models appeared in machine learning, [uo et al.) (2005) demonstrated that the information in this Gaussian noise channel, :(;;
exactly re- lated to the mean square eιror for optimal signal reconstruction. This result was influential because demonstrated for the first time that pntrmatiion-hhoreee of optimal denoisers. In this paper, we are interested in extending this result to other mutual infor- mation estimators, and to pointwise estimates of mutual information. Our focus is not on learning the reverse or denoising process for generating samples, but instead to measure relationships using information theory.
quantities could be related to estimation
For our results, we require the optimal denoiser, or Minimum Mean Square Error (MMSE) denoiser for predicting e at each noise level, a.
é, (a:) = arg min (p(a)).(() [lle- E(x,,) ]|3]
(1)
a(-)
2

---

Published as a conference paper at ICLR 2024
Note that we predict the noise, e, but could equivalently predict This optimal denoiser is exactly what diffusion models are trained to estimate. Instead of using denoisers for generation, we will see how to use them for measuring information theooretith relationships.
For the denoiser in Eq. the following expression holds exactly.
- -logp(x) = 1/2 Epic) [lle Ea (xa))) da + const
The value of the constant, const, will be irrelevant (MI) estimators and a decomposition from this expression. This expression shows that solving a denoising regression problem (which is easy for neural networks) is equivalent density modeling. No differential equations need to be referenced or solved to make this exact connection, unlike the approaches appearing in (Song al.] (2020) and McAllester (2023). The derivation of this result in Kong et al. (2022) closely mirrors Guo et al. (2005)'s original result, and is shown in App. A for completeness. This expression is extremely powerful and versatile for deriving -ne-grained information estimators, as we now show.
proceed to build Mutual Information
2.2 MUTUAL INFORMATION AND POINTWISE ESTIMATORS
Note that Eq. also holds with arbitrary conditioning. Let æ, y ~ p(X x, =y)) and éa (2.ly) be the optimal denoiser for p(ær (y) as in Eq. I Then we can write the conditional density as follows.
log (a (mly) = 1/2 Pepce) []le &o (m. 1)]l da + const.
(3)
This directly leads to an estimate of the following useful Log Likelihood Ratio (LLR). é, logg(x))) b - p(x) = 1/2 Epte) [lle- é, (x,,)) I| - ||c é, (x..lv)) da
(4)
The LLR is the integrated reduction in MMSE from conditioning on auxiliary variable, y. The mu- tual information, T(X: Y) can be defined via this LLR, T(x:Y) =E (Ep(a..) v) (ogg((>))) y) - log p(x)
We write MI using information theory notation, where capital X,Y are used to refer function- als of random variables with the distributions p(z, y) Cover & Thomas (2006). MI is an average measure of dependence, but we are often interested the strength of a relationship for a single point, or pointwise information. Pointwise information for a specific x; y is sometimes written with lowercase as i(x: y) and is defined so that the average recovers MI (Finn & Lizier 2018). Fano (1961) referred to what we call "mutual information" " as "average mutual information" and con- sidered what we call pointwise mutual information to be the more fundamental quantity. Pointwise information has been especially influential in NIP Levy & Goldberg (2014).
T(X;Y)= = p [i v)]
Defining property of pointwise information
Pointwise information is not unique and both quantities below satisfy this property.
is (x 3) 1/2 Ep(c)) [lle E (xa))1 ||e- E da
(5)
ia y) = 1/2 Eppc) (xx) En (x 12) da
The first standard definition comes from using more compact definition is derived using the orthogonality principle in SB] i has higher variance due to the presence of extra terms, while 1" has lower variance and is always non-negative. We will explore both estimators, but find the lower variance version that exploits the orthogonality principle is generally more useful (see $C.2 Note that while (average) MI is always non-negative, pointwise MI can be negative as ( y) log p(x)))) yy) log p(x) < O occurs when the observation of makes appear less likely. We can say negative pointwise information signals "misinformative" observation (Finn& & Lizier 2018).
log p(x) written via Eq. The second
All these expressions can be given conditional variants, where we condition on random vari- able, C. defining the context. CMI and its pointwise expression can be related as [(X; YIC)= Ep(n.y.e) [i(x; y/c]]. The pointwise versions of Eq. 5 can be obtained by conditioning all the denois- C, e.g. éo (Faly) (x c).
3

---

Published as a conference paper at ICLR 2024
2.3 PIXEL-WISE INFORMATION DECOMPOSITION
The pointwise information, i(æ;; y), does not tell us which variables, Is's, is are informative about which variables, yh- If æ is an image and y represents text prompt, then this would tell us which parts of the image particular word is informative about. One reason that information decomposition is highly nontrivial is that scenarios can arise where information in variables is synergistic, for example (Williams & Beer 2010). Our decomposition instead proceeds from the observation that the correspondence between information and MMSE leads to natural decomposition of information into a sum of terms for each variable. If æ E RR" we can write i(æ; y) v (2;; y) with:
(( = =122/ Ep(e) [(e (x(2n))3 - (E En (2 1y) da
(6)
y) 1/2 Epic) [(Ea(z(a) Ea( da
In other words, both variations of pointwise information can be written in terms of squared errors, and we can decompose the squared error into the error for each variable. For pixel-wise information for images with multiple channels, we sum over the contribution from each channel.
We can easily extend this for conditional information. Let x: represent a particular image, y = {y- c} = ("object", "a person holding an _".. Then we can estimate iy(2; /clc), which represents the information that word y. has about variable Zj. conditioned on the context, c. To get estimates using Eq. [ we just add conditioning on on both sides. Denoising images conditioned on arbitrary text is a standard task for diffusion models. An example of the estimator is shown in Fig. [ where the highlighted region represents the pixel-wise value ig (z;; y-le for pixel j.
2.4 NUMERICAL INFORMATION ESTIMATES
All information estimators we have introduced require evaluating a one-dimensional integral over an infinite range of SNRs. To estimate in practice we use importance sampling to evaluate the integral as in (Kingma et al. |022 Kong et al. 2022). We use truncated logistic for the importance sampling distribution for a. Empirically, we find that contributions to the integral for both very small and very large values of a are close to zero, so that truncating the distribution has little effect. Unlike MINE Belghazi et al. (2018) or variational estimators (Poole et al., 2019) the estimators presented optimizing direct upper or lower bound on MI. Instead, the estimator depends on finding the MMSE of both the conditional and unconditional denoising problems, and then combining them to estimate MI using Eq. 4 However, these two MMSE terms appear with opposite signs. In general, any neural network trained to minimize MSE may not achieve the global minimum for either both terms, we cannot guarantee that the estimate is either upper or lower bound. In practice, neural networks excel at regression problems, so we expect to achieve reasonable estimates. In all our results we use pretrained diffusion models which have been trained to minimize mean square under Gaussian noise, as required by Eq. [ (different papers differ in how much weight each a term receives in the objective, but in principle the MMSE for each a is independent, so the weighting shouldn't 't strongly affect an expressive enough neural network).
here do not depend
3 RESULTS
Section 2 establishes a precise connection between optimal denoisers and information. For ex- periments we consider diffusion models as approximating optimal denoisers which can be used to estimate information. All our experiments are performed with pre-trained latent space diffusion models RRmbach et al. (2022). Stable Diffusion v2.1 from Hugging Face unless otherwise noted. Latent diffusion models use a pre-trained autoencoder to embed images in lower resolution space before doing typical diffusion model training. We always consider x as the image in this lower dimensional space, but in displayed images show the images after decoding, and we use bilinear interpolation when visualizing heat maps at the higher resolution. See for experiment details and links to open source code.
4

---

Published as a conference paper at ICLR 2024
3.1 RELATION TESTING WITH POINTWISE INFORMATION
First, we consider information decomposition at the "pointwise" or per-image level. We make use of our estimator to compute summary statistics of image-text pair and quantify qualitative differ- ences across samples. As a novel application scenario, we apply our pointwise estimator to analyze compositional understanding of Stable Diffusion on the ARO benchmark Yuksekgonul al. (2022). Referring readers to Yuksekgonul ct al. (222)) for more detailed descriptions, the ARO benchmark is a suite of discriminative tasks wherein VLM is commissioned align an image a with its ground- truth caption from set of perturbations p == {ū; 1 induced by randomizing constituent orders from y. The compositional understanding capability of a VLM thus measured by the accuracy:
Ez [1
arg max s(x, y )))
E'E(v)
where s X' x y IR is an alignment score. For contrastive VLMs, s is chosen to be the cosine similarity between encoded image-text representations. We choose s(x, y) i"(æ; y) as our score function for diffusion models. In contrast, He al. (2023) compute & as an aggregate of latent atren- tion maps, while Krojer et (2023) adopt the negative MMSE. In Table [ we report performances of OpenCLIP Ilharco et al. (2021) and Stable Diffusion 2.1 Rombach et al. (2022) on the ARO benchmark, while controllinng model checkpoints with the same text encoder for fair comparison.
Table 1: Accuracy (%) of Stable Diffusion and its OpenCLIP backbone on the ARO Benchmark. *; conducts additional fine-tuning with compositional-awarc hard negatives.
Method
VG-A
VG-R
COCO
Flickr30k
OpenCLIP Ilharco et al.
2021
64.6
51.4
32.8
40.5
DifITM Krojer et al
2023)
62.9
50.0
23.5
33.2
HardNeg- DiffITM MifiNNee-DfiIIMM Krojer et al. 2023
67.6
52.3
34.4
48.6
Info. (Ours)
72.0
69.1
40.1
49.3
We observe that Stable Diffusion markedly improves in compositional understanding over Open- CLIP. Since the text encoder is frozen, can attribute this improvement solely to the de- noising objective and the visual component. More importantly, our information estimator sig- nificantly outperforms MMSE Krojer et al.] (2023), decidedly proving that previous works un- derestimate compositional understanding capabilities of diffusion models. Our observation pro- vides favorable evidence for adapting diffusion models for discriminative image-text matching (He et al. (003). However, these improvements are smaller compared to contrastive pre-training approaches that make use of composition-avee aware negative samples (Yuksekgonul et al.] 2022).
In Table 2 we report a subset of fine-grained per- formances across relation types, highlighting those with over 30% performance improvement in green and those with over 30% performance decrease in magenta. Interestingly, our highlighted categories correlate well with performance changes incurred by composition-awarr negative training, despite us- ing different CLIP backbones (c.f. Table 2 of Yuk- sekgonul et al. (2022). ). Most improvements are at- tributed to verbs that associate subjects with con- ceptually distinctive objects (e.g. "A boy sitting on a chair"). Our observation suggests that improve- ments in compositional understanding may stem predominantly from these "low-hanging fruits", and partially corroborate the hypothesis in Rassin et al. (2023), which posits that incorrect asso- ciations between entities and their visual attributes are attributed to the text encoder's inability to encode linguistic structure. We refer readers to for complete fine-grained results, implementa- tion details, and error analysis.
Table 2: Selected results on VG-R.
Info. (1)
OpenCLIP (t)
Accuracy (%)
69.1
51.4
beneath
50.0
90.0
covered in
14.3
50.0
feeding
50.0
100.0
grazing on
60.0
30.0
sitting on
78.9
49.7
wearing
84.1
44.9
3.2 PIXEL-WISE INFORMATION AND WORD LOCALIZATION
Next, we explore "pixel-wise information" that words in a caption contain about specific pixels in an image. According to 2.2 and 22.3 it naturally leads consider two potential approaches
5

---

Published as a conference paper at ICLR 2024
for validating this nuanced relationship. The first entails concentrating solely the mutual information between the object word and individual pixels. The second approach involves inves- tigating this mutual information when given the remaining context in the caption, i.c., conditional mutual information. As the success of our experiment relies heavily on the alignment between im- ages and text, carefully filtered two datasets, COCO-IT and COCO-WI, from the MSCOCO (Lin ict al. (2015) validation dataset. For specific details about dataset construction, please refer to Meanwhile, also provide an image-level information analysis on these datasets in C and visualize the diffusion process and its relation to information for 10 cases in
3.2.1 VISUALIZE MUTUAL INFORMATION AND CONDITIONAL MUTUAL INFORMATION
Given an image-text pair (æ: y) where y {y.: c), we compute pixel-Ievel mutual information as ig ((: y.), and the conditional one i (ac; y. |c) for pixel j, from Eq. 6 Eight cases are displayed in Fig. and we put more examples in C.3 and $C.4
Real COCO
P()y--)
Attention
Real COCO
P(xy--()
Pory.))
Attention
The old, adult stands near wire fence. elephant (noun)
A yellow dog grab yellow frisbee in the grass (verb)
group grazing the y, horses (noun)
sitting resting.
y- elderyy (adj)
toilet with sitting on the seat. = teddy bear (noun)
person an skis with one standing. vertically (adv)
leaving trail In the sky. airplane
cows grazing lake next to road.
turboprop the hangar repair.
standing on
people.
ys airplane (noun)
y- next to (prep)
Examples of localizing different types of words in images. The left half presents noun words, while the right half displays abstract words.
Figure 2:
In Fig. the visualizations of pixel-level MI and CMI are presented using common color scale. Upon comparison, becomes evident that CMI has a greater capacity to accentuate objects while simultaneously diminshing the background. This is attributed to the fact that MI solely computes the relationship between noun word y. and pixels, whereas CMI factors out context-related information from the complete prompt-to-pice pixel information. However, there are occasional instances of the op- posite effect, as observed in the example where 'y. = airplane' (left bottom in Fig. (2). In this case, it appears that CMI fails to highlight pixels related to 'airplane', while MI succeeds. This discrepancy arises from the presence of the word 'turboprop" in the context. Therefore the context, c, accurately describes the content in the image and 'airplane' adds no additional information. Compared to at- tention, CMI and MI qualitatively appear to focus more fine details of objects, like the eyes, ears, and tusks of an elephant, or the face of horses. For object words, we can use segmentation task to make quantitative statements in 3.2.2

---

Published as a conference paper at ICLR 2024
We also explore whether pixel-level CMI or MI provide intriguing insights for other types of entities. Fig. 2 (right) presents four different types of entities besides nouns, including verbs, adjec- tives, adverbs, numbers and prepositions. These words are quite abstract, and even through manual annotation, it can be challenging to precisely determine the corresponding pixels for them. The visualization results indicate that MI gives intuitively plausible results for these abstract items, pecially adjectives and adverbs, that highlight relevant finer details of objects more effectively than attention. Interestingly, MI's ability to locate abstract words within images aligns with the findings presented in $31 regarding relation testing (Table I0P.
3.2.2 LOCALIZING WORD INFORMATION IN IMAGES
The visualizations provided above offer intuitive demonstration of how pixel-wise CMI relates parts of an image to parts of a caption. Therefore, our curiosity naturally extends to whether this approach can be applied to word localization within images. Currently, the prevalent evaluation in- volves employing attention layers within Stable Diffusion models for object segmentation Tang et al.) (2022; Tian al. (2023); Wang et al. (203)). These methodologies heavily rely on attention layers and meticulously crafted heuristic heatmap generation. It's worth highlighting that during image generation, the utilization of multi-scale cross-attention layers allows for the rapid computation of the Diffuse Attention Attribution Map (DAAM) Tang et al. (2022). This not only facilitates segmen- tation but also introduces various intriguing opportunities for word localization analyses. We opted for DAAM our baseline choice due to its versatile applicability, and the detailed experimental design is documented in the $D2
We use mean Intersection over Union (mloU) as the evaluation
Table 3: Unsupervised Object Segmentation mIoU (%) Results COCO-IT
Method
50 steps
100 steps
200 steps
Whole Image Mask
14.94
14.94
14.94
Attention Tang et al. 2022)
34.52
34.90
35.35
Conditional Mutual Info.
32.31
33.24
33.63
Attention-Informatiion
42.46
42.71
42.84
metric for assessing the performance of pixel-level object segmentation. Table 3 illustrates that, in contrast to attention-baa based methods, pixel-wise CMI proves less effective for object segmentation, with an error analysis appearing in Table 9 The attention mechanism in DAAM combines high and low resolution image features across the multi-scale attention layers, akin to Feature Pyramid Networks (FPN) Lin al. (2017). facilitating superior feature fusion. CMI tends to focus more on the specific details that are unique to the target object rather than capturing the overall context of it. Although pixel-level CMI did not perform exceptionally well in object segmentation, the results from "Attention+Inforaatio
clearly demonstrate that the information coomattion-thoortiic diffusion process Kong al. (2022) does enhance the capacity of attention layers capture features.
attention versus information versus segmentation Looking at the heatmaps, it is clear that some parts of an object can be more informative than others, like faces or edges. Con- versely, contextual parts of an image that are not part of an object can still be informative about the object. For instance, we may identify an airplane by its vapor trail (Fig. ( Hence, neither attention nor information perfectly aligns with the goal of segmentation. One difference be- tween attention and mutual information is that when pixel pays "attention" to word in the prompt, it does not imply that modifications to the word would modify the prompt. This is best highlighted with conditional mutual information, where little information in a larger context ("turboprop jet"). To highlight this difference, we propose an experiment where we test whether intervening on words changes generated image. Our hypothesis is that if a word has low CMI, then its omission should not change the result. For attention, on the other hand, this is not necessarily true.
Discussion:
that
that informative word ("jet") may contribute
3.3 SELECTIVE IMAGE EDITING VIA PROMPT INTERVENTION
Diffusion models have gained widespread adoption by providing non-technical users with a natural language interface to create diverse, realistic images. Our focus so far has been how well diffusion
7

---

Published as a conference paper at ICLR 2024
models understand structure in real images. We can connect real and generated images by studying how well we can modify a real image, which is a popular use case for diffusion models discussed in $ We can validate our ability to measure informative relationships by seeing how well the measures align with effects under prompt intervention.
For this experiment, we adopt the perspective that diffusion models equivalent to continuous, invertible normalizing flows (Song et al., (020). In this case, the denoising model is interpreted as a score function, and an ODE depending on this score function smoothly maps the data distribution to a Gaussian, or vice versa. The solver we use for this ODE is the 2nd order deterministic solver with 100 steps from Karras al. )2022) We start with real image and prompt, then use the (conditional) score model to map the image to Gaussian latent space. In principle, this mapping is invertible, so if we reverse the dynamics we will recover the original image. We see in CC. that the original image is almost always recovered with high fidelity.
Next, we consider adding an intervention. While doing the reverse dynamics, we modify the (condi- tional) score by changing the denoiser prompt in some way. We focus in experiments on the effects of omitting a word, or swapping a word for a categorically similar word ("bear" + "elephant", for example). Typically, we find that much of the detail in the original image is preserved, with only the parts that relate to the modified word altered. Interestingly, we also find that in some cases interventions to a word are seemingly ignored (see Fig.
We want to explore whether attention or information measures predict the effect of interventions. When intervening by omitting word, we consider the conditional (pointwise) mutual information ((:; y ylc) from Eq. and for word swaps we use a difference of CMIs with y, y. representing the swapped word, as Fig. [ For attention, we aggregate the attention corresponding to a certain word during generation using the code from TTng et al. (2022). We find that a word can be ignored even if the attention heatmap highlights an object related to the word. In other words, attention to a word does not imply that it affects the generated outcome. One reason for this is that a word may provide little information beyond that in the context. For instance, in Fig. B] a woman in hospital is assumed to be on a bed, so omitting this word has no effect. CMI, on the other hand, correlates well with the effect of intervention, and "bed" in this example has low CMI.
To quantify our observation that conditional mutual information better correlates with the effect of intervention than attention models, we measure the Pearson correlation between a score (CMI or attention heatmap) and the effect of the intervention using L.2 distance between images before and after intervention. To get an image-level correlation, we correlated the aggregate scores per image across examples on COCO100-IT. We also consider the pixel-level correlations between L2 change per pixel and metrics, CMI and attention heatmaps. We average these correlations over all images and report the results in Table 4 CMI is much better predictor of changes at the image-level, which reflects the fact that it directly quantifies what additional information a word contributes after taking context into account. At the per-pixel level. CMI and attention typically correlated very well when changes were localized, but both performed poorly in cases where a small prompt change led to a global change in the image. Results visualizing this effect are shown along with additional experiments with word swap interventions in $C.55 Small dependence, as measured by information, correctly implied small effects from intervention. Large dependence, however, can lead to complex, global changes due to the nonlinearity of the generative process.
EWOO Imate
Intrrvation
10 std
lttwerion
<i wonar laying roddellug to mochines.
Table 4: Pearson Correlation with Im- age Change
c. brown fuzzy leaf fild tree. bear
(a:ylo)
Attention
hhart
Image-level
0.34 [= 010)
0.24 (#.011)
Pixel-level
0.27 1.02
0.21 (1.002)
10. bo toca
Figure 3: COCO images edited by omitting words from the prompt. Conditional mutual information better reflects the actual changes in the image after intervention. Table shows Pearson correlation between metrics (CMI or attention) versus L2 changes in image after intervention, at the image-level and pixel-level as discussed in $3.3
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Visual perception via diffusion models: Diffusion models' success in image generation has piqued interest in their text-image understanding abilities. A variety of pipeline designs are emerging in the realm of visual perception and caption comprehension, offering fresh perspectives and complexities. Notably, ODISE introduced a pipeline that combines diffusion models with discriminative models, achieving excellent results in open-vocabulary segmentation (Xu al., 2023). Similarly, OVDiff uses diffusion models to sample support images for specific textual categories and subsequently ex- tracts foreground and background features (Karazija et al. 2023). It then employs cosine similarity, often known as the CLIP (Radford et al., (02I)) filter, to segment objects effectively. MaskDiff (Le) et al., (023) and DifFSS (Tan et al. (2023) introduce a new breed of conditional diffusion models tai- lored for few-shot segmentation. DDP, on the other hand, takes the concept of conditional diffusion models and applies it to dense visual prediction tasks (Ji et al. 2023). VDP incorporates diffusion for range of downstream visual perception tasks such as semantic segmentation and depth estima- tion (Zhao et al. 2023). Several methods have begun to explore the utilization of attention layers in diffusion models for understanding text-image relationships in object segmentation (Tang al.,
2022
Tran et al. 2023 Wang et al. 2023 Zhang et al. 2023 Ma et al. (023).
Image editing via diffusion: Editing real images is area of growing interest both academically and commercially with approaches using natural text, similar images, and latent space modifications
to achieve desired effects (Nichol al. [02T]] Mao al. [023]] Liu et al.] [0233] BBaljji al.) 2022]
Kawar et al.) 2023 Mokady et al., (023). Su et al. 's approach most closely resembles the procedure used in our intervention experiments. Unlike previous work, we focus not on edit quality
but on using edits to validate the learned dependencies as measured by our information estimators.
Interpretable ML: Traditional methods for attribution based on gradient sensitivity rather than at- tention (Sundararajan et al.) 2017:: Lundberg & Lee| (017)) are seldom used in computer vision due to the well-known phenomenon that (adversarial) perturbations based on gradients are uncorrelated with human perception (Szegedy et al. (014). information-theoree
approaches for interpretabil- ity based on information decomposition (Williams & Beer (010) are mostly unexplored because no canonical approach exists (Kolchinsky (2022) and existing approaches are completely intractable for high-dimensional use cases (Reing et al. 202T), though there are some recent attempts to decom- pose redundant information with neural networks (Kleinman et al.. 2021 Liang et al., 2023), ) Our decomposition decomposes an information contribution from each variable, but does not explicitly separate unique and redundant components. "Interpretability" in machine learning is fuzzy con- cept which should be treated with caution (Lipton) 2018). We adopted an operational interpretation from information theory, which considers y - as a noisy channel and asks how many bits of information are communicated, using diffusion models to characterize the channel.
CONCLUSION
The eye-catching image generation capabilities of diffusion models have overshadowed the equally important and underutilized fact that they also represent the state-of-the-art in density model- ing (Kingma et al. 2021). Using the tight link between diffusion and information, we were able to introduce novel and tractable information decomposition. This significantly expands the useful-
ness of neural information estimators (Belghazi et al. 2018 Poole et al. 2019 Brekelmans et al.) 2023] by giving an interpretable measure of fine-grained relationships the level of individual sam- ples and variables. While we focused on vision tasks for ease of presentation and validation, infor- mation decomposition can be particularly valuable in biomedical applications like gene expression where we want to identify informative relationships for further study (Pepke & Ver Steeg 2017). Another promising application relates to contemporaneous works on mechanistic interpretability (Wang et al.] 2022, Hanna et al. 2023] inter alia) which seeks to identify "circuits"" subgraph of a neural network responsible for certain behaviors-by ablating individual network components to observe performance differences. For language models, differences are typically measured as the change in total probability of interested vocabularies; whereas metric design remains an open question for diffusion models. Our analyses indicate that the CMI estimators ing compositional understanding and localizing image edits. For future work, we are interested in exploring their potential metric candidates for identifying relevant eircuits in diffusion models.
apt for captur-
9

---

Published as a conference paper at ICLR 2024
6 ETHICS STATEMENT
Recent developments in internet-scale image-text datasets have raised substantial concerns over their lack of privacy, stereotypical representation of people, and political bias (Birhane et al. 2021 Peng et al. 2021 Schuhmann et al. 2021 inter alia). Dataset contamination entails considerable safety ramifications. First, models trained these datasets are susceptible to memorizing similar pitfalls. Second, complex text-to-image generation models pose significant challenges to interpretability and system monitoring (Hendryeks al. 2023), especially given the increasing popularity in black- box access to these models (Ramesh et al. 2022). These risks are exacerbated by these models' capability to synthesize photorealistic images, forming an "echo chamber" that reinforces existing biases in dataset collection pipelines. We, as researchers, shoulder the responsibility to analyze, monitor, and prevent the risks of such systems at scale.
We believe the application of our work has meaningful ethical implications. Our work aims to char- acterize the fine-graine -reaggg relationship between image and text. Although we primarily conduct our study on entities that do not explicitly entail societal implications, our approach can be conceiv- ably adapted to attribute prompt spans responsible for generating images that amplify demographic stereotypes (Bianchi et al., 2023), among many other potential risks. Aside from detecting known risks and biases, we should aim to design approaches that automatically identify system anomalics. In our image generation experiments, during prompt intervention. It is tempting to hypothesize that these differences may be reflective of dataset idiosyncrasies. We hope that estimator can contribute safeguards Al systems in high-stake application scenarios (Barrett et al. 2023). as diffusion mod- els could extrapolate beyond existing text-to-image generation to sensitive domains such as protein
observe changes in mutual information to be inconsistent
growing body of work that
design (Watson et al., (203), molecular structure discovery [[gashov et al. 2022). and interactive decision making (Chi et al. 2023).
Finally, it is imperative to address the ethical concerns associated with the use of low-wage labor for dataset annotation and evaluation (Perrigo 2023). Notably, existing benchmarks for assessing semantic understanding of image generation resort to manual evaluation (Conwell & Ullman 2022
Liu et al. 2022, Saharia al. 2022). In $3.1 we adapt our estimator for compositional understand- ing, and aim develop an automated metric for evaluation on a broader spectrum of tasks.
ACKNOWLEDGMENTS
GV thanks participants of the DEMICS workshop hosted at MPI Dresden for valuable feedback on this project
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 2.i::2211.01124,, 2022.
Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. Clustering with breg- man divergences. Journal of machine learning research, 6(10), 2005.
Clark Barrett, Brad Boyd, Ellie Burzstein, Nicholas Carlini. Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, al. Identifying and mitigating the security risks of generative ai. arXiv preprint arXiv:2308. 14840, 2023.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, pp. 531-540 2018.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to- image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference Fairness, Accountability, and Transparency, pp. 1493-1504, 2023,
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pomography, and malignant stereotypes. arXiv preprint arXiv:21 10.01193,, 2021.
10

---

Published as a conference paper at ICLR 2024
Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi, Greg Ver Steeg, Roger Grosse, and Alireza Makhzani. Improving mutual information estimation with annealed and energy-based bounds. arXiv preprint wwXiv:2333006992 2023.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 0rXiv:2303.04137, 2023.
Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv: 228.00005, 2022.
Thomas M Cover and Joy A Thomas. Elements of information theory. Wiley-Intersciennce 2006.
Robert M Fano. Transmission of Information: Statistical Theory of Communications. MIT Press, 1961.
Conor Finn and Joseph T Lizier. Pointwise partial information decompositionusing the specificity and ambiguity lattices. Entropy, 20(4):297, 2018
Dongning Guo, Shlomo Shamai, and Sergio Verdú. Mutual information and minimum mean- -quare error in gaussian channels. IEEE transactions on information theory, (1(4):1261-12) 2005.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- Inter- preting mathematical abilities in pre-trained language model. arXiv preprint arXiv:2305.00586 2023.
Xuchai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. Discriminative diffusion models few-shot vision and language learners, 2023.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306. 02006.22000 /2001, 2023.
Ilia Igashov, Hannes Stärk, Clément Vignac, Victor Garcia Satorras, Pascal Frossard, Max Welling, Michael Bronstein, and Bruno Correia. Equivariant 3d-conditional diffusion models for molecular linker design. arXiv preprint arXiv:2210.05274, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL http:a:ttcp.....ooo
zenodo.... 5143773
Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction, 2023.
Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero- shot open-vocabulary segmentation. arXiv preprint 23Xv:2306.09336, 2023.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: bast real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Steven M Kay, Fundamentals of statistical signal processing: estimation theory. Prentice-Hall, Inc., 1993.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021.
Michael Kleinman, Alessandro Achille, Stefano Soatto, and Jonathan Kao. Redundant informa- tion neural estimation. Entropy, 23(7):922, 2021.

---

Published as a conference paper at ICLR 2024
Artemy Kolchinsky. A novel approach to the partial information decomposition. Entropy, 24(3): 403, 2022.
Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoree diffusion. In The Eleventh International Conference Learning Representations (ICLR), 2022.
Benno Krojer, Elinor Poole-Dayar Vikram Voleti, Christopher Pal, and Siva Reddy. Are diffusion
models vision-and-language reasoners? In Thirty-seventhh Conference on Neural Information Processing Systems, 2023.
Minh-Quan Le, Tam V. Nguyen, Trung-Nghia Le, Thanh-Toan Do, Minh N. Do, and Minh-Triet Tran. Maskdiff: Modeling mask distribution with diffusion probabilistic model for few-shot instance segmentation, 2023.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Ad- vances in Neural Information Processing Systems, pp. 2177-2185, 2014.
Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Factorized contrastive learning: Going beyond multi-vier redundancy. arXiv preprint WrXiv:2306.05268, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015,
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection, 2017.
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of inter- pretability is both important and slippery. Queue, 16(3):31-57, 2018.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp- 423-439. Springer, 2022.
Nan Liu, Yilun Du, Shuang Li, Joshua B Tenenbaum, and Antonio Torralba. Unsuper- vised compositional concepts discovery with text-to-image generative models. arXiv preprint 05355 2023.
Scott Lundberg and Su-In Lee. unified approach to interpreting model predictions, 2017.
Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery, 2023,
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. arXiv preprint arXiv:2305. 033822, 2023.
David McAllester. On the mathematics of diffusion models. arXiv preprint I 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever. and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.107411 2021.
Kenny Peng, Arunesh Mathur, and Arvind Narayanan. Mitigating dataset harms requires steward- ship: Lessons from 1000 papers. arXiv preprint arXiv: 108. 100.0092,, 2021.
Shirley Pepke and Greg Ver Steeg. Comprchensive discovery of subsample gene expression com- ponents by information explanation: therapeutic implications cancer. BMC medical genomics, 10(1):12, 2017. URL URL https: //cci//c.i/////// wrr.oo/////// /ttp://1.1111111112220-0--111110
Billy Perrigo. Openai used kenyan workers less than $2 per hour: Exclusive, Jan 2023.
12

---

Published as a conference paper at ICLR 2024
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123. 2022.
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusson models: Enhancing attribute correspondence through attention map alignment. arXiv preprint arXiv:2306. 08877, 2023.
Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Influence decompositions for neural network attri- bution. In The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11444 11487, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprins arXiv:2111.021 14, 2021.
C.E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27: 379-423. 1948.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv preprint crXiv:1503.03556 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
Xuan Su. Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. arXiv preprint arXhv:2203.08382. 2022.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017.
C. Szegedy, W. Zaremba, I. Sutskever, Bruna, Erhan, I. Goodfellow, and Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Weimin Tan, Siyuan Chen, and Bo Yan. Diffss: Diffusion model for few-shot semantic segmenta- tion, 2023.
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stene- torp, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention, 2022.
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion, 2023.
Jinglong Wang. Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, and Dong Xu. Diffusion model is secretly training-free open vocabulary semantic segmenter, 2023.
13

---

Published as a conference paper at ICLR 2024
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, pp. 1-3, 2023.
P.L. Williams and R.D. Beer. arXiv:1004.251,, 2010.
Nonnegative decomposition of multivariate information.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why lision-language models behave like bag-of-words models, and what to do about it? arXiv preprint arXiv:2210.01988 2022.
Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy Ma. Diffusionengine: Diffusion model is scalable data engine for object detection, 2023.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception, 2023.
14


Reviews:
[{"review_id": "9tqpR4KZ9h", "reviewer": "Reviewer_EMcD", "summary": "This work presents a new method for information decomposition at both pointwise and pixel-wise level via well-trained diffusion models, utilizing the results such that there is an exact connection between mutual information and MMSE denoising estimators under Gaussian noise, as well as the exact correspondence between denoising and density modeling. The work demonstrates the ability of such information decomposition to measure the compositional understanding of diffusion models, to perform unsupervised localization of objects in images, and to measure effects of selective image editing via prompt interventions. The proposed metric, *Conditional Mutual Information (CMI) estimators*, is displayed as an alternative to attention in capturing the effect of text interventions in image generation.", "questions": "1. Could the authors give a quick example of how $i^o(\\mathbf{x};\\mathbf{y})$ from Eq. (5) is computed? Under the DDPM parameterization, the noise network predicts the forward noise, thus it makes sense to start with a real image $\\mathbf{x}$, sample $\\mathbf{\\epsilon}$ to obtain $\\mathbf{x}_{\\alpha}$, and compute the L2 term: this utilizes the forward diffusion. Meanwhile, the goal is to decompose the *generated* samples, which uses the reverse diffusion. Therefore, it’s a bit unclear on how these mutual information estimators are calculated.\n2. The authors mentioned that Eq. (2) would hold for an “optimal denoiser”: what does it mean for a denoiser to be optimal?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The finding that attention to a word doesn’t necessarily imply this word would affect the generated outcome is interesting, and Figure 1 shows how CMI can be helpful in interpreting the model for the cases where attention cannot.\n2. The authors present a standalone section to discuss ethical implications of the models they are working on, as well as how their method could help deal with the potential risks.", "weaknesses": "1. The walkthrough of some key results that help construct the proposed information decomposition approach is too brief: it would be nice to have a more in-depth review of these results, for example, the derivation of Eq. (2), the part of diffusion model parameterization that’s related to MMSE, etc. Meanwhile, the experiment section is relatively lengthy: some parts could be moved to the appendix, leaving the contents most closely connected to information decomposition in the main paper.\n2. Error bars in Table 3 for the comparison between CMI and attention are not reported.\n3. Source code is not shared to facilitate reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ScLqdfOwT", "reviewer": "Reviewer_ajkd", "summary": "This paper proposes a method to visualize the relationship between image and text caption based on the mutual information (MI) estimated by a pre-trained diffusion model. The core observation is that the log probability density can be estimated up to constant by integrating the diffusion-model loss over timesteps (McAllester, 2023), which enables the MI between image and text-caption to be computed by the difference of the estimated log density from the conditional and the unconditional inference result of a text-to-image diffusion model. While the original MI is the average over the samples, the authors consider per-sample (point-wise) MI and further decompose it to per-pixel MI, which gives rise to a visualization of which region in the image is related with a certain word. In experiments, the proposed visualization method showed superior interpretation over the attention-based method.", "questions": "- Are the “standard definition” and the “orthogonality principle” the full list of the decomposition possibilities? If not, we might need some justification that why these two are better suited as pointwise information.\n    - The main manuscript mentions that both $i^s$ and $i^o$ will be explored. Can you locate where the $i^s$ results are?\n- Is there a notable relationship between $i^o$ and the classifier-free guidance, considering their similarity in computation? For example, if the guidance scale gets larger, does it make a enhanced visualization or a larger MI between the text and image?\n- The Eq. (5) and (6) contains expectation over the noise, $\\epsilon$. But different noises will make dramatically different images. How are the visualizations in Fig. 2 are generated, well aligned with the image?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper considers an important topic in text-to-image diffusion models. Analyzing the influence of each word to the generated image has been attracting many researchers, since it serves as an essential tool to come up with ideas for improving or debugging the semantic alignment of the generated image and for developing image-editing methods.\n\nWhile a large body of literature exploits the cross-attention maps for the problem, this paper considers a new approach acts on pixels. The approach itself is quite similar to the classifier-free guidance that contrasts the conditional and unconditional inference results, which has been used in most of the diffusion models, but have strength by grounding on information theory. \n\nAlso, experimental results demonstrate the proposed method gives better interpretability compared to the attention-based methods.", "weaknesses": "A weak point of this paper is that the main manuscript seems to be not self-contained. While each of the following can be rather minor, they together drops readability.\n\n- References:\n    - The proposed method is heavily relying on the log-density estimator of McCallester, 2023, but lacks details. Further explanation on the practical behavior of this estimator and how this estimator can be derived, at least in abstract sense, would help the understanding.\n- Rather important figures and tables are in appendix.\n    - There is a fair amount of discussion in the main manuscript with Table 9 and Figure 10, but these are on the appendix thus making difficult to read.\n- Dataset information\n    - CoCo-IT is the main dataset used for evaluation, but how it has been filtered from CoCo is not described at least briefly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9YDsVttBil", "reviewer": "Reviewer_qKVB", "summary": "The paper introduces the application of information theory to interpret text-to-image diffusion models. The authors present a way to measure and utilize the mutual information between the image and the text conditioning to visualize \"where\" a specific prompt maps to. They also demonstrate how this mapping from text to pixels can be used to relate images with text and localize words as well as help in interpretability by visualizing the effects of the text prompt in the generation process.", "questions": "- How is the attention extracted in Figure 2? The proposed per-pixel mutual information averages the result over all timesteps of the diffusion process. However, the attention seems to be extracted from a single forward pass of the model. That could be disadvantageous as the attention mechanism highlights different scales of features at different timesteps. In section 3.2.2 the authors acknowledge the existence of a  \"feature pyramid\" in the attention but is this again for a single time step of the diffusion process?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel approach to measuring the relationship between the text conditioning and the image pixels in text-to-image diffusion models. Previous attempts that focused on interpreting the cross-attention operations within the model were architecture-specific and usually required additional processing or fine-tuning to translate attention into meaningful signals. In contrast, the authors show that their information-theoretic approach can be easily interpreted without additional overhead.\n\n- The experiments section clearly demonstrates both the advantages and disadvantages of the proposed approach. The paper paints a clear picture of when the mutual information can be useful and where attention can be used instead.\n\n- The method is well-grounded, with Section 2 presenting the approach and necessary background thoroughly. The paper is overall well-written.", "weaknesses": "- The authors present this paper as an approach to interpretability for text-to-image diffusion models but there is no clear demonstration of where this could be applied other than better visualizations of text-image relations. It would be informative to discuss (or even demonstrate) whether the mutual information can be exploited to better align text-to-image models with human preferences during inference time or fine-tuning. The broader scope of interpreting the generation process by relating pixels to words is missing from the paper and could greatly benefit it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EJ9WnYNaZB", "reviewer": "Reviewer_S4G8", "summary": "Denoising diffusion models are used for complex data relationships like images and text, and this paper has revealed a link between these models and information decomposition, enabling precise characterization of relationships, including mutual information, which can be used to understand and measure the relationships between specific elements in high-dimensional data like words and image pixels.", "questions": "1. Inadequate preliminary information on related information theory makes Section 2 difficult to comprehend.\n2. Clicking on citations does not redirect to the bibliography, making it challenging to locate the correct reference.\n3. Apart from the formations, understanding the implementation of this method is challenging, necessitating more detailed explanations from the authors.\n4. Improved demonstration of how this method enhances the generation process in experiments would be beneficial.\n5. Does Section 6 exceed the limitation of 9 pages?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a new perspective to explore the relationship between image and text in the Generation field, and quantify compositional understanding capabilities of diffusion models at a certain level.", "weaknesses": "However, the primary drawback lies in the paper's somewhat inadequate presentation, making comprehension challenging. Additionally, a limitation is the insufficient experimentation, which fails to demonstrate how this method could enhance the generation process. To conclude, I strongly advise the authors to undergo a significant revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBRMMY2XEB", "reviewer": "Reviewer_gEHf", "summary": "This work discusses the interpretability of denoising diffusion models, which are used for conditional generation and density modeling of complex relationships like images and text. The authors aim to understand the relationships learned by these models by establishing a connection between diffusion and information decomposition. They provide exact expressions for mutual information and conditional mutual information in terms of the denoising model. The paper also introduces methods to quantify informative relationships between words and pixels in an image. The authors utilize these relationships to measure the compositional understanding of diffusion models, perform unsupervised localization of objects in images, and measure effects when editing images through prompt interventions.", "questions": "- The example in first row of Fig. 1 seems to have difference other than the bear and elephant, why does the heat map only show difference in salient objects? Other examples have similar questions, but Fig. 1 is most obvious. \n- What's the advantage compared to attention based method?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper provides an approach to illuminate the fine-grained relationships learned by diffusion models.\n- The authors introduce a natural non-negative decomposition of mutual information, which can be beneficial for understanding high-dimensional data.\n- The paper offers practical applications, such as unsupervised localization of objects in images.\n- The research uses information theory to provide a black-box method for understanding relationships in complex spaces like text and images.", "weaknesses": "- The method is relatively trivial, replacing word in the prompt and comparing the difference in pixel level\n- lack of comprehensive comparison with attention based method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRFmhe0zG5", "reviewer": "Reviewer_U1vn", "summary": "This paper proposes to investigate the fine-grained relationships learned by diffusion models by noticing a precise relationship between diffusion and information decomposition.  It exploits these new relations to measure the compositional understanding of diffusion models, to do unsupervised localization of objects in images, and to measure effects of prompt interventions.", "questions": "Refer to the weakness and questions above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It is interesting to view diffusion models from the information theory prespective. For examle, exact expressions for mutual information and conditional mutual information can be expressed in terms of the denoising model.\n2. Compared with attention, the proposed mutual information seems more exact in identifying the relationship between the text and image. Massive visualization in the supplement demonstrate the effectiveness of this method.", "weaknesses": "1. It is still unclear of the concrete implementation. Without access to the network, but just with API, how to get the mutual information and corresponding visualization.\n2. What is the application field. Attention-based methods are widely applied to image editing and show impressive results. What is the advantage of this decomposition method, compared to attetion.\n3. The paper is poor structured and hard to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)