What the DAAM: Interpreting Stable Diffusion Using Cross Attention
Year: 2023

Abstract:
Diffusion models are a milestone in text-toimage generation, but they remain poorly understood, lacking interpretability analyses. In this paper, we perform a text–image attribution analysis on Stable Diffusion, a recently opensourced model. To produce attribution maps, we upscale and aggregate cross-attention maps in the denoising module, naming our method DAAM. We validate it by testing its segmentation ability on nouns, as well as its generalized attribution quality on all parts of speech, rated by humans. On two generated datasets, we attain a competitive 58.8–64.8 mIoU on noun segmentation and fair to good mean opinion scores (3.4–4.2) on generalized attribution. Then, we apply DAAM to study the role of syntax in the pixel space across head–dependent heat map interaction patterns for ten common dependency relations. We show that, for some relations, the head map consistently subsumes the dependent, while the opposite is true for others. Finally, we study several semantic phenomena, focusing on feature entanglement; we find that the presence of cohyponyms worsens generation quality by 9%, and descriptive adjectives attend too broadly. We are the first to interpret large diffusion models from a visuolinguistic perspective, which enables future research. Our code is at https://github.com/castorini/daam.

heading: 1 Introduction
text: Diffusion models trained on billions of captioned images represent state-of-the-art text-to-image generation (Yang et al., 2022), with some achieving photorealism, such as Google’s Imagen (Saharia et al., 2022) and OpenAI’s DALL-E 2 (Ramesh et al., 2022). However, despite their quality and popularity, the dynamics of their image synthesis remain undercharacterized. Citing ethics, corporations have restricted the general public from using the models and their weights, preventing effective ∗Equal contribution.
analysis. To overcome this barrier, Stability AI recently open-sourced Stable Diffusion (Rombach et al., 2022), a 1.1 billion-parameter latent diffusion model pretrained and fine-tuned on the LAION 5- billion image dataset (Schuhmann et al., 2022).
We probe Stable Diffusion to provide insight into large diffusion models. Focusing on text–image attribution, our central question is, “How does an input word influence parts of a generated image?” To this, we propose to produce 2D attribution maps for each word by combining cross-attention maps in the model. A related work in prompt-guided editing from Hertz et al. (2022) conjectures that per-head cross attention relates words to areas in Imagen-generated images, but they fall short of constructing global per-word attribution maps. We name our method diffusion attentive attribution maps, or “DAAM;” see Figure 1 for an example.
To evaluate the veracity of DAAM, we apply it to a semantic segmentation task (Lin et al., 2014) on generated imagery, comparing DAAM maps with annotation. We attain a 58.8–64.8 mean intersection over union (mIoU) score competitive with unsupervised segmentation models, described in Section 3.1. We further bolster these results using a generalized study covering all parts of speech (all in Penn Treebank; Marcinkiewicz, 1994), such as adjectives and verbs. Through human annotation, we show that the mean opinion score (MOS) is above fair to good (3.4–4.2) on interpretable words.
Next, we study how relationships in the syntactic space of prompts relate to those in the pixel space
5644
of images. We assess head–dependent DAAM interactions across ten common syntactic relations (enhanced Universal Dependencies; Schuster and Manning, 2016), finding that, for some, the heat map of the dependent strongly subsumes the head’s, while the opposite is true for others. For others, such as coreferent word pairs, the words’ maps greatly overlap, indicating coreferent understanding during generation. We assign intuition to our observations; for example, we observe that the maps of verbs contain their subjects, suggesting that verbs strongly contextualize the generation of both the subjects and their surroundings.
Finally, we form hypotheses to further our syntactic findings, studying semantic phenomena using DAAM, particularly those affecting image quality. In Section 5.1, we demonstrate that, in constructed prompts with two distinct nouns, cohyponyms have worse quality (9% worse than non-cohyponyms), e.g., “a giraffe and a zebra” generates a giraffe or a zebra, but not both. Cohyponym status and generation incorrectness each increases the amount of heat map overlap, advancing DAAM’s utility toward improving diffusion models. We also show in Section 5.2 that descriptive adjectives attend too broadly across the image, far beyond their nouns. If we fix the scene layout (Hertz et al., 2022) and vary only the adjective, the entire image changes, not just the noun. These two phenomena suggest feature entanglement, where objects are entangled with both the scene and other objects.
In summary, our contributions are as follows: (1) we propose and evaluate an attribution method, novel within the context of interpreting diffusion models, measuring which parts of the generated image the words influence most; (2) we provide new insight into how syntactic relationships map to generated pixels, finding evidence for directional imbalance in head–dependent DAAM map overlap, alongside visual intuition (and counterintuition) in the behaviors of nominals, modifiers, and function words; and (3) we shine light on failure cases in diffusion models, showing that descriptive adjectival modifiers and cohyponyms result in entangled features and DAAM maps.

heading: 2 Our Approach

heading: 2.1 Preliminaries
text: Latent diffusion models (Rombach et al., 2022) are a class of denoising generative models that are trained to synthesize high-fidelity images from ran-
dom noise through a gradual denoising process, optionally conditioned on text. They generally comprise three components: a deep language model like CLIP (Radford et al., 2021) for producing word embeddings; a variational autoencoder (VAE; Kingma and Welling, 2013) which encodes and decodes latent vectors for images; and a timeconditional U-Net (Ronneberger et al., 2015) for gradually denoising latent vectors. To generate an image, we initialize the latent vectors to random noise, feed in a text prompt, then iteratively denoise the latent vectors with the U-Net and decode the final vector into an image with the VAE.
Formally, given an image, the VAE encodes it as a latent vector ℓt0 ∈ Rd. Define a forward “noise injecting” Markov chain p(ℓti |ℓti−1) := N (ℓti ; √ 1− αtiℓt0 , αtiI) where {αti}Ti=1 is defined following a schedule so that p(ℓtT ) is approximately zero-mean isotropic. The corresponding denoising reverse chain is then parameterized as
p(ℓti−1 |ℓti) := N (ℓti−1 ; 1√1−αti (ℓti + αtiϵθ(ℓti , ti)), αtiI), (1)
for some denoising network ϵθ(ℓ, t) with parameters θ. Intuitively, the forward process iteratively adds noise to some signal at a fixed rate, while the reverse process, using a neural network, removes noise until recovering the signal. To train the network, given caption–image pairs, we optimize
minθ ∑T
i=1 ζiEp(ℓti |ℓt0 )∥ϵθ(ℓti , ti)−∇ℓti log p(ℓti |ℓt0)∥ 2 2, (2)
where {ζi}Ti=1 are constants computed as ζi := 1−∏ij=1(1− αj). The objective is a reweighted form of the evidence lower bound for score matching (Song et al., 2021). To generate a latent vector, we initialize ℓ̂tT as Gaussian noise and iterate
ℓ̂ti−1 = 1√
1−αti (ℓ̂ti + αtiϵθ(ℓ̂ti , ti)) +
√ αtizti . (3)
In practice, we apply various optimizations to improve the convergence of the above step, like modeling the reverse process as an ODE (Song et al., 2021), but this definition suffices for us. We can additionally condition the latent vectors on text and pass word embeddings X := [x1; · · · ;xlW ] to ϵθ(ℓ, t;X). Finally, the VAE decodes the denoised latent ℓ̂t0 to an image. For this paper, we use the publicly available weights of the state-ofthe-art, 1.1 billion-parameter Stable Diffusion 2.0 model (Rombach et al., 2022), trained on 5 billion caption–image pairs (Schuhmann et al., 2022) and implemented in HuggingFace’s Diffusers library (von Platen et al., 2022).

heading: 2.2 Diffusion Attentive Attribution Maps
text: Given a large-scale latent diffusion model for textto-image synthesis, which parts of an image does each word influence most? One way to achieve this would be attribution approaches, which are mainly perturbation- and gradient-based (AlvarezMelis and Jaakkola, 2018; Selvaraju et al., 2017), where saliency maps are constructed either from the first derivative of the output with respect to the input, or from input perturbation to see how the output changes. Unfortunately, gradient methods prove intractable due to needing a backpropagation pass for every pixel for all T time steps, and even minor perturbations result in significantly different images in our pilot experiments.
Instead, we use ideas from natural language processing, where word attention was found to indicate lexical attribution (Clark et al., 2019), as well as the spatial layout of Imagen’s images (Hertz et al., 2022). In diffusion models, attention mechanisms cross-contextualize text embeddings with coordinate-aware latent representations (Rombach et al., 2022) of the image, outputting scores for each token–image patch pair. Attention scores lend themselves readily to interpretation since they are already normalized in [0, 1].Thus, for pixelwise attribution, we propose to aggregate these scores over the spatiotemporal dimensions and interpolate them across the image.
We turn our attention to the denoising network ϵθ(ℓ, t;X) responsible for the synthesis. While the subnetwork can take any form, U-Nets remain the popular choice (Ronneberger et al., 2015) due to their strong image segmentation ability. They consist of a series of downsampling convolutional blocks, each of which preserves some local context, followed by upsampling deconvolutional blocks, which restore the original input size to the output. Specifically, given a 2D latent ℓt ∈ Rw×h, the downsampling blocks output a series of vectors {h↓i,t}Ki=1, where h ↓ i,t ∈ R ⌈ w ci ⌉×⌈ h ci ⌉ for some c > 1. The upsampling blocks then iteratively upscale h↓K,t to {h ↑ i,t}0i=K−1 ∈ R ⌈ w ci ⌉×⌈ h ci ⌉. To condition these representations on word embeddings, Rombach et al. (2022) use multi-headed cross-attention layers (Vaswani et al., 2017)
h↓i,t := F (i) t (ĥ ↓ i,t,X) · (W (i)v X), (4)
F (i) t (ĥ ↓ i,t,X) := softmax
( (W (i) q ĥ ↓ i,t)(W (i) k X) T / √ d ) , (5)
where F (i)↓t ∈ R⌈ w ci ⌉×⌈ h ci ⌉×lH×lW and Wk, Wq, and Wv are projection matrices with lH attention
heads. The same mechanism applies when upsampling h↑i . For brevity, we denote the respective attention score arrays as F (i)↓t and F (i)↑ t , and we implicitly broadcast matrix multiplications as per NumPy convention (Harris et al., 2020). Spatiotemporal aggregation. F (i)↓t [x, y, ℓ, k] is normalized to [0, 1] and connects the kth word to the intermediate coordinate (x, y) for the ith downsampling block and ℓth head. Due to the fully convolutional nature of U-Net (and the VAE), the intermediate coordinates locally map to a surrounding affected square area in the final image, the scores thus relating each word to that image patch. However, different layers produce heat maps with varying scales, deepest ones being the coarsest (e.g., h↓K,t and h ↑ K−1,t), requiring spatial normalization to create a single heat map. To do this, we upscale all intermediate attention score arrays to the original image size using bicubic interpolation, then sum them over the heads, layers, and time steps:
DRk [x, y] := ∑
i,j,ℓ
F̃ (i)↓ tj ,k,ℓ [x, y] + F̃ (i)↑ tj ,k,ℓ [x, y], (6)
where k is the kth word and F̃ (i)↓tj ,k,ℓ[x, y] is shorthand for F (i)↓t [x, y, ℓ, k], bicubically upscaled to fixed size (w, h).1 Since DRk is positive and scale normalized (summing normalized values preserves linear scale), we can visualize it as a soft heat map, with higher values having greater attribution. To generate a hard, binary heat map (either a pixel is influenced or not), we can threshold DRk as
DIτk [x, y] := I ( DRk [x, y] ≥ τ max
i,j DRk [i, j]
) , (7)
where I(·) is the indicator function and τ ∈ [0, 1]. See Figure 2 for an illustration of DAAM. 1We show that aggregating across all time steps and layers is indeed necessary in Section A.1.

heading: 3 Attribution Analyses

heading: 3.1 Object Attribution
text: Quantitative evaluation of our method is challenging, but we can attempt to draw upon existing annotated datasets and methods to see how well our method aligns. A popular visuosemantic task is image segmentation, where areas (i.e., segmentation masks) are given a semantically meaningful label, commonly nouns. If DAAM is accurate, then our attention maps should arguably align with the image segmentation labels for these tasks—despite not having been trained to perform this task. Setup. We ran Stable Diffusion 2.0-base using 30 inference steps per image with the DPM (Lu et al., 2022) solver—see Appendix A.1. We then synthesized one set of images using the validation set of the COCO image captions dataset (Lin et al., 2014), representing realistic prompts, and another set by randomly swapping nouns in the same set (holding the vocabulary fixed), representing unrealism. The purpose of the second set was to see how well the model generalized to uncanny prompts, whose composition was unlikely to have been encountered at training time. We named the two sets “COCO-Gen” and “Unreal-Gen,” each with 500 prompt–image pairs. For ground truth, we extracted all countable nouns from the prompts, then hand-segmented each present noun in the image.
To compute binary DAAM segmentation masks, we used Eqn. 7 with thresholds τ ∈ {0.3, 0.4, 0.5}, for each noun in the ground truth. We refer to these methods as DAAM-⟨τ⟩, e.g., DAAM-0.3. For supervised baselines, we evaluated semantic segmentation models trained explicitly on COCO,
like Mask R-CNN (He et al., 2017) with a ResNet101 backbone (He et al., 2016), QueryInst (Fang et al., 2021) with ResNet-101-FPN (Lin et al., 2017), and Mask2Former (Cheng et al., 2022) with Swin-S (Liu et al., 2021), all implemented in MMDetection (Chen et al., 2019), as well as the open-vocabulary CLIPSeg (Lüddecke and Ecker, 2022) trained on the PhraseCut dataset (Wu et al., 2020). We note that CLIPSeg’s setup resembles ours since the image captions are assumed given as well. However, theirs is supervised since they train their model on segmentation labels as well. Our unsupervised baselines consisted of the state-ofthe-art STEGO (Hamilton et al., 2021) and PiCIE + H (Cho et al., 2021). As is standard (Lin et al., 2014), we evaluated all approaches using the mean intersection over union (mIoU) over the prediction– truth mask pairs. We denote mIoU80 when restricted to the 80 COCO classes that the supervised baselines were trained on and mIoU∞ as the mIoU without the class restriction; see Sec. B for details.
Results. We present results in Table 1. The COCO-supervised models (rows 1–3) are constrained to COCO’s 80 classes (e.g., “cat,” “cake”), while DAAM (rows 5–7) is open vocabulary; thus, DAAM outperforms them by 22–28 points in mIoU∞ and underperforms by 20 points in mIoU80. CLIPSeg (row 4), an open-vocabulary model trained on semantic segmentation datasets, achieves the best of both worlds in mIoU80 and mIoU∞, with the highest mIoU∞ overall and high mIoU80. However, its restriction to nouns precludes it from generalized segmentation (e.g., verbs). DAAM largely outperforms both unsupervised baselines (rows 6–7) by a margin of 6–27 points (except for STEGO on COCO-Gen mIoU∞, where it’s similar), likely because we assume the prompts to be provided. Similar findings hold on the unrealistic Unreal-Gen set, showing that DAAM is resilient to nonsensical texts, confirming that DAAM works when Stable Diffusion has to generalize in composition.
As for τ , 0.4 works best on all splits, though it isn’t too sensitive, varying by 0.1–5 points in mIoU. We also show that all layers and time steps contribute to DAAM’s segmentation quality in Section A.1. Overall, DAAM forms a strong baseline of 58.1–64.8 mIoU80. As our goal is to prove sanity, not state of the art, we conclude that DAAM is sane for noun attribution, which we extend to all parts of speech in the next section.

heading: 3.2 Generalized Attribution
text: We extend our veracity analyses beyond nouns to all parts of speech, such as adjectives and verbs, to show that DAAM is more generally applicable. A high-quality, reliable analysis requires human annotation; hence, we ask human raters to evaluate the attribution quality of DAAM maps, using a five-point Likert scale.
This setup generalizes that of the last section because words in general are not visually separable, which prevents effective segmentation annotation. For example, in the prompt “people running,” it is unclear where to visually segment “running.” Is it just the knees and feet of the runners, or is it also the swinging arms? On the contrary, if annotators are instead given the proposed heat maps for “running,” they can make a judgement on how well the maps reflect the word.
Setup. To construct our word–image dataset, we first randomly sampled 200 words from each of the 14 most common part-of-speech tags in COCO, extracted with spaCy, for a total of 2,800 unique word–prompt pairs. Next, we generated images alongside DAAM maps for all pairs, varying the random seed each time. To gather human judgements, we built our annotation interface in Amazon MTurk, a crowdsourcing platform. We presented the generated image, the heat map, and the prompt with the target word in red, beside a question asking expert workers to rate how well the highlighting reflects the word. They then selected a rating among one of “bad,” “poor,” “fair,” “good,” and “excellent”, as well as an option to declare the image itself as too poor or the word too abstract to inter-
pret. For quality control, we removed annotators failing attention tests. For further robustness, we assigned three unique raters to each example. We provide further details on the user interface and annotation process in the appendix section A.2.
Results. Our examples were judged by a total of fifty raters, none producing more than 18% of the total number of annotations. We filtered out all word–image pairs deemed too abstract (e.g., “the”), when any one of the three assigned raters selected that option. This resulted in six interpretable partof-speech tags with enough judgements; see the appendix for detailed statistics. To compute the final score of each word–image pair, we took the median of the three raters’ opinions.
We plot our results in Figure 3. In the top subplot, we show that DAAM maps for adjectives, verbs, nouns, and proper nouns attain close to or slightly above “good,” whereas the ones for numerals and adverbs are closer to “fair.” This agrees with the generated examples in Figure 4, where numerals (see the giraffes’ edges) and adverbs (feet and ground motion blur) are less intuitively highlighted than adjectives (blue part of teapot), verbs (fists and legs in running form), and nouns. Nevertheless, the proportion of ratings falling between fair and excellent are above 80% for numerals and adverbs and 90% for the rest—see the bottom of Figure 3. We thus conclude that DAAM produces plausible maps for each interpretable part of speech.
One anticipated criticism is that different heat maps may explain the same word, making a qualitative comparison less meaningful. In Figure 4, “quickly” could conceivably explain “running” too. We concede to this, but our motivation is not to compare quality but rather to demonstrate plausibility. Without these experiments, the DAAM maps for words like “running” and “blue” could very well have been meaningless blotches.

heading: 4 Visuosyntactic Analysis
text: Equipped with DAAM, we now study how syntax relates to generated pixels. We characterize pairwise interactions between head–dependent DAAM maps, augmenting previous sections and helping to form hypotheses for further research.
Setup. We randomly sampled 1,000 prompts from COCO, performed dependency parsing with CoreNLP (Manning et al., 2014), and generated an image for each prompt and DAAM maps for all words. We constrained ourselves to the top-10 most common relations, resulting in 8,000 head– dependent pairs. Following Section 3.1, we then binarized the maps to quantify map pair interactions with set-based similarity statistics. We computed three statistics between the DAAM map of the head and that of the dependent: first, the mIoU, i.e., |A∩B||A∪B| ; second, the intersection over the dependent (mIoD; |A∩B||A| ); and third, the intersection over the head (mIoH; |A∩B||B| ). MIoU measures similarity, and the difference between mIoD and mIoH quantifies dominance. If mIoD > mIoH, then the head contains (dominates) the dependent more, and vice versa—see Appendix B for a visual tutorial.
Results. We present our results in Table 2 and Figure 5. We computed baseline overlap statistics for unrelated word pairs and all head–dependent pairs. Unsurprisingly, both baselines show moderate similarity and no dominance (43–48 mIoU, ∆ ≤ 1; rows 1–2). For syntactic relations, we observe no dominance for noun compounds (row 3), which is
expected since the two nouns complement one another (e.g., “ice cream”). Punctuation and articles (punct, det; rows 4 and 6) also lack dominance, possibly from having little semantic meaning and attending broadly across the image (Figure 5, top right). This resembles findings in Kovaleva et al. (2019), who note BERT’s (Devlin et al., 2019) punctuation to attend widely. For nouns joined with “and” (row 5), the maps overlap less (38.7 mIoU vs. 50+), likely due to visual separation (e.g., “cat and dog”). However, the overlap is still far above zero, which we attribute partially to feature entanglement, further explored in Section 5.1.
Starting at row 8, we arrive at pairs where one map dominates the other. A group in core arguments arises (nsubj, obj), where the head word dominates the noun subject’s or object’s map (12– 29-point ∆), perhaps since verbs contextualize both the subject and the object in its surroundings—see the middle of and bottom left of Fig. 5. We observe another group in nominal dependents (nmod:of, amod, acl), where nmod:of mostly points to collective nouns (e.g., “pile of oranges”), whose dominance is intuitive. In contrast, adjectival modifiers (amod) behave counterintuitively, where descriptive adjectives (dependents) visually dominate the nouns they modify (∆ ≈ 15). We instead expect objects to contain their attributes, but this is not the case. We again ascribe this to entanglement, elucidated in Section 5.2. Lastly, coreferent word pairs exhibit the highest overlap out of all relations (66.6 mIoU), indicating coreference resolution.

heading: 5 Visuosemantic Analyses

heading: 5.1 Cohyponym Entanglement
text: To further study the large nconj:and overlap found in Section 4, we hypothesize that semantically similar words in a prompt have worse generation quality, where only one of the words is generated in the image, not all. Setup. To test our hypothesis, we used WordNet (Miller, 1995) to construct a hierarchical ontology expressing semantic fields over COCO’s 80 visual objects, of which 28 have at least one other cohyponym across 16 distinct hypernyms (as listed in the appendix). Next, we used the prompt template, “a(n) <noun> and a(n) <noun>,” depicting two distinct things, to generate our dataset. Using our ontology, we randomly sampled two cohyponyms 50% of the time and two non-cohyponyms other times, producing 1,000 prompts from the template (e.g., “a giraffe and a zebra,” “a cake and a bus”). We generated an image for each prompt, then asked three unique annotators per image to select which objects were present, given the 28 words. We manually verified the image–label pairs, rejecting and republishing incorrect ones. Finally, we marked the overall label for each image as the top two most commonly picked nouns, ties broken by submission order. We considered generations correct if both words in the prompt were present in the image. For more setup details, see the appendix. Results. Overall, the non-cohyponym set attains a generation accuracy of 61% and the cohyponym set 52%, statistically significant at the 99% level according to the exact test, supporting our hypothesis. To see if DAAM assists in explaining these effects, we compute binarized DAAM maps (τ = 0.4, the best value from Sec. 3.1) for both words and quan-
tify the amount of overlap with IoU. We find that the mIoU for cohyponyms and non-cohyponyms are 46.7 and 22.9, suggesting entangled attention and composition. In the top of Figure 6, we further group the mIoU by cohyponym status and correctness, finding that incorrectness and cohyponymy independently increase the overlap. In the bottom subplot, we show that the amount of overlap (mIoU) differentiates correctness, with the low, mid, and high cutoff points set at ≤ 0.4, 0.4–0.6, and ≥ 0.6, following statistics in Section 4. We observe accuracy to be much better on pairs with low overlap (71.7–77.5%) than those with high overlap (9.8–36%). We present some example generations and maps in Figure 7, which supports our results.

heading: 5.2 Adjectival Entanglement
text: We examine prompts where a noun’s modifying adjective attends too broadly across the image. We start with an initial seed prompt of the form, “a <adj> <noun> <verb phrase>,” then vary the adjective to see how the image changes. If there is no entanglement, then the background should
not gain attributes pertaining to that adjective. To remove scene layout as a confounder, we fix all cross-attention maps to those of the seed prompt, which Hertz et al. (2022) show to equalize layout.
Our first case is, “a {rusty, metallic, wooden} shovel sitting in a clean shed,” “rusty” being the seed adjective. As shown in Figure 8, the DAAM map for “rusty” attends broadly, and the background for “rusty” is surely not clean. When we change the adjective to “metallic” and “wooden,” the shed changes along with it, becoming grey and wooden, indicating entanglement. Similar observations apply to our second case, “a {bumpy, smooth, spiky} ball rolling down a hill,” where “bumpy” produces rugged ground, “smooth” flatter ground, and “spiky” blades of grass. In our third case, we study color adjectives using “a {blue, green, red} car driving down the streets,” presented in Figure 9. We discover the same phenomena, with the difference that these prompts lead to quantifiable notions of adjectival entanglement. For, say, “green,” we can conceivably measure the amount of additional green hue in the background, with the car cropped out—see bottom row. A caveat is that entanglement is not necessarily unwanted; for instance, rusty shovels likely belong in rusted areas. It strongly depends on the use case of the model.

heading: 6 Related Work and Future Directions
text: The primary area of this work is in understanding neural networks from the perspective of computational linguistics, with the goal of better informing future research. A large body of relevant papers exists, where researchers apply textual perturbation (Wallace et al., 2019), attention visualization (Vig, 2019; Kovaleva et al., 2019; Shimaoka et al., 2016), and information bottlenecks (Jiang et al., 2020) to relate important input tokens to the outputs of large language models. Others explicitly test for linguistic constructs within models, such as probing vision transformers for verb understand-
ing (Hendricks and Nematzadeh, 2021) and examining visual grounding in image-to-text transformers (Ilinykh and Dobnik, 2022). Our distinction is that we carry out an attributive analysis in the space of generative diffusion models, as the pixel output relates to syntax and semantics. As a future extension, we plan to assess the unsupervised parsing ability of Stable Diffusion with syntactic– geometric probes, similar to Hewitt and Manning’s (2019) work in BERT.
The intersection of text-to-image generation and natural language processing is substantial. In the context of enhancing diffusion models with prompt engineering, Hertz et al. (2022) apply crossattention maps for the purpose of precision-editing generated images using text, and Woolf (2022) proposes negative prompts for removing undesirable, scene-wide attributes. Related as well are works for generative adversarial networks, where Karras et al. (2019) and Materzyńska et al. (2022) disentangle various features such as style and spelling. Along this vein, our work exposes more entanglement in cohyponyms and adjectives. A future line of work is how to disentangle such concepts and improve generative quality.
Last but not least are semantic segmentation works in computer vision. Generally, researchers start with a backbone encoder, attach decoders, and then optimize the model in its entirety end-to-end on a segmentation dataset (Cheng et al., 2022), unless the context is unsupervised, in which case one uses contrastive objectives and clustering (Cho et al., 2021; Hamilton et al., 2021). Toward this, DAAM could potentially provide encoder features in a segmentation pipeline, where its strong raw baseline numbers suggest the presence of valuable latent representations in Stable Diffusion.

heading: 7 Conclusions
text: In this paper, we study visuolinguistic phenomena in diffusion models by interpreting word–pixel cross-attention maps. We prove the correctness of our attribution method, DAAM, through a quantitative semantic segmentation task and a qualitative generalized attribution study. We apply DAAM to assess how syntactic relations translate to visual interactions, finding that certain maps of heads inappropriately subsume their dependents’. We use these findings to form hypotheses about feature entanglement, showing that cohyponyms are jumbled and adjectives attend too broadly.
Limitations
Our analysis has both methodological and technical limitations. While dependency parsers are the most robust semanto-syntactic tools available to us, we are limited both by the quality of the parser’s output and its paradigm. All automated tools make errors, and while our work uses short and simple phrases that are comparatively easy for these tools to handle, it is possible that even systematic errors could seep into the analysis. It is also possible that other semanto-syntactic tools would highlight different phenomena and improve (or worsen) the quality of the analysis.
Due to the dataset used, which we picked for quantitative comparison to prior art, there is an inherent bias towards concrete concepts, as they are derived from image captions. We are therefore limited in the understanding of how our method applies to more abstract concepts (say, “love” and “dignity”), potentially warranting further study.
There are also concerns about the internal validity of attention maps as an interpretability tool. For example, Serrano and Smith (2019) argue, “[In many cases,] gradient-based rankings of attention weights better predict [models’] effects than their magnitudes.” However, for the analysis of diffusion models, gradient methods are intractable because a backpropagation pass is required for every pixel for all time steps, as stated in Section 2.2. Therefore, attention scores remain the most feasible method.
Lastly, we have consciously limited ourselves to purely making analytical observations regarding attribution and entanglement. This has arguably allowed us to cover a very wide range of phenomena and make a large number of observations, but this choice naturally limits us to not providing a method to resolve the issues we have observed with existing models, which is something we have left (and described in Section 6) as future work.

heading: Acknowledgments
text: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, companies sponsoring the Vector Institute, and the HuggingFace team. In particular, we would like to thank Aleksandra (Ola) Piktus, who helped us get a community grant for our public demonstration on HuggingFace spaces.

heading: A Supplements for Attribution Analyses
text: A.1 Object Attribution
Generation setup. For all images, we ran the Stable Diffusion 2.0 base model (512 by 512 pixels) with 30 inference steps, the default 7.5 classifier guidance score, and the state-of-the-art DPM solver. We automatically filtered out all offensive images, against which the 2.0 model has both training-time and after-inference protection. We also steered clear of offensive prompts, which were absent to start with in COCO. Our computational environment consisted of PyTorch 1.11.0 and CUDA 11.4, running on Titan RTX and A6000 graphics cards. Our spaCy model was en_core_web_md.
Segmentation process. To draw the ground-truth segmentation masks, we used the object selection tool, the quick selection tool, and the brush from Adobe Photoshop CC 2022 to fill in a black mask for each area corresponding to a present noun. We then exported each mask (without the background image) as a binary PNG mask and attached it to the relevant noun—see Figure 10 for some examples. Two trained annotators worked on the total set of 1000 image–prompt pairs, with one completing 180 on each dataset and the other 320 on each.
Layer and time step ablation. We conducted ablation studies to see if summing across all time steps and layers, as in Eqn. 6, is necessary. We searched both sides of the summation: for one study, we restricted DAAM to j ≤ j∗, as j∗ = 1 → T ; for its dual study, we constrained j ≥ j∗. We applied the same methods to layer resolution,
i.e., ci. We present our results in Fig. 11, which suggests that all time steps and layers contribute positively to segmentation quality.
Dice score and pixel accuracy. We also ran COCO-Gen experiments with pixel accuracy and dice score metrics, two less common ones in the segmentation literature. In terms of pixel accuracy, CLIPSeg attained 90%, DAAM-0.4 90%, and Mask2former 85%, which largely agrees with our mIoU∞ findings. We conjecture that DAAM and Mask2former improve against CLIPSeg because pixel accuracy penalizes outliers less. For dice score, CLIPSeg achieved 72, DAAM-0.4 68, and Mask2former 30, which also agrees with our mIoU∞ results. We conclude that, in addition to mIoU, both metrics support the use of DAAM.
A.2 Generalized Attribution
Annotation process. We designed our annotation UIs for Amazon MTurk, a popular crowdsourcing platform, where we submitted jobs requiring three unique annotators at the master level to complete each task. We presented the UI pictured in Figure 12, asking them to rate the relevance of the red word to the highlighted area in the image. If the image was too poor or if the word was missing, they could also choose options 6 and 7.
To filter out low-quality or inattentive annotators, we randomly asked workers to interpret punctuation, such as periods. Since these tokens are self-evidently too abstract and missing in the image, we removed workers who didn’t select one of those two options. However, we found overall attention to be high, having a reject rate of less than 2% of the tasks, consistent with Hauser and Schwarz’s (2016) findings that MTurk users outperform subject pool participants. We show response statistics in Figure 13, where adpositions, coordinating conjunctions, participles, punctuation, and articles have high non-interpretable rates.
Preliminary CLIPSeg comparison. We briefly conducted additional experiments within Section 3.2 using CLIPSeg to compare its attribution ability to DAAM. We find that DAAM significantly (p < 0.02; unpaired t-test) outperforms CLIPSeg on verbs, proper nouns, and adverbs, because CLIPSeg was unable to produce viable maps. No significant differences were noted on nouns and adjectives, which CLIPSeg can segment. Overall, DAAM outperforms CLIPSeg by 0.9 MOS points (3.4 vs 2.5). We conclude that, while CLIPSeg is plausible for some parts-of-speech, such as nouns, it is implausible for others.

heading: B Supplements for Syntactic Analyses
text: Measures of overlap. We use three measures of overlap to characterize head–dependent map interactions: mean intersection over union (mIoU), intersection over the dependent (mIoD), and intersection over the head (mIoH). When mIoU is high, the maps overlap greatly; when mIoD is high but mIoH is low, the head map occupies more of the dependent than the dependent does the head; when the opposite is true, the dependent occupies more.
Concretely, given a sequence of binarized DAAM map pairs {(DIτ(i1), D Iτ (i2))}ni=1, where i1 are dependent indices and i2 head indices, we compute mIoU as
1
n
n∑
i=1
∑ (x,y)D Iτ (i1)[x, y] ∧D
Iτ (i2)[x, y]∑
(x,y)D Iτ (i1)[x, y] ∨D Iτ (i2)[x, y]
, (8)
where ∧ is the logical-and operator, returning 1 if both sides are 1, 0 otherwise, and ∨ the logicalor operator, returning 1 if at least one operand is 1, and 0 otherwise. Let the top part of the inner fraction be the intersection, or INT for short. Define mIoD as
1
n
n∑
i=1
INT∑ (x,y)D Iτ (i1)[x, y] , (9)
and mIoH as
1
n
n∑
i=1
INT∑ (x,y)D Iτ (i2)[x, y] , (10)
We visually present our mIoD and mIoH statistics in Figure 14.
To compute mIoU∞, we compute mIoU (Eqn. 8) without restricting ourselves to the typical 80 COCO classes. For mIoU80, we only look at objects with one of those labels.

heading: C Supplements for Semantic Analyses
text: Semantic relation ontology. We present our relation ontology below, continued on the next page:
[ROOT] [BAG]
backpack handbag suitcase
[FOOD] [BAKED GOODS]
cake donut
[DISH] hot dog pizza sandwich
[FRUIT] apple banana orange
[ELECTRICAL DEVICE] [APPLIANCE]
oven refrigerator toaster
[MONITOR DEVICE] cell phone laptop tv
[FURNITURE] bench chair couch
[MAMMAL] [FARM ANIMAL]
cow horse sheep
[PETS] cat dog
[WILD ANIMAL] bear elephant giraffe zebra
Cohyponym annotation process. Similar to the generalized attribution annotation process, we designed our UIs for Amazon MTurk. We submitted a job requiring three unique annotators at the master level to complete each task. We presented to them the UI shown in Figure 15. We manually verified each response, removing workers whose quality was consistently poor. This included workers who didn’t include all objects generated. Overall, the worker quality was exceptional, with a reject rate below 2%. Out of a pool of 30 workers, no single worker annotated more than 16% of the examples.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
After conclusions
7 A2. Did you discuss any potential risks of your work? This approach does the opposite and helps to expose risks in large-scale diffusion models.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2, 3, 4, A.1
3 B1. Did you cite the creators of artifacts you used? 2, 3, 4, A.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 2
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3, 4, 5
C 3 Did you run computational experiments? 3, 4, 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Error bars in Sec 4, significance tests throughout 3-5.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Appendix A.2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A.2
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A.2
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? It wasn’t necessary due to the simplicity of the task.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? It was determined exempt following research ethics board approval procedure: https://uwaterloo. ca/research/sites/ca.research/files/uploads/files/research_or_quality_ assurance_decision_tree.pdf
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? It wasn’t available.

heading: Future Work
text: This has arguably allowed us to cover a very wide range of phenomena and make a large number of observations, but this choice naturally limits us to not providing a method to resolve the issues we have observed with existing models, which is something we have left (and described in Section 6) as future work