SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where each  $\operatorname{E}(\boldsymbol{x} \mid \boldsymbol{c}_i)$  represent independently trained EBM with concept code  $c_i$ . While it is an appealing solution to the controllable generation, it is notoriously difficult to train EBMs [9, 27] in a way to make themselves scalable to high-resolution image generation. Instead of directly training EBMs in pixel space, we leverage the cross-attention space EBMs and the generative power of state-of-the-art pre-trained DMs to achieve high-fidelity compositional generations.
More specifically, assume that we have main context vectors  $C_1$  embedded from a main prompt, e.g. "a castle next to a river", and a set of independent editing context vectors,  $C = \{C_2, \ldots, C_M\}$ , each embedded from different editorial prompt, e.g. "monet style", "boat on a river", etc. Then, we define the keys  $K_{l,s}$  for context  $s$  within a cross-attention layer of index  $l$  as  $K_{l,s} = C_s W_{K,l}, \forall s \in \{1, 2, \ldots, M\}$ . The index  $l$  will be omitted for notational simplicity. Then, for a given key  $K_s$ , we consider the energy function in cross-attention space w.r.t queries  $Q$ :
$$
\operatorname {E} \left(\boldsymbol {Q}; \boldsymbol {K} _ {s}\right) = \frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {Q} \boldsymbol {Q} ^ {T}\right) - \sum_ {i = 1} ^ {P ^ {2}} \log \operatorname {s u m e x p} \left(\boldsymbol {K} _ {s} \boldsymbol {q} _ {i} ^ {T}, \beta\right), \tag {14}
$$
which is directly motivated by (3). We then introduce the compositional energy function  $\hat{\mathbf{E}}$ , for the concept conjunction of  $C$  as in (13) and the updated rule as done in (6):
$$
\begin{array}{l} \hat {\mathrm {E}} (\boldsymbol {Q}; \left\{\boldsymbol {K} _ {s} \right\} _ {s = 1} ^ {M}) = \frac {1}{M} \sum_ {s = 1} ^ {M} \mathrm {E} (\boldsymbol {Q}; \boldsymbol {K} _ {s}) (15) \\ = \frac {1}{2} \operatorname {d i a g} (\boldsymbol {Q} \boldsymbol {Q} ^ {T}) - \frac {1}{M} \sum_ {s = 1} ^ {M} \sum_ {i = 1} ^ {P ^ {2}} \log \operatorname {s u m e x p} (\boldsymbol {K} _ {s} \boldsymbol {q} _ {i} ^ {T}, \beta), \\ \boldsymbol {Q} _ {T F} ^ {\text {n e w}} = \frac {1}{M} \sum_ {s = 1} ^ {M} \alpha_ {s} \operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {Q} _ {T F} \boldsymbol {K} _ {T F, s} ^ {T}\right) \boldsymbol {V} _ {T F}, (16) \\ \end{array}
$$
where  $Q_{TF}, K_{TF,s}$  and  $V_{TF}$  directly follows the definition in (6) and the degree and direction of  $s$ -th composition can be controlled for each concept individually by setting the scalar  $\alpha_s$ , with  $\alpha_s < 0$  for concept negation [8]. Note that this is exactly a linear combination of transformer cross-attention outputs from different contexts with  $\beta = 1 / \sqrt{d_H}$ . We refer to this process as the Energy-based Composition of Queries (EBCQ).
Figure 3: Energy analysis for multi-concept image generation. The right graph displays  $E(Q;K)$  across 16 cross-attention layers within each sampling step. The red and green lines correspond to the Stable Diffusion and ours, respectively. Mean and standard deviation are calculated and displayed. The bottom plot shows the cumulative sum of the posterior energy difference between Stable Diffusion and the proposed method, and intermediate denoised estimates are displayed together.
This update rule implies that the compositional energy can be minimized implicitly through modifications to the forward path of EBCA, thereby guiding the integration of semantic information conveyed by editorial prompts. Notably, this is a significant departure from the existing energy-based compositional methods [8, 26, 9]. Specifically, no training or fine-tuning is required. Instead, cross-attention outputs of the main and editorial prompts are simply obtained in parallel, averaged, and propagated to the next layer. Moreover, the introduction of EBMs in cross-attention space is orthogonal to [22], which conducts compositional generation by treating a pre-trained diffusion model  $\epsilon_{\theta^{*}}$  itself as an implicitly parameterized EBM.
To verify our claim of energy minimization via modified cross-attention, we conduct various experiments in text-guided image generation to verify semantic alignment, namely (1) multi-concept generation [22, 11], (2) text-guided image inpainting [23, 41], and (3) compositional generation [12, 1, 28] which includes real and synthetic image editing. In this work, while the first and third applications address similar challenges, we categorize them separately based on the implementation derived from the energy-based interpretation. Specifically, the Energy-based Bayesian Context Update (EBCU) is applied to every task, and the Energy-based Composition of Queries (EBCQ) is additionally leveraged in the third task. Note that all applications have been done without additional training.
Setup. The proposed framework can be widely mounted into many state-of-the-art text-to-image DMs due to its unique functionality of context updates and cross-attention map compositions. Here, we verify the effectiveness of energy-based cross-attention with Stable Diffusion (SD) [31]. The SD is an LDM that is pre-trained on a subset of the large-scale image-language pair dataset, LAION-5B [34] followed by the subsequent fine-tuning on the LAION-aesthetic dataset. For the text embedding, we use a pre-trained CLIP [29] model following theImagen [33]. The pre-trained SD is under the creativeML OpenRAIL license. Detailed experimental settings are provided in the appendix.
4.1 Analysis on Energy during the sampling
We perform a comprehensive analysis on (8) and (9) during the forward path through the modified cross-attention, which offers insights into the energy behavior for real applications. Specifically, we examine the energy dynamics involved in the multi-concept image generation that is straightforward and could be readily applied to other applications with minimal effort. We record computed energy values along layers and sampling steps 30 times with a fixed text prompt and then display them in Figure 3 with the generated samples, where red and green lines denote the energy involved with the Stable-Diffusion (SD) and the proposed method, respectively. For each sampling step block that is alternately shaded, the energy values are plotted through 16 cross-attention layers.
Across all layers and sampling steps, the energy associated with the proposed method remains consistently lower than that of the SD. This is in line with the semantic alignment between intermediate denoised estimates and the given text prompt. In both cases of the SD and the proposed method,  $E(Q;K)$  decreases over sampling steps. This implies that the iterative refinement of the updated query carried over to subsequent sampling steps, resulting in a better match to the given context. Note that the proposed method even achieves lower energy with the EBCU. More analyses are provided in the appendix E including the ablation study.
Figure 4: Multiconcept generation comparison. Each row indicates the Stable Diffusion, the Structured Diffusion, and the proposed method applied to the Stable Diffusion, respectively. Generated samples in the same column used the same text prompt and the same random seed.
Figure 5: Text-guided inpainting comparison. The masked region is conditionally generated by given text-prompt positioned on the left-end of each row.
4.2 Multi-Concept Generation
We empirically demonstrate that the proposed framework alleviates the catastrophic neglect and attribute binding problems defined in the existing literature [4]. As shown in Figures 3 and 4, the EBCU effectively mitigates these problems by promoting attention to all relevant concepts in the prompt. Specifically, the regularization term introduced by the prior energy prevents a single context from dominating the attention, while the attention term updates the context vectors in the direction of input queries, improving semantic alignment and facilitating the retrieval of related information.
4.3 Text-guided Image Inpainting
In addition, we have evaluated the efficacy of the proposed energy-based EBCU on a text-guided inpainting task. Although existing state-of-the-art DMs such as DALLE and SD can be employed for inpainting by exploiting the ground-truth unmasked area [23], they usually require computationally expensive fine-tuning and tailored data augmentation strategies [23, 41]. In contrast, as shown in Figure 5, our proposed energy-based framework significantly enhances the quality of inpainting without any fine-tuning. Specifically, we incorporate the energy-based cross-attention into the StableRepaint (SR) and Stable-Inpaint (SI) models, which can struggle to inpaint multiple objects (e.g., birthday hat and bow tie) or unlikely combinations of foreground and background objects (e.g., Teddy bear on the Starry Night painting). In contrast, the proposed approach accurately fills in semantically relevant objects within the target mask region.
4.4 Compositional Generation
We demonstrate that the proposed framework improves the controllability and compositionality of DMs, which is a significant challenge for generative models. To assess this, we split the task into synthetic and real image editing.
Figure 6: Image editing comparison. The left three columns show examples of synthetic image editing, while the right four columns show examples of real image editing. To ensure a fair comparison, the same editing prompt, positioned on the left-end of each row, is given to all methods being compared.
Figure 7: Real image editing with multiple editing prompts. Given a real image at the left-end of each row, the proposed method allows robust image editing with different prompts over each sample.
Synthetic image editing. Text-to-image DMs provide impressive results, but it is difficult to generate images perfectly aligned with user intentions [1]. Although modifying prompts can guide generation, it can also introduce unintended changes in the generated content. Our results, shown in Figure 5, demonstrate that the proposed method can edit image contents while maintaining the original-prompt-related identity of the generated images thanks to the EBCU, which continuously harmonizes the latent representations and text prompts. While composable-diff [22] can generate compositions, they often fail to preserve the original properties of images, such as the color in the second row, or fail to compose the desired concept, such as the boat in the first row.
Real image editing. We demonstrate that the proposed framework can also edit real images, which is especially challenging as existing methods often dramatically alter input content or introduce unexpected variations. To do this, we integrate our framework with DDIM inversion [35, 25], which inverts images with meaningful text prompts into the domain of pre-trained diffusion models. First, we use the image captioning network BLIP [20] to automatically caption the interested image, following [28]. Next, we obtain the inverted noise latents of the input image using Diffusion Pivotal Inversion [25]. Then, we apply EBCQ and EBCU while denoising the inverted latents. The optimized unconditional textual embedding vector is also targeted for additional EBCUs. The results in Figure 6 demonstrate that our method achieves better editing performance by avoiding undesired changes.
Quantitative comparisons. We further conducted a comparative analysis of the proposed framework against several state-of-the-art diffusion-based image editing methods [24, 12, 25, 28]. For our evaluations, we focus on two image-to-image translation tasks. The first one is an animal transition task, which translates (1) cat  $\rightarrow$  dog, (2) horse  $\rightarrow$  zebra, and (3) adding glasses to cat input images (cat  $\rightarrow$  cat with glasses). The second one is a human transition task, which translates (1) woman  $\rightarrow$  man, (2) woman  $\rightarrow$  woman with glasses, and (3) woman  $\rightarrow$  man with glasses. Source images are retrieved from the LAION 5B dataset [34] and CelebA-HQ [15]. Motivated by [39, 28], we measure CLIP Accuracy and DINO-ViT structure distance. Experimental details are provided in the appendix E including explanations of baselines, datasets, metrics, and hyperparameter configurations.
Table 1: (Animal transition) Comparison to state-of-the-art diffusion-based editing methods. Dist for DINO-ViT Structure distance. Baseline results are from [28].
Method	(a) Cat → Dog	(b) Horse → Zebra	(c) Cat → Cat w/ glasses
CLIP Acc (↑)	Dist (↓)	CLIP Acc (↑)	Dist (↓)	CLIP Acc (↑)	Dist (↓)
SDEdit [24] + word swap	71.2%	0.081	92.2%	0.105	34.0%	0.082
DDIM + word swap	72.0%	0.087	94.0%	0.123	37.6%	0.085
prompt-to-prompt [12]	66.0%	0.080	18.4%	0.095	69.6%	0.081
pix2pix-zero [28]	92.4%	0.044	75.2%	0.066	71.2%	0.028
Stable Diffusion + ours	93.7%	0.040	90.4%	0.061	81.1%	0.052
Figure 8: Real image editing on (a) animal transition and (b) human transition tasks.
Table 1 and 2 in the appendix show that the proposed energy-based framework gets a high CLIP-Acc while having low Structure Dist. It implies that the proposed framework can perform the best edit while still retaining the structure of the original input image. While DDIM + word swap records remarkably high CLIP-Acc in horse  $\rightarrow$  zebra task, Figure 8 shows that such improvements are based on unintended changes in the overall structure. More visualizations are provided in the appendix E.
5 Conclusion, Limitations and Societal Impacts
Conclusion. In this work, we formulated the cross-attention with an energy perspective and proposed the EBCU, a modified cross-attention that could implicitly minimize the energy in the latent space. Furthermore, we proposed EBCQ which is inspired by energy-based formulation of multiple text composition. The proposed method is versatile as shown by multiple applications, theory-grounded, easy to implement, and computationally almost free.
Limitations and Societal Impacts. The framework presented in this study generates images based on user intentions, which raises concerns regarding potential misuse for creating deepfakes or other forms of disinformation. It is crucial to ensure that these methods are implemented ethically and regulated appropriately. Additionally, while the framework performs well across various tasks, it requires pre-trained deep models, rendering it challenging to apply to out-of-domain datasets, such as medical images.
This research was supported by the KAIST Key Research Institute (Interdisciplinary Research Group) Project, National Research Foundation of Korea under Grant NRF-2020R1A2B5B03001980, Field-oriented Technology Development Project for Customs Administration through National Research Foundation of Korea(NRF) funded by the Ministry of Science & ICT and Korea Customs Service(**NRF-2021M3I1A1097938**), Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711137899, KMDF_PR_20200901_0015), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2021-0-02068, Artificial Intelligence Innovation Hub), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2023-00233251, System3 reinforcement learning with high-level brain functions), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2019M3E5D2A01066267).
[1] M. Brack, F. Friedrich, D. Hintersdorf, L. Struppek, P. Schramowski, and K. Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprint arXiv:2301.12247, 2023.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[4] H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826, 2023.
[5] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188-8197, 2020.
[6] M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and F. Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-299, 2017.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[8] Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
[9] Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020.
[10] Y. Du and I. Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689, 2019.
[11] W. Feng, X. He, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022.
[12] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[13] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[34] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[35] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[36] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[37] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[38] N. Tumanyan, O. Bar-Tal, S. Bagon, and T. Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10748-10757, 2022.
[39] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572, 2022.
[40] C. H. Wu and F. De la Torre. Unifying diffusion models' latent space, with applications to cyclediffusion and guidance. arXiv preprint arXiv:2210.05559, 2022.
[41] S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.05034, 2022.
[42] J. C. Ye. Geometry of Deep Learning. Springer, 2022.
[43] A. L. Yuille and A. Rangarajan. The concave-convex procedure (cccp). Advances in neural information processing systems, 14, 2001.
In computing a derivative of a scalar, vector, or matrix with respect to a scalar, vector, or matrix, we should be consistent with the notation. In this paper, we follow the denominator layout convention as described in [42]. To make the paper self-contained, we briefly introduce the denominator layout and associate calculus.
The main motivation for using the denominator layout is from the derivative with respect to the matrix. More specifically, for a given scalar  $c$  and a matrix  $\mathbf{W} \in \mathbb{R}^{m \times n}$ , according to the denominator layout, we have
$$
\frac {\partial c}{\partial \boldsymbol {W}} = \left[ \begin{array}{c c c} \frac {\partial c}{\partial w _ {1 1}} & \dots & \frac {\partial c}{\partial w _ {1 n}} \\ \vdots & \ddots & \vdots \\ \frac {\partial c}{\partial w _ {m 1}} & \dots & \frac {\partial c}{\partial w _ {m n}} \end{array} \right] \in \mathbb {R} ^ {m \times n}. \tag {17}
$$
Furthermore, this notation leads to the following familiar result:
$$
\frac {\partial \boldsymbol {a} ^ {\top} \boldsymbol {x}}{\partial \boldsymbol {x}} = \frac {\partial \boldsymbol {x} ^ {\top} \boldsymbol {a}}{\partial \boldsymbol {x}} = \boldsymbol {a}. \tag {18}
$$
Accordingly, for a given scalar  $c$  and a matrix  $\mathbf{W} \in \mathbb{R}^{m \times n}$ , we can show that
$$
\frac {\partial c}{\partial \boldsymbol {W}} := \operatorname {U N V E C} \left(\frac {\partial c}{\partial \operatorname {V E C} (\boldsymbol {W})}\right) \in \mathbb {R} ^ {m \times n}, \tag {19}
$$
in order to be consistent with (17), where  $\mathrm{VEC}()$  and  $\mathrm{UNVEC}()$  refer to the vectorization operation and its reverse, respectively. Under the denominator layout notation, for given vectors  $\pmb{x} \in \mathbb{R}^m$  and  $\pmb{y} \in \mathbb{R}^n$ , the derivative of a vector with respect to a vector given by
$$
\frac {\partial \boldsymbol {y}}{\partial \boldsymbol {x}} = \left[ \begin{array}{c c c} \frac {\partial y _ {1}}{\partial x _ {1}} & \dots & \frac {\partial y _ {n}}{\partial x _ {1}} \\ \vdots & \ddots & \vdots \\ \frac {\partial y _ {1}}{\partial x _ {m}} & \dots & \frac {\partial y _ {n}}{\partial x _ {m}} \end{array} \right] \in \mathbb {R} ^ {m \times n}. \tag {20}
$$
Then, the chain rule can be specified as follows:
$$
\frac {\partial c (\boldsymbol {g} (\boldsymbol {u}))}{\partial \boldsymbol {x}} = \frac {\partial \boldsymbol {u}}{\partial \boldsymbol {x}} \frac {\partial \boldsymbol {g} (\boldsymbol {u})}{\partial \boldsymbol {u}} \frac {\partial c (\boldsymbol {g})}{\partial \boldsymbol {g}}. \tag {21}
$$
Finally, the following property of the Kronecker delta product is useful throughout the paper [42]
$$
\operatorname {V E C} (\boldsymbol {A} \boldsymbol {B} \boldsymbol {C}) = \left(\boldsymbol {C} ^ {T} \otimes \boldsymbol {A}\right) \operatorname {V E C} (\boldsymbol {B}) \tag {22}
$$
$$
\left(\boldsymbol {A} \otimes \boldsymbol {B}\right) ^ {T} = \boldsymbol {A} ^ {T} \otimes \boldsymbol {B} ^ {T} \tag {23}
$$
Using this, we first prove the following key lemmas.
Lemma 1. For a given column vector  $\pmb{x} \in \mathbb{R}^N$ , we have
$$
\frac {\partial \operatorname {l o g s u m e x p} (\boldsymbol {x} , \beta)}{\partial \boldsymbol {x}} = \operatorname {s o f t m a x} (\beta \boldsymbol {x})
$$
$$
\begin{array}{l} \frac {\partial \operatorname {l o g s u m e x p} (\boldsymbol {x} , \beta)}{\partial \boldsymbol {x}} = \beta^ {- 1} \frac {\partial \log \sum_ {j = 1} ^ {N} \exp (\beta x _ {j})}{\partial \boldsymbol {x}} = \left[ \begin{array}{c} \frac {\exp (\beta x _ {1})}{\sum_ {j = 1} ^ {N} \exp (\beta x _ {j})} \\ \vdots \\ \frac {\exp (\beta x _ {P 2})}{\sum_ {j = 1} ^ {N} \exp (\beta x _ {j})} \end{array} \right] \\ = \operatorname {s o f t m a x} (\beta \boldsymbol {x}) \\ \end{array}
$$
Lemma 2. Let  $\pmb{k}_i$  denote the  $i$ -th row vector of  $\pmb{K} \in \mathbb{R}^{N \times d}$ . Then, we have
$$
\frac {\partial \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}}{\partial \boldsymbol {K}} = 2 \boldsymbol {e} _ {N} ^ {i} (\boldsymbol {e} _ {N} ^ {i}) ^ {T} \boldsymbol {K}
$$
where  $\mathbf{e}_N^i$  represents a  $N$ -dimensional column vector where only the  $i$ -th entry is 1, with all other entries set to zero.
Proof. First, note that the expression  $\pmb{k}_i^T$  can be equivalently rewritten as  $\pmb{K}^T\pmb{e}^i$ . Then, we have