SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Geon Yeong Park $^{1*}$  Jeongsol Kim $^{1*}$  Beomsu Kim $^{2}$  Sang Wan Lee $^{1,2,3\dagger}$  Jong Chul Ye $^{2\dagger}$
$^{1}$ Bio and Brain Engineering,  $^{2}$ Kim Jaechul Graduate School of AI,  $^{3}$ Brain and Cognitive Sciences
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea
*, †: Co-first and Co-corresponding authors
{pky3436, jeongsol, beomsu.kim, sangwan,jong.ye}@kaist.ac.kr
Despite the remarkable performance of text-to-image diffusion models in image generation tasks, recent studies have raised the issue that generated images sometimes cannot capture the intended semantic contents of the text prompts, which phenomenon is often called semantic misalignment. To address this, here we present a novel energy-based model (EBM) framework for adaptive context control by modeling the posterior of context vectors. Specifically, we first formulate EBMs of latent image representations and text embeddings in each cross-attention layer of the denoising autoencoder. Then, we obtain the gradient of the log posterior of context vectors, which can be updated and transferred to the subsequent cross-attention layer, thereby implicitly minimizing a nested hierarchy of energy functions. Our latent EBMs further allow zero-shot compositional generation as a linear combination of cross-attention outputs from different contexts. Using extensive experiments, we demonstrate that the proposed method is highly effective in handling various image generation tasks, including multi-concept generation, text-guided image inpainting, and real and synthetic image editing. Code: https://github.com/EnergyAttention/Energy-Based-CrossAttention.
Diffusion models (DMs) have made significant advances in controllable multi-modal generation tasks [37], including text-to-image synthesis. The recent success of text-to-image diffusion models, e.g., Stable Diffusion [31], Imagen [33], etc., is mainly attributed to the combination of high-fidelity DMs with high-performance large language models (LMs).
Although text-to-image DMs have shown revolutionary progress, recent studies have shown that the current state-of-the-art models often suffer from semantic misalignment problems, where the generated images do not accurately represent the intended semantic contents of the text prompts. For example, [4] discovered the catastrophic neglect problem, where one or more of the concepts of the prompt are neglected in generated images. Moreover, for a multi-modal inpainting task with text and mask guidance, [41] found that the text-to-image DMs may often fail to fill in the masked region precisely following the text prompt.
Therefore, this work focuses on obtaining a harmonized pair of latent image representations and text embeddings, i.e., context vectors, to generate semantically aligned images. In order to mitigate the misalignment, instead of leveraging fixed context vectors, we aim to establish an adaptive context by modeling the posterior of the context, i.e.  $p(\text{context} \mid \text{representations})$ . Note that this is a significant departure from the previous methods which only model  $p(\text{representations} \mid \text{context})$  with frozen context vectors encoded by the pretrained textual encoder.
(A) Multi-concept
(B) Inpainting
"A corgi wearing a bow tie and a birthday hat"
(C) Synthetic Image Editing
"A teddy bear"
"A castle next to a river" (+) Monet, Boat
"A photo of an woman" (+) smile (-) curls
(D) Real Image Editing
Figure 1: The energy-based cross-attention improves the semantic alignment between given text and the generated sample. The proposed method could be leveraged for multiple applications without additional training.
Specifically, we introduce a novel energy-based Bayesian framework, namely energy-based cross-attention (EBCA), which approximates maximum a posteriori probability (MAP) estimates of context vectors given observed latent representations. Specifically, to model  $p(\text{context} \mid \text{representations})$ , we first consider analogous  $p(K_l|Q_{t,l})$  in the latent space of the intermediate cross-attention layer of a denoising auto-encoder, i.e., cross-attention space. Here,  $K_l$  and  $Q_{t,l}$  correspond to the key and query in  $l$ -th layer at time  $t$  that encode the context from the text and the image representation, respectively. Inspired by the energy-based perspective of attention mechanism [30], we then formulate energy functions  $E(K_l; Q_{t,l})$  in each cross-attention space to model the posterior. Finally, we create a correspondence between the context and representation by minimizing these parameterized energy functions. More specifically, by obtaining the gradient of the log posterior of context vectors, a nested hierarchy of energy functions can be implicitly minimized by cascading the updated contexts to the subsequent cross-attention layer (Figure 2).
Moreover, our energy-based perspective of cross-attention also allows zero-shot compositional generation due to the inherent compositionality of Energy-Based Models (EBMs). This involves the convenient integration of multiple distributions, each defined by the energy function of a specific text embedding. In practical terms, this amalgamation can be executed as a straightforward linear combination of cross-attention outputs that correspond to all selected editing prompts.
We demonstrate the effectiveness of the proposed EBM framework in various text-to-image generative scenarios, including multi-concept generation, text-guided inpainting, and compositional generation. The proposed method is training-free, easy to implement, and can potentially be integrated into most of the existing text-to-image DMs.
2.1 Diffusion Model
Diffusion models [36, 13, 37, 18, 16] aims to generate samples from the Gaussian noise by iterative denoising processes. Given clean data  $\pmb{x}_0 \sim p_{data}(\pmb{x}_0)$ , diffusion models define the forward sampling from  $p(\pmb{x}_t | \pmb{x}_0)$  as  $\pmb{x}_t = \sqrt{\bar{\alpha}_t} \pmb{x}_0 + \sqrt{1 - \bar{\alpha}_t} \pmb{z}_t$ , where  $\pmb{z}_t \sim \mathcal{N}(0, I)$ ,  $t \in [0, 1]$ . Here, for the Denoising Diffusion Probabilistic Model (DDPM) [13], the noise schedule  $\beta_t$  is an increasing sequence of  $t$ , with  $\bar{\alpha}_t := \prod_{i=1}^t \alpha_t$ ,  $\alpha_t := 1 - \beta_t$ . The goal of diffusion model training is to obtain a neural network  $\epsilon_{\theta^*}$  that satisfies
$$
\theta^ {*} = \underset {\theta} {\operatorname {a r g m i n}} \mathrm {E} _ {\boldsymbol {x} _ {t} \sim p (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0}), \boldsymbol {x} _ {0} \sim p _ {\text {d a t a}} (\boldsymbol {x} _ {0}), \epsilon \sim \mathcal {N} (0, \mathbf {I})} \left[ \| \epsilon_ {\theta} (\boldsymbol {x} _ {t}, t) - \boldsymbol {\epsilon} \| _ {2} ^ {2} \right], \tag {1}
$$
Figure 2: Comparison between the Stable-Diffusion and the proposed method. (A) The Stable-Diffusion uses fixed context embedding encoded by pre-trained CLIP. (B) The proposed method allows adaptive context embedding through energy-based context update (EBCU) and energy-based composition of queries (EBCQ).
so that the reverse sampling from  $q(\pmb{x}_{t-1}|\pmb{x}_t, \pmb{x}_{\theta^*}(\pmb{x}_t, t))$  is achieved by
$$
\boldsymbol {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {x} _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta^ {*}} (\boldsymbol {x} _ {t}, t)\right) + \sigma_ {t} \boldsymbol {z} _ {t}, \tag {2}
$$
where  $z_{t} \sim \mathcal{N}(0, I)$  and  $\sigma^2$  is variance which is set as  $\beta$  in DDPM. With iterative process, we can sample the image  $x_0$  from initial sample  $x_{T} \sim \mathcal{N}(0, I)$ .
Since diffusion models require the iterative sampling on high dimensional space, they are computationally expansive and time consuming. To mitigate this limitation, Latent Diffusion Model (LDM) [31] has proposed diffusion processes on the compressed latent space using pre-trained auto-encoder. Furthermore, by introducing language model-based cross-attention to diffusion model's U-Net neural backbone [32], LDM enables token-based conditioning method such as text-to-image.
2.2 Energy-based Perspective of Attention Mechanism
Definition. Given an  $N$ -dimensional vector  $\pmb{v}$ , its logsumexp and softmax are defined as
$\mathrm{logsumexp}(\pmb {v},\beta)\coloneqq \beta^{-1}\log (\sum_{i = 1}^{N}\exp (v_i)),\quad \mathrm{softmax}(\pmb {v})\coloneqq \exp (\pmb {v} - \mathrm{logsumexp}(\pmb {v},1)).$
where  $v_{i}$  denotes the  $i$ -th element of  $\mathbf{v}$ . For a given a matrix  $\mathbf{A}$ , its logsumexp $_i(\mathbf{A})$  and softmax $_i(\mathbf{A})$  are understood as taking the corresponding operation along the  $i$ -th dimension of  $\mathbf{A}$ . So, for instance,  $i = 1$ , softmax $_i(\mathbf{A})$  consists of column vectors that sum to 1.
Energy model. Following the success of the attention mechanism, several studies have focused on establishing its theoretical foundation. One promising avenue is interpreting the attention mechanism using the Energy-Based Model (EBM) framework. This line of research begins with recent research on modern Hopfield networks [30, 14], which gradually builds up to the self-attention mechanism of the Transformer model.
The Hopfield network is a dense associative memory model that aims to associate an input with its most similar pattern. Specifically, it constructs an energy function to model an energy landscape that contains basins of attraction around desired patterns. Recently, modern Hopfield networks [30, 6, 19] has introduced a new family of energy functions, which aim to improve its pattern storage capacity or make it compatible with continuous embeddings. To this end, [30] proposed the following energy function of a state pattern (query)  $\zeta \in \mathbb{R}^d$  parameterized by  $N$ -stored (key) patterns  $\pmb{X} = [\pmb{x}_1 \dots, \pmb{x}_N] \in \mathbb{R}^{d \times N}$  and  $\beta > 0$ :
$$
E (\boldsymbol {\zeta}; \boldsymbol {X}) = \frac {1}{2} \boldsymbol {\zeta} ^ {T} \boldsymbol {\zeta} - \operatorname {l o g s u m e x p} \left(\boldsymbol {X} ^ {T} \boldsymbol {\zeta}, \beta\right). \tag {3}
$$
Intuitively, the first term ensures the query remains finite, while the second term measures the individual alignment of the query with every stored pattern. Based on the Concave-Convex-Procedure (CCCP) [43], [30] derives the update rule for a state pattern  $\mathcal{E}$  and time  $t$  as follows:
Proposition 1 ([30]). Define the update rule  $f: \mathbb{R}^d \to \mathbb{R}^d$  as follows:
$$
\boldsymbol {\zeta} ^ {n e w} = f (\boldsymbol {\zeta}) = \boldsymbol {X} \operatorname {s o f t m a x} \left(\beta \boldsymbol {X} ^ {T} \boldsymbol {\zeta}\right) \tag {4}
$$
Then, the update rule decreases the loss (3) and converges globally. In another word, for  $\zeta^{(t + 1)} = f(\zeta^{(t)})$ , the energy  $E(\zeta^{(t)}) \to E(\zeta^{*})$  for  $t \to \infty$  and a fixed point  $\zeta^{*}$ .
Note that (4) is equivalent to a gradient descent update to minimize (3) with a step size  $\eta = 1$ :
$$
\zeta \leftarrow \zeta - \eta \nabla_ {\zeta} \operatorname {E} (\zeta ; \boldsymbol {X}) = \zeta - \eta (\zeta - \boldsymbol {X} \operatorname {s o f t m a x} (\beta \boldsymbol {X} ^ {T} \zeta)). \tag {5}
$$
Connection to the attention of the transformer. Remarkably, this implicit energy minimization is closely related to the attention mechanism as shown in [30]. To see this, suppose that  $\pmb{y}_i,\pmb {r}_i\in$ $\mathbb{R}^d$  is given as stored (key) and state (query) patterns, respectively. Let  $\pmb {W}_K,\pmb {W}_Q\in \mathbb{R}^{d\times d_H}$  represent linear maps for  $\pmb{y}_i$  and  $\pmb{r}_i$ , respectively. Then, we introduce  $\pmb{x}_i = \pmb{W}_K^T\pmb {y}_i\in \mathbb{R}^{d_H}$  and  $\zeta_{i} = W_{Q}^{T}\pmb{r}_{i}\in \mathbb{R}^{d_{H}}$ . Let  $\pmb {Y} = (\pmb {y}_1,\dots ,\pmb {y}_N)^T\in \mathbb{R}^{N\times d}$ , and  $\pmb {R} = (\pmb {r}_1,\dots ,\pmb {r}_S)^T\in \mathbb{R}^{S\times d}$ . We define  $\pmb {X}^T = \pmb {K} = \pmb {Y}\pmb {W}_K\in \mathbb{R}^{N\times d_H}$ , and  $\pmb {Q} = \pmb {R}\pmb {W}_Q\in \mathbb{R}^{S\times d_H}$ . By plugging  $\pmb {X}^T = \pmb {K}$  and  $\pmb {\zeta} = q_{i}$  into (4) for all  $i$ , we obtain that:  $\pmb {Q}^T = \pmb {K}^T\mathrm{softmax}_1(\beta \pmb {K}\pmb {Q}^T)\in \mathbb{R}^{d_H\times S}$ . By taking the transpose, we obtain  $\pmb {Q} = \mathrm{softmax}_2(\beta \pmb {Q}\pmb {K}^T)\pmb {K}$ . Let  $\pmb {V}\in \mathbb{R}^{N\times d_V}$  denote the value matrix as  $\pmb {V} = Y\pmb{W}_{K}\pmb{W}_{V} = \pmb{K}\pmb{W}_{V}$ , which will replace the  $\pmb{K}$  outside of  $\mathrm{softmax}_2$ . Then, we finally obtain
$$
\boldsymbol {Q} ^ {\text {n e w}} = \operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {Q} \boldsymbol {K} ^ {T}\right) \boldsymbol {V}, \tag {6}
$$
which is exactly the transformer attention with  $\beta = 1 / \sqrt{d_H}$ . This connection affords us the insightful theoretical ground of the attention mechanism: the update step of the attention mechanism in a Transformer layer acts as an inner-loop optimization step, minimizing an implicit energy function that is determined by queries, keys, and values.
3 Energy-based Cross-Attention
Recall that our objective is to generate semantically correct images by harmonizing latent (U-Net) representations and context vectors within the denoising autoencoder. To achieve this, we propose a simple but effective Bayesian framework, namely energy-based cross-attention (EBCA). Specifically, we perform test-time optimization of context vectors within the cross-attention spaces of a denoising autoencoder to implicitly minimize a specially designed energy function. Note that this is a significant departure from the conventional text-guided diffusion models which have leveraged fixed context vectors embedded by pre-trained CLIP [29] to all cross-attention layers.
3.1 Energy-based Bayesian Context Update
Energy function. Our focus is on the cross-attention space of a time-dependent denoising autoencoder, utilizing the conventional U-Net neural architecture. Here, we refer to latent representations as the representations of intermediate layers in U-Net unless otherwise specified. Let  $L$  be the number of cross-attention layers. For each  $l$ -th layer at time  $t$ , we define the queries matrix  $Q_{t,l} \in \mathbb{R}^{P_l^2 \times d_l}$  and the keys and values matrices  $K_l \in \mathbb{R}^{N \times d_l}$  and  $V_l \in \mathbb{R}^{N \times d_l}$ , respectively. Here,  $P_l$  represents the spatial dimension of latent representations in the  $l$ -th layer. Given context vectors  $C \in \mathbb{R}^{N \times d_c}$ , we map  $K_l$  and  $V_l$  with  $W_{K,l}, W_{V,l} \in \mathbb{R}^{d_c \times d_l}$ , such that  $K_l = CW_{K,l}$  and  $V_l = CW_{V,l}$ . In the following, time  $t$  and layer index  $l$  are omitted in notations  $Q_{t,l}$  and  $K_l$  for simplicity.
The main goal is to obtain a maximum a posteriori probability (MAP) estimate of context vectors  $\mathbf{C}$  given a set of observed latent representations of queries  $Q_{t,l}$ . First, based on the energy functions (8) and (9), the posterior distribution of  $\mathbf{K}$  can be defined by using Bayes' rule:  $p(\mathbf{K} \mid \mathbf{Q}) = p(\mathbf{Q} \mid \mathbf{K})p(\mathbf{K}) / p(\mathbf{Q})$ , where (8) and (9) are leveraged to model the distribution  $p(\mathbf{Q} \mid \mathbf{K})$  and  $p(\mathbf{Q})$ , respectively. Then, in order to obtain a MAP estimation of  $\mathbf{C}$ , we approximate the posterior inference using the gradient of the log posterior. The gradient can be estimated as follows:
$$
\nabla_ {\boldsymbol {K}} \log p (\boldsymbol {K} \mid \boldsymbol {Q}) = \nabla_ {\boldsymbol {K}} \log p (\boldsymbol {Q} \mid \boldsymbol {K}) + \nabla_ {\boldsymbol {K}} \log p (\boldsymbol {K}) = - \left(\nabla_ {\boldsymbol {K}} \operatorname {E} (\boldsymbol {Q}; \boldsymbol {K}) + \nabla_ {\boldsymbol {K}} \operatorname {E} (\boldsymbol {K})\right). \tag {7}
$$
Motivated by the energy function in (3), we first introduce a new conditional energy function w.r.t  $\pmb{K}$  as follows:
$$
\operatorname {E} (\boldsymbol {Q}; \boldsymbol {K}) = \frac {\alpha}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right) - \sum_ {i = 1} ^ {N} \log \operatorname {s u m e x p} \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}, \beta\right), \tag {8}
$$
where  $\pmb{k}_i$  denotes the  $i$ -th row vector of  $\pmb{K}$ ,  $\alpha \geq 0$ , and  $\mathrm{diag}(\mathbf{A}) = \sum_{i=1}^{N} A_{i,i}$  for a given  $\mathbf{A} \in \mathbb{R}^{N \times N}$ . Intuitively,  $\log \mathrm{sumexp}(\pmb{Q}\pmb{k}_i^T, \beta)$  term takes a smooth maximum alignment between latent
representations  $\pmb{q}_j, j = 1, \dots, P^2$  and a given text embedding  $\pmb{k}_i$ . Let  $j^* = \arg \max_j \pmb{q}_j \pmb{k}_i^T$ . Then, the implicit minimization of  $-\log \text{sumexp}$  term encourages each  $\pmb{k}_i$  to be semantically well-aligned with a corresponding spatial token representation  $\pmb{q}_{j^*}$ . In turn, this facilitates the retrieval and incorporation of semantic information encoded by  $\pmb{k}_i$  into the spatial region of  $\pmb{q}_{j^*}$ .
The  $\mathrm{diag}(\pmb {K}\pmb{K}^T)$  term in (8) serves as a regularizer that constrains the energy of each context vector  $\pmb{k}_i$ , preventing it from exploding during the maximization of  $\log \mathrm{sumexp}(\pmb {Q}\pmb{k}_i^T,\beta)$ , thereby ensuring that no single context vector excessively dominates the forward attention path. In this regard, we also propose a new  $Q$ -independent prior energy function  $\operatorname {E}(\pmb {K})$  given by:
$$
\operatorname {E} (\boldsymbol {K}) = \log \operatorname {s u m e x p} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right), 1\right) = \log \sum_ {i = 1} ^ {N} \exp \left(\frac {1}{2} \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}\right). \tag {9}
$$
Instead of penalizing each norm uniformly, it primarily regularizes the smooth maximum of  $\| \pmb{k}_i\|$  which improves the stability in implicit energy minimization. We empirically observed that the proposed prior energy  $\operatorname {E}(\pmb {K})$  serves as a better regularizer compared to the original  $\mathrm{diag}(KK^T)$  term (related analysis in (12) and appendix E).
Although our energy function is built based on the energy function, in contrast to (3), the proposed one is explicitly formulated for implicit energy minimization w.r.t keys  $\mathbf{K}$  and the associated context vectors  $\mathbf{C}$ , which is different from the theoretical settings in Section 2.2 w.r.t  $\mathbf{Q}$  [30]. It is worth noting that the two energy functions are designed to serve different purposes and are orthogonal in their application.
MAP estimation. Based on the proposed energy functions, we derive the update rule of key  $K$  and context  $C$  following (7):
Theorem 1. For the energy functions (8) and (9), the gradient of the log posterior is given by:
$$
\nabla_ {\boldsymbol {K}} \log p (\boldsymbol {K} \mid \boldsymbol {Q}) = \operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {K} \boldsymbol {Q} ^ {T}\right) \boldsymbol {Q} - \left(\alpha \boldsymbol {I} + \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right)\right)\right)\right) \boldsymbol {K}, \tag {10}
$$
where  $\mathcal{D}(\cdot)$  is a vector-to-diagonal-matrix operator.
Then, by using the chain rule the update rule of context vectors  $C$  is derived as follows:
$$
\boldsymbol {C} _ {n + 1} = \boldsymbol {C} _ {n} + \gamma \left(\operatorname {s o f t m a x} _ {2} (\beta \boldsymbol {K Q} ^ {T}) \boldsymbol {Q} - \left(\alpha \boldsymbol {I} + \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} (\boldsymbol {K K} ^ {T})\right)\right)\right) \boldsymbol {K}\right) \boldsymbol {W} _ {K} ^ {T}, \tag {11}
$$
where  $\gamma >0$  is a step size.
In practice, we empirically observed that the nonzero  $\alpha$  in (8) often leads to an over-penalization of contexts, which can ultimately cause some contexts to vanish. To address this, we set  $\alpha = 0$ . Moreover, we found it beneficial to assign distinct step sizes  $\gamma_{\mathrm{attn}}$  and  $\gamma_{\mathrm{reg}}$  as follows:
$$
\boldsymbol {C} _ {n + 1} = \boldsymbol {C} _ {n} + \gamma_ {\text {a t t n}} \underbrace {\operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {K Q} ^ {T}\right) \boldsymbol {Q W} _ {K} ^ {T}} _ {\text {A t t e n t i o n}} - \gamma_ {\text {r e g}} \underbrace {\boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K K} ^ {T}\right)\right)\right) \boldsymbol {K W} _ {K} ^ {T}} _ {\text {R e g u l a r i z a t i o n}}, \tag {12}
$$
where the first and second terms are named as attention and regularization term, respectively.
Our theoretical observations offer valuable insights into implicit energy minimization by modifying the forward path of cross-attention. Inspired by these observations, we have transplanted the energy-based cross-attention layers to pre-trained text-to-image diffusion models. We first illustrate the challenges associated with adopting EBCA in a deep denoising auto-encoder and subsequently demonstrate how to overcome them in practice.
If a single recurrent Transformer block were available, a single energy function would be minimized for a given cross-attention layer by recurrently passing the updated context  $C_{n+1}$ . However, in practical applications, there are multiple layer- and time-dependent energy functions in conventional deep denoising auto-encoder, which makes it infeasible to minimize each energy function individually. To address this challenge, we implicitly minimize a nested hierarchy of energy functions in a single forward pass based on our theoretical observations. More details are followed.
Algorithms. At a given time step  $t$ , we initialize the context vectors  $C_{1,t}$  with text embeddings  $C_{CLIP}$  obtained from a pre-trained CLIP. Then, within the  $n$ -th cross-attention layer with  $C_{n,t}$  ( $1 \leq n \leq L$ ), we compute the updated context vectors  $C_{n+1,t}$  using the gradients in Theorem 1 and (12). We then cascade  $C_{n+1,t}$  to the next ( $n+1$ )-th cross-attention layer. We do not forward the final context  $C_{L+1,t}$  at time  $t$  to time  $t+1$ , as the distributions of  $Q_t$  and  $Q_{t+1}$  can be significantly different due to the reverse step of the diffusion model. Instead, we reinitialize  $C_{1,t+1}$  with  $C_{CLIP}$ . The pseudo-code for the proposed framework is provided in appendix B.
The sequence of energy-based cross-attention layers is designed to minimize nested energy functions. While a single-layer update in a single step may not lead to optimal convergence, the proposed layer-cascade context optimization can synergistically improve the quality of context vectors. Notably, our proposed method does not incur additional computational costs in practice since the gradient term in Theorem 1 relies on the existing keys, queries, and attention maps in the cross-attention layer, which can be obtained for free in the forward path.
3.2 Energy-based Composition of Queries
In addition to the above key advantages, the cross-attention space EBMs shed new light on the zero-shot compositional generation. Recent studies [8, 10] have demonstrated that the underlying distributions of EBMs can be combined to represent different compositions of concepts. For example, given a data point  $\pmb{x}$  and independent concept vectors  $c_{1},\ldots ,c_{n}$ , the posterior likelihood of  $\pmb{x}$  given a conjunction of specific concepts is equivalent to the product of the likelihood of each individual concept as follows:
$$
p \left(\boldsymbol {x} \mid \boldsymbol {c} _ {1}, \dots , \boldsymbol {c} _ {n}\right) = \prod_ {i} p \left(\boldsymbol {x} \mid \boldsymbol {c} _ {i}\right) \propto e ^ {- \sum_ {i} \mathrm {E} \left(\boldsymbol {x} \mid \boldsymbol {c} _ {i}\right)}, \tag {13}
$$