SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T} = (\boldsymbol {e} _ {N} ^ {i}) ^ {T} \boldsymbol {K} \boldsymbol {K} ^ {T} \boldsymbol {e} _ {N} ^ {i}
$$
Furthermore, using (22), we have  $\pmb{K}^T\pmb{e}_N^i = ((\pmb{e}_N^i)^T\otimes \pmb {I}_d)\mathrm{VEC}(\pmb{K}^T)$  so that
$$
\frac {\partial \boldsymbol {K} ^ {T} \boldsymbol {e} ^ {i}}{\partial \operatorname {V E C} (\boldsymbol {K} ^ {T})} = \left(\boldsymbol {e} _ {N} ^ {i} \otimes \boldsymbol {I} _ {d}\right)
$$
where  $I_{d}$  denotes the  $d\times d$  identity matrix. Using the chain rule, we have
$$
\frac {\partial \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}}{\partial \operatorname {V E C} (\boldsymbol {K} ^ {T})} = \frac {\partial \boldsymbol {K} ^ {T} \boldsymbol {e} ^ {i}}{\partial \operatorname {V E C} (\boldsymbol {K} ^ {T})} \frac {\partial \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}}{\partial \boldsymbol {K} ^ {T} \boldsymbol {e} ^ {i}} = 2 (\boldsymbol {e} _ {N} ^ {i} \otimes \boldsymbol {I} _ {d}) \boldsymbol {K} ^ {T} \boldsymbol {e} _ {N} ^ {i}
$$
$$
\mathrm {U N V E C} \left(\frac {\partial \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}}{\partial \mathrm {V E C} (\boldsymbol {K} ^ {T})}\right) = 2 \mathrm {U N V E C} \left(\left(\boldsymbol {e} _ {N} ^ {i} \otimes \boldsymbol {I} _ {d}\right) \boldsymbol {K} ^ {T} \boldsymbol {e} _ {N} ^ {i}\right) = 2 \boldsymbol {K} ^ {T} \boldsymbol {e} _ {N} ^ {i} \left(\boldsymbol {e} _ {N} ^ {i}\right) ^ {T}
$$
where we again use (22). Therefore, by taking the transpose, we have
$$
\frac {\partial \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}}{\partial \boldsymbol {K}} = 2 \boldsymbol {e} _ {N} ^ {i} (\boldsymbol {e} _ {N} ^ {i}) ^ {T} \boldsymbol {K}.
$$
Lemma 3.
$$
\nabla_ {\boldsymbol {K}} \operatorname {l o g s u m e x p} \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}, \beta\right) = \boldsymbol {e} _ {N} ^ {i} \left(\operatorname {s o f t m a x} \left(\beta \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}\right)\right) ^ {T} \boldsymbol {Q} \tag {24}
$$
$$
\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T} = \boldsymbol {Q} \boldsymbol {K} ^ {T} \boldsymbol {e} _ {N} ^ {i} = \boldsymbol {Q} \boldsymbol {K} ^ {T} \boldsymbol {e} _ {N} ^ {i} = \left(\left(\boldsymbol {e} _ {N} ^ {i}\right) ^ {T} \otimes \boldsymbol {Q}\right) \operatorname {V E C} (\boldsymbol {K} ^ {T}).
$$
Hence, using (18) we have
$$
\frac {\partial \boldsymbol {Q} \boldsymbol {K} ^ {T} \boldsymbol {e} ^ {i}}{\partial \operatorname {V E C} (\boldsymbol {K} ^ {T})} = \boldsymbol {e} _ {N} ^ {i} \otimes \boldsymbol {Q} ^ {T} \tag {25}
$$
Furthermore, using Lemma 1 we have
$$
\begin{array}{l} \frac {\partial \operatorname {l o g s u m e x p} \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T} , \beta\right)}{\partial \operatorname {V E C} \left(\boldsymbol {K} ^ {T}\right)} = \frac {\partial \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}}{\partial \operatorname {V E C} \left(\boldsymbol {K} ^ {T}\right)} \frac {\partial \operatorname {l o g s u m e x p} \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T} , \beta\right)}{\partial \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}} \\ = \left(\boldsymbol {e} _ {N} ^ {i} \otimes \boldsymbol {Q} ^ {T}\right) \operatorname {s o f t m a x} \left(\beta \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}\right) \\ \end{array}
$$
leading to the following:
$$
\begin{array}{l} \frac {\partial \operatorname {l o g s u m} \exp \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T} , \beta\right)}{\partial \boldsymbol {K} ^ {T}} = \mathrm {U N V E C} \left(\frac {\partial \operatorname {l o g s u m} \exp \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T} , \beta\right)}{\partial \mathrm {V E C} \left(\boldsymbol {K} ^ {T}\right)}\right) \\ = \operatorname {U N V E C} \left(\left(\boldsymbol {e} _ {N} ^ {i} \otimes \boldsymbol {Q} ^ {T}\right) \operatorname {s o f t m a x} \left(\beta \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}\right)\right) \\ = \boldsymbol {Q} ^ {T} \operatorname {s o f t m a x} \left(\beta \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}\right) \left(\boldsymbol {e} _ {N} ^ {i}\right) ^ {T} \\ \end{array}
$$
where we again use (22). Now, by taking the transpose, we can prove (24). Q.E.D.
$$
\frac {\partial \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right)}{\partial \boldsymbol {K}} = 2 \boldsymbol {K} \tag {26}
$$
$$
\frac {\partial \log \sum_ {i = 1} ^ {N} \exp \left(\frac {1}{2} \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}\right)}{\partial \boldsymbol {K}} = \left[ \begin{array}{c c c} {[ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {1}} & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & {[ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {N}} \end{array} \right] \boldsymbol {K} \tag {27}
$$
where  $\mathrm{diag}(\mathbf{A})$  denotes the sum of the diagonal element, i.e. trace of  $\mathbf{A}$ , and  $\pmb{x}_i$  is a column matrix whose  $i$ -th element is given by  $\pmb{k}_i\pmb{k}_i^T / 2$ , and  $[\cdot]_i$  dents the  $i$ -th element.
Proof. First,
$$
c = \operatorname {d i a g} (\boldsymbol {K} \boldsymbol {K} ^ {T}) = \sum_ {i, j} K _ {i j} ^ {2}
$$
where  $K_{ij}$  denotes the  $(i,j)$ -th element. Therefore, using the denominator layout in (17), it is trivial to show that
$$
\frac {\partial c}{\partial \boldsymbol {K}} = 2 \boldsymbol {K}.
$$
Second, let us construct a column vector  $\pmb{x}$  whose  $i$ -th element is given by  $x_{i} := \pmb{k}_{i}\pmb{k}_{i}^{T} / 2$ . Then, using Lemmas 1 and 2 and the chain rule, we have
$$
\begin{array}{l} \frac {\partial \log \sum_ {i = 1} ^ {N} \exp \left(\frac {1}{2} \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}\right)}{\partial \boldsymbol {K}} = \sum_ {i} \frac {\partial x _ {i}}{\partial \boldsymbol {K}} \frac {\partial \operatorname {l o g s u m e x p} (\boldsymbol {x} , 1)}{\partial x _ {i}} \\ = \sum_ {i} \boldsymbol {e} _ {N} ^ {i} (\boldsymbol {e} _ {N} ^ {i}) ^ {T} \boldsymbol {K} [ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {i} \\ = \sum_ {i} \boldsymbol {e} _ {N} ^ {i} \boldsymbol {k} _ {i} [ \text {s o f t m a x} (\boldsymbol {x}) ] _ {i} \\ = \left[ \begin{array}{c c c} [ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {1} & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & [ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {N} \end{array} \right] \boldsymbol {K} \\ \end{array}
$$
Theorem 1. For the energy functions
$$
\operatorname {E} (\boldsymbol {Q}; \boldsymbol {K}) = \frac {\alpha}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right) - \sum_ {i = 1} ^ {N} \log \operatorname {s u m e x p} \left(\boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T}, \beta\right) \tag {28}
$$
$$
E (\boldsymbol {K}) = \log \sum_ {i = 1} ^ {N} \exp \left(\frac {1}{2} \boldsymbol {k} _ {i} \boldsymbol {k} _ {i} ^ {T}\right), \tag {29}
$$
the gradient of the log posterior is given by:
$$
\nabla_ {\boldsymbol {K}} \log p (\boldsymbol {K} \mid \boldsymbol {Q}) = \operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {K} \boldsymbol {Q} ^ {T}\right) \boldsymbol {Q} - \left(\alpha \boldsymbol {I} + \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right)\right)\right)\right) \boldsymbol {K}, \tag {30}
$$
Then, by using the chain rule the update rule of context vectors  $\mathbf{C}$  is derived as follows:
$$
\boldsymbol {C} _ {n + 1} = \boldsymbol {C} _ {n} + \gamma \left(\operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {K} \boldsymbol {Q} ^ {T}\right) \boldsymbol {Q} - \left(\alpha \boldsymbol {I} + \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right)\right)\right)\right) \boldsymbol {K}\right) \boldsymbol {W} _ {K} ^ {T}, \tag {31}
$$
where  $\gamma > 0$  is a step size, and  $\pmb{D}(\cdot)$  is a vector-to-diagonal-matrix operator.
Proof. Based on the Bayes' theorem, the gradient of the log posterior is derived as:
$$
\nabla_ {\boldsymbol {K}} \log p (\boldsymbol {K} \mid \boldsymbol {Q}) = - \left(\nabla_ {\boldsymbol {K}} \operatorname {E} (\boldsymbol {Q}; \boldsymbol {K}) + \nabla_ {\boldsymbol {K}} \operatorname {E} (\boldsymbol {K})\right). \tag {32}
$$
Using Lemmas 1,2,3 and 4, we have
$$
\begin{array}{l} \nabla_ {\boldsymbol {K}} \operatorname {E} (\boldsymbol {Q}; \boldsymbol {K}) = \alpha \boldsymbol {K} - \sum_ {i} \boldsymbol {e} _ {N} ^ {i} (\operatorname {s o f t m a x} (\beta \boldsymbol {Q} \boldsymbol {k} _ {i} ^ {T})) ^ {T} \boldsymbol {Q} \\ = \alpha \boldsymbol {K} - \operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {Q} \boldsymbol {K} ^ {T}\right) \boldsymbol {Q} \tag {33} \\ \end{array}
$$
Second, by noting that  $\operatorname{E}(\pmb{K}) = \log \sum_{i=1}^{N} \exp\left(\frac{1}{2} \pmb{k}_i \pmb{k}_i^T\right)$ , Lemma 4 informs us
$$
\begin{array}{l} \nabla_ {\boldsymbol {K}} \operatorname {E} (\boldsymbol {K}) = \left[ \begin{array}{c c c} [ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {1} & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & [ \operatorname {s o f t m a x} (\boldsymbol {x}) ] _ {N} \end{array} \right] \boldsymbol {K} \\ = \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right)\right)\right) \boldsymbol {K} \\ \end{array}
$$
where  $\mathcal{D}(\cdot)$  is a vector-to-diagonal-matrix operator that takes  $N$ -dimensional softmax vector as an input and returns a  $N \times N$  diagonal matrix with softmax values as main diagonal entries. Therefore, one can finally obtain:
$$
\nabla_ {\boldsymbol {K}} \log p (\boldsymbol {K} \mid \boldsymbol {Q}) = \operatorname {s o f t m a x} _ {2} \left(\beta \boldsymbol {K} \boldsymbol {Q} ^ {T}\right) \boldsymbol {Q} - \left(\alpha \boldsymbol {I} + \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} \left(\boldsymbol {K} \boldsymbol {K} ^ {T}\right)\right)\right)\right) \boldsymbol {K}. \tag {34}
$$
By using the chain rule with  $\pmb{K} = \pmb{C}\pmb{W}_{K}$ , the update rule of context vectors  $\pmb{C}$  is derived as in (11).
B Pseudo-code for EBCU and EBCQ
This section provides the description of the pseudocode for the proposed Energy-based Bayesian Context Update (EBCU) and Energy-based Composition of Queries (EBCQ). Algorithm 1 outlines the cascaded context propagation across cross-attention layers within the UNet model during the sampling step  $t$ . Note that the context is reinitialized at the beginning of each sampling step. On the other hand, Algorithm 2 details the EBCU implemented in each cross-attention layer. Specifically, the proposed EBCU provides a significant computational efficiency by reusing the similarity  $QK^T$ , which requires computational cost  $\mathcal{O}(N^2)$ , to compute  $\nabla_{\boldsymbol{K}}E(\boldsymbol{Q};\boldsymbol{K})$ . Consequently, there is only a small amount of additional computational overhead associated with the proposed EBCU.
Algorithm 1 Context cascade at sampling step  $t$
Require:  $Q_{t}, C_{clip}$ , UNet  
1:  $C_{t} \gets C_{clip}$  // Re-initialize  
2: for layer in UNet do  
3: if layer is CrossAttention then  
4:  $Q_{t}, C_{t} \gets \text{layer}(Q_{t}, C_{t})$  // Algorithm 2  
5: else  
6:  $Q_{t} \gets \text{layer}(Q_{t})$   
7: end if  
8: end for  
9:  $Q_{t+1} \gets Q_{t}$   
10: return  $Q_{t+1}$
Algorithm 2 Energy-based Bayesian Context Update (EBCU)
Require:  $Q, C, W_{q}, W_{k}, W_{v}, \alpha, \beta, \gamma_{\mathrm{attn}}, \gamma_{\mathrm{reg}}$   
1:  $Q, K, V \gets QW_{q}, CW_{k}, CW_{v}$   
2:  $S = QK^{T}$   
3:  $Q \gets \operatorname{softmax}_{2}(\beta S)V$   
4:  $\nabla_{K}E(Q; K) = \operatorname{softmax}_{2}(\beta S^{T})Q$   
5:  $\nabla_{K}E(K) = -(\alpha I + D(\operatorname{softmax}(\frac{1}{2}\operatorname{diag}(KK^{T})))K$   
6:  $\Delta C = (\gamma_{\mathrm{attn}}\nabla_{K}E(Q; K) + \gamma_{\mathrm{reg}}\nabla_{K}E(K))W_{k}^{T}$   
7:  $C \gets C + \Delta C$   
8: return  $Q, C$
Algorithm 3 outlines the pseudocode for the EBCQ implemented for  $M$  given contexts. For the simplicity, we exclude the EBCU from the algorithm. Nonetheless, the EBCU and the EBCQ could be leveraged together.
Algorithm 3 Energy-based Composition of Queries (EBCQ)
Require:  $Q,C = \{C_1,\dots ,C_M\} ,W_q,W_k,W_v,\alpha_s,\beta$  1:  $Q\gets QW_{q}$    
2: for s in [1,...,M] do   
3:  $K_{s},V_{s}\leftarrow C_{s}W_{k},C_{s}W_{v}$    
4:  $S_{s} = QK_{s}^{T}$    
5: end for   
6:  $Q\gets \frac{1}{M}\sum_{s = 1}^{M}\alpha_{s}\mathrm{softmax}_{2}(\beta S_{s})V_{s}$    
7: return  $Q$
C Experimental setups
In this section, we describe detailed experimental setups for three applications including baseline method, hyper-parameter of the proposed method, and dataset if it is the case. Code: https://github.com/EnergyAttention/Energy-Based-CrossAttention.
C.1 Common experimental setup
We mainly leverage pre-trained Stable Diffusion v1-5 (except Table 1: v1-4) which is provided by diffusers, a Python library that offers various Stable Diffusion pipelines with pre-trained models. All images are sampled for
50 steps via PNDM sampler [21] using NVIDIA RTX 2080Ti. In every experiment, we set the parameter  $\alpha$  in Equation (9) to zero, focusing solely on controlling the values of  $\gamma_{attn}$  and  $\gamma_{reg}$ . EBCU is applied to every task, and EBCQ is additionally employed in C.4.
Different learning rate for each token It is worth noting that the  $\gamma_{\text{attn}}$  and  $\gamma_{\text{reg}}$  could be expressed as vectors. In other words, if the context  $C \in \mathbb{R}^{N \times d_c}$  is given,  $\gamma_{\text{attn}}$  and  $\gamma_{\text{reg}}$  are  $N$ -dimensional vectors. Hence, we have the flexibility to adjust the learning rate  $\gamma_{\{\cdot\}}$ , allowing us to increase or decrease the impact of certain tokens based on the user's intent. Unless otherwise noted,  $\gamma_{\text{attn}}$  and  $\gamma_{\text{reg}}$  is set to a constant for each text token.
Learning rate scheduling Since the proposed EBCU is leveraged for the diffusion model, one can readily introduce scheduling strategies for  $\gamma_{attn}$  and  $\gamma_{reg}$  along the sampling step  $t$ . We implement multiple variants such as 'constant', 'step', and 'exponential decay' as follows.
$$
[ \text {c o n s t a n t} ] \quad \gamma (t) = \gamma_ {0}
$$
$$
[ \text {s t e p} ] \quad \gamma (t) = \gamma_ {0} \cdot \operatorname {R e L u} (t - \tau) \tag {35}
$$
$$
\left[ \exp \text {- d e c a y} \right] \quad \gamma (t) = \gamma_ {0} \cdot \lambda^ {t}
$$
where  $\gamma_0$  is the initial value,  $\mathrm{ReLu}(x) = 0$  if  $x \leq 0$ , otherwise  $1, \tau$  denotes the temporal threshold, and  $\lambda$  denotes the decay ratio. Unless stated otherwise, the scheduling strategy is set to the 'constant'.
C.2 Multi-concept image generation
We compared the performance of the proposed method with Structured Diffusion [11] which does not require additional training as our method. We leveraged the open-sourced official implementation  ${}^{1}$  .
For the proposed method, we set the  $\gamma_{attn}$  and  $\gamma_{reg}$  differently for each sample within [1e-2, 1.5e-2, 2e-2]. As shown in the following ablation studies E, large  $\gamma_{attn}$  tends to generate saturated images while large  $\gamma_{reg}$  results in mixed/vanished contents.
We found that using different learning rates for each context token is useful for multi-concept generation, especially when a single concept tends to dominate with a constant learning rate. For example, given the main prompt "A cat wearing a shirt", we set the  $\gamma_{attn}$  for the "shirt" to 3e-2, while  $\gamma_{attn}$  is set to 1.5e-2 for other tokens. We have observed that doubling the  $\gamma_{attn}$  for a text token to be emphasized is sufficient to achieve balanced multi-concept image generation for most cases.
C.3 Text-guided image inpainting
Additionally, we conducted a performance comparison between our proposed method and two alternative approaches: (a) Stable Inpaint², which fine-tunes the weights of Stable Diffusion through inpainting training, and (b) Stable Repaint³, which leverages the work of Lugmayr et al. [23] on the latent space of Stable Diffusion for the inpainting task. In the case of Stable Repaint, the mask is downsized and transferred into the latent space. We applied the Energy-based Bayesian Context Update (EBCU) technique to both methods, resulting in improved results compared to their respective baselines.
Masked EBCU. To further enhance the performance for the inpainting task, we introduce the concept of masked Energy-based Bayesian Context Update (masked EBCU). Specifically, let  $M \in \mathbb{R}^{P_l^2 \times P_l^2}$  represent a diagonal matrix where the main diagonal values are derived from the downsampled inpainting mask for the  $l$ -th cross-attention layer, with an output spatial size of  $P_l^2$ . In Equation (36), we modify the attention term (12) by incorporating the downsampled mask, effectively covering the query matrix as follows:
$$
\boldsymbol {C} _ {n + 1} = \boldsymbol {C} _ {n} + \gamma \left(\operatorname {s o f t m a x} _ {2} (\beta \boldsymbol {K} \boldsymbol {Q} ^ {T}) \boldsymbol {M} \boldsymbol {Q} - \left(\alpha \boldsymbol {I} + \boldsymbol {\mathcal {D}} \left(\operatorname {s o f t m a x} \left(\frac {1}{2} \operatorname {d i a g} (\boldsymbol {K} \boldsymbol {K} ^ {T})\right)\right)\right) \boldsymbol {K}\right) \boldsymbol {W} _ {K} ^ {T}. \tag {36}
$$
As evident in Equation (12), the attention term updates the context vectors, aligning  $\pmb{k}_i$  towards  $\pmb{q}_j$ ,  $j = 1, \dots, P_l^2$ , while considering the alignment strength between each  $\pmb{q}_j$  and  $\pmb{k}_i$ . However, in the inpainting task, we have prior knowledge that the context vectors should be most aligned with the semantically relevant masked regions. Therefore, we mask out unrelated background spatial representations, allowing for the context vectors to be updated with a specific focus on the masked regions. This approach facilitates the incorporation of semantic information encoded by  $\pmb{k}_i$  specifically into the spatial mask regions.
In our proposed method, we set different values for  $\gamma_{attn}$  and  $\gamma_{reg}$  for each sample, selected from the set [1e-2, 1.5e-2, 2e-2, 2.5e-2], to account for variations in the input samples.
$^{1}$ https://github.com/weixi-feng/Structured-Diffusion-Guidance  
 $^{2}$ https://huggingface.co/runwayml/stable-diffusion-inpainting  
 $^{3}$ https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-repaint
C.4 Image editing via compositional generation
We present empirical evidence demonstrating the effectiveness of our energy-based framework for compositional synthetic and real-image editing. The Energy-based Bayesian Context Update (EBCU) technique can be readily applied to both the main context vector  $(C_1$  in Section 3.2,  $s = 1)$  and editorial context vectors  $(C_{s > 1})$ . Each EBCU operation influences the attention maps used in the Energy-based Composition of Queries (EBCQ), enhancing the conveyance of semantic information associated with each context. Note that  $\alpha_{s}$  in (16) represents the degree of influence of the  $s$ -th concept in the composition. In practice, we fix  $\alpha_{1} = 1$  for the main context, while  $\alpha_{s > 1}$  is tuned within the range of (0.5, 1.0).
Let  $\gamma_{attn,s}$  and  $\gamma_{reg,s}$  denote the step sizes for EBCU of the  $s$ -th context vector. If the editing process involves changing the identity of the original image (e.g., transforming a "cat" into a "dog"), we set both  $\gamma_{attn,1}$  and  $\gamma_{reg,1}$  to zero. Otherwise, if the editing maintains the original identity, we choose values for  $\gamma_{attn,1}$  and  $\gamma_{reg,1}$  from the range of (5e-4, 1e-3), similar to  $\gamma_{attn,(s > 1)}$  and  $\gamma_{reg,(s > 1)}$ . All hyperparameters, including  $\alpha_s$  and  $\gamma_s$ , are fixed during the quantitative evaluation process (more details in Section D and Table 3).
To ensure consistent results, we maintained a fixed random seed for both real and synthetic image editing. For real image editing, we employed null-text pivotal inversion [25] to obtain the initial noise vector.
During the reverse diffusion process in Sections C.2 and C.3, we kept  $\gamma$  fixed as a constant value. However, for compositional generation, we utilized step scheduling (Equation 35) for  $\gamma_{s}$  and  $\alpha_{s}$ . After converting the initial noise vector for real images or using a fixed random seed for synthetic images, EBCU and EBCQ are applied after a threshold time  $\tau_{s} > 0$  for the  $s$ -th editorial context. This scheduling strategy helps to preserve the overall structure of generated images during the editing process. In our observations, a value of  $\tau_{s} \in [10, 25]$  generally produces satisfactory results, considering a total number of reverse steps set to 50. However, one can increase or decrease  $\tau_{s}$  for more aggressive or conservative editing, respectively.
The exemplary real images presented in Figures 5 and 6 of the main paper were sampled from datasets such as FFHQ [17], AFHQ [5], and ImageNet [7]. For a detailed quantitative analysis, please refer to Section D.
D Quantitative Comparison
In this section, we conducted a comparative analysis of the proposed framework against several state-of-the-art diffusion-based image editing methods [24, 12, 25, 28], following the experimental setup of [28]. To ensure a fair comparison, all methods utilize the pre-trained Stable Diffusion v1-4, employ the PNDM sampler with an equal number of sampling steps, and adopt the same classifier-free guidance scale.
D.1 Baseline Methods
In addition to the Plug-and-Play method discussed in the main paper, we include the following baselines for comprehensive quantitative comparison:
SDEdit [24] + word swap. This method introduces the Gaussian noise of an intermediate timestep and progressively denoises images using a new textual prompt, where the source word (e.g., Cat) is replaced with the target word (e.g., Dog).
Prompt-to-prompt (P2P) [12]. P2P edits generated images by leveraging explicit attention maps from a source image. The source attention maps  $M_{t}$  are used to inject, re-weight, or override the target maps based on the desired editing operation. These original maps act as hard constraints for the edited images.
DDIM + word swap [25]. This method applies null-text inversion to real input images, achieving high-fidelity reconstruction. DDIM sampling is then performed using inverted noise vectors and an edited prompt generated by swapping the source word with the target.
pix2pix-zero [28]. pix2pix-zero first derives a text embedding direction vector  $\triangle c_{\mathrm{edit}}$  from the source to the target by using a large bank of diverse sentences generated from a state-of-the-art sentence generator, such as GPT-3 [2]. Inverted noise vectors are denoised with the edited text embedding,  $c + \triangle c_{\mathrm{edit}}$ , and cross-attention guidance to preserve consensus.
Cycle-diffusion [40]. Cycle-diffusion reformulates diffusion models as deterministic mappings from a Gaussian latent code to images. It presents a DPM-Encoder that allows for encoding images into this latent space.
D.2 Dataset
For the animal transition task, we focus on three image-to-image translation tasks: (1) translating cats to dogs (cat  $\rightarrow$  dog), (2) translating horses to zebras (horse  $\rightarrow$  zebra), and (3) adding glasses to cat input images (cat  $\rightarrow$  cat with glasses). Following the data collection protocol of [28], we retrieve 250 relevant cat images and 213
horse images from the LAION 5B dataset [34] using CLIP embeddings of the source text description. We select images with a high CLIP similarity to the source word for each task.