Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models
Authors: Geon Yeong Park
Jeongsol Kim
Beomsu Kim
Sang Wan Lee
Jong Chul Ye
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion model
Energy-based model
Text-to-image generation
TL;DR: We propose a novel energy-based cross-attention mechanism for the text-to-image diffusion models.

Abstract: Despite the remarkable performance of text-to-image diffusion models in image generation tasks, recent studies have raised the issue that generated images sometimes cannot capture the intended semantic contents of the text prompts, which phenomenon is often called semantic misalignment. To address this, here we present a novel energy-based model (EBM) framework for adaptive context control by modeling the posterior of context vectors. Specifically, we first formulate EBMs of latent image representations and text embeddings in each cross-attention layer of the denoising autoencoder. Then, we obtain the gradient of the log posterior of context vectors, which can be updated and transferred to the subsequent cross-attention layer, thereby implicitly minimizing a nested hierarchy of energy functions. 
Our latent EBMs further allow zero-shot compositional generation as a linear combination of cross-attention outputs from different contexts. 
Using extensive experiments, we demonstrate that the proposed method is highly effective in handling various image generation tasks, including multi-concept generation, text-guided image inpainting, and real and synthetic image editing. Code: https://github.com/EnergyAttention/Energy-Based-CrossAttention.

Energy-Based Cross Attention for Bayesian Context Update in Text-to-lmage Diffusion Models
Geon Yeong Park Jeongsol Kim Beomsu Kim? 2
Sang Wan 12.3f Jong Chul Year
Bio and Brain Engineering. Kim Jaechul Graduate School of AI, BBrinn and Cognitive Sciences Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea
*, Co-first and Co-corresponding authors (pky3436, jeongsol, beomsu. kim, sangwan, jong.yel yeaateeeeetteeee kcr
Abstract
Despite the remarkable performance of text-to-image diffusion models in image generation tasks, recent studies have raised the issue that generated images some- times cannot capture the intended semantic contents of the text prompts, which phenomenon is often called semantic misalignment. To address this, here we present novel energy-based model (EBM) framework for adaptive context trol by modeling the posterior of context vectors. Specifically, we first formulate EBMs of latent image representations and text embeddings in each cross-attention layer of the denoising autoencoder. Then, we obtain the gradient the log pos- terior of context vectors, which can be updated and transferred the subsequent cross-attention layer, thereby implicitly minimizing a nested hierarchy of energy functions. Our latent EBMs further allow zero-shot compositional generation a linear combination of cross-atie cross-attention outputs from different contexts. Using extensive experiments, we demonstrate that the proposed method highly effective in handling various image generation tasks, including multi-concept generation, text-guided image inpainting, and real and synthetic image editing. Code: https //github. com/
-aergy-Baseed-rrooeee Based- Crassdd-CrossAttention
Introduction
Diffusion models (DMs) have made significant advances in controllable multi-modal generation tasks [37], including text-to-image synthesis. The recent success of text-to-image diffusion models, e.g.. Stable Diffusion (3I]. Imagen (33], etc., is mainly attributed to the combination of high-fidelity DMs with segh-performncc large language models (LMs).
Although text-to-ine tot-too-mmae DMs have shown revolutionary progress, recent studies have shown that the current state-of-the-an models often suffer from semantic misalignment problems, where the generated images do not accurately represent the intended semantic contents of the text prompts. For example, (4) discovered the catastrophic neglect problem, where
of the concepts of the prompt are neglected in generated images. Moreover, for a multi-modal inpainting task with text and mask guidance, [41] found that the text-to-image DMs may often fail to fill in the masked region precisely following the text prompt.
Therefore, this work focuses obtaining harmonized pair of latent image representations and text embeddings, i.e., context vectors, to generate semantically aligned images. In order to mitigate the misalignment, instead of leveraging fixed context vectors, aim to establish adaptive context by modeling the posterior the context, i.c. p(context representations). Note that this is significant departure from the previous methods which only model prrepresentations context) with frozen context vectors encoded by the pretrained textual encoder.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(A) Molli-concept
(B) Inpainting
Synthelic Image Editing
"A monkkey wear Kimono" Moroe and dog"
Mesked Ieput
Siabi-Offtsiio
Cuni
Stable Difusiion
'A corgl wearing bow and birthday hat"
A costo noxt to fiver
(+1 Monet, Boot
Ours
"A teddy
smile curfs
(D) Image Editing
Oriinal image
"olorrll car"
"ink car
"dog
Tox"
"wearing sunglasses'
Figure 1: The energy-based cross-attention improves the semantic alignment between given text and the generated sample. The proposed method could be leveraged for multiple applications without additional training.
Specifically, we introduce a novel energy-basee Bayesian framework, namely energy-based cross- attention (EBCA), which approximates maximum posteriori probability (MAP) estimates of context vectors given observed latent representations. Specifically, to model p(context representations), we first consider analogous p(Kr\Q... in the latent space of the intermediate cross-attention layer of denoising auto-encoder, i.c., cross-attention space. Here, K2 and Qt., correspond to the key and query in 1-th layer time that encode the context from the text and the image representation, respectively, Inspired by the energy-based perspective of attention mechanism [30], we then formulate energy functions E( K;; Qu,)) each atteettotnn space to model the posterior. Finally, we create correspondence between the context and representation by minimizing these parameterized energy functions. More specifically, by obtaining the gradient of the log posterior of context vectors, a nested hierarchy of energy functions can be implicitly minimized by cascading the updated contexts to the subsequent pross-attentie layer (Figure
Moreover, our energy-based perspective of cross-attention also allows zero-shot compositional generation due the inherent compositionality of Energy-Based Models (EBMs). This involves the convenient integration of multiple distributions, each defined by the energy function of a specific text embedding. In practical terms, this amalgamation can be executed combination of cross-ates -atss-attnntion outputs that correspond to all selected editing prompts.
straightforward linear
We demonstrate the effectiveness of the proposed EBM framework in various text-to -mtate generative scenarios, including multi-concept generation, text-guided inpainting, and compositional generation. The proposed method is training-free, easy to implement, and can potentially be integrated into most of the existing text-to-image DMs.
2 Preliminaries
2.1 Diffusion Model
Diffusion models 36
13 37
18
16 aims to generate samples from the Gaussian noise by iterative
denoising processes. Given clean data 2. (Xo). diffusion models define the forward sampling from ((zz(ao) as = Vā + v1 αz where zt AO I). E [0. 1]. Here, for the Denoising Diffusion Probabilistic Model (DDPM) [3], the noise schedule B, is an increasing
sequence of f, with ā := I at; 04 = 1 3r. The goal of diffusion model training is to obtain neural network Eg- that satisfies
0+ = argmin En, Sppe (tn) (ng) Pa (xa) [1), t) El
(I)

---

Zp-1
W
Attn
Attn
Query
EBCU
Contest
Contest
Figure 2: Comparison between the Stable- Diffe-Diffusoon and the proposed method. (A) The Stable- Diffusion uses fixed context embedding encoded by pre-trained CLIP. (B) The proposed method allows adaptive context embedding through energy-based context update (EBCU) and energy-based composition of queries (EBCQ).
so that the reverse sampling from q(
1 12,,, zg- (x t)) achieved by
1 aq zt
2t-1
Eg- (zc: t) + oi
(2)
Var
V1
where NN, I) and 2² is variance which is set as 3 in DDPM. With iterative process, we can sample the image zy from initial sample æ ~ NV I).
Since diffusion models require the iterative sampling on high dimensional space, they are com- putationally expansive and time consuming. To mitigate this limitation, Latent Diffusion Model (LDM) 331 has proposed diffusion processes on the compressed latent space using pre-trained auto-encoder. Furthermore, by introducing language model-based cross-attention diffusion model's U-Net neural backhone 332 LDM enables token-basee conditioning method such as text-to-image.
2.2 Easrgy-based Perspective of Attention Mechanism
Definition. Given an N-dimensional vector v, its logsumexp and softmax are defined
( EM- exp(v:)), softmax(e) exp(v logsumexp(e; 1)). where v; denotes the i-th element of v. For a given matrix A, its logsumexp, (A) and softmax,(A). are understood as taking the corresponding operation along the i-th dimension of A. So, for instance, i 1, softmax, (A) consist of column vectors that to 1.
logsumexp(er: 3) 3-1 log
Energy model. Following the success of the attention mechanism, several studies have focused on establishing its theoretical foundation. One promising avenue is interpreting the attention mechanism using the Energy- -ased Model (EBM) framework. This line of research begins with recent research on modern Hopfield networks (30 14, which gradually builds up to the self-attention mechanism of the Transformer model.
The Hopfield network is dense associative memory model that aims to associate an input with its most similar patter. Specifically, it constructs an energy function model an energy landscape that contains basins of attraction around desired patterns. Recently, modern Hopfield networks 30 I9 has introduced new family of energy functions, which aim improve its pattern storage capacity make it compatible with continuous embeddings. To this end, [30] proposed the following energy function of a state pattern (query) Ed parameterized by -sstored (key) patterns X = [ ..... ÆN] E Rdx N and B > 0:
E((: X) = 1 ---- logsumexp( X (
Intuitively, the first term ensures the query remains finite, while the second term measures the individual alignment the query with every stored pattern. Based on the Concave-Convex ConeeeCoooxx--oeeuuu (CCCP) 43 30 derives the update rule for state pattern & and time as follows: Proposition
Define the update rule f .Rd + Rdd as follows:
new =((C) = X softmax(33'
(4)
Then, the update rule decreases the loss 33 and converges globally. In another word, for ((++1) )), the energy ((((")) - E((") )(())frr for t + and a fixed point S"
3

---

Note that 4 is equivalent to gradient descent update to minimize with step size rj 1: = -- - X softmaxx [xx ()).
< 7- -v
(5)
Connection to the attention of the transformer. Remarkably, this implicit energy minimization is closely related to the attention mechanism shown in [30]. To see this, suppose that v,, E R is given as stored (key) and state (query) patterns, respectively. Let W k, We E Rdxdæ represent linear maps for y; and ri, respectively. Then, we introduce I; = W KD; Rud and 5; W rr; E Rdd, Let Y = (yI....., yN E RNNx and R = (ri,,..,, Tss) E BSdd We define X
K YWK K RN NNdH, and Q = RW E RSdy, By plugging x = K and Ç = 9; into 4 for all i, we obtain that: Q² TT softmaxi max (3KQ") E Rd XS By taking the transpose, we obtain Q softmaxx (atrmaxx(3QBB) (3QK K. Let V E Rxxy denote the value matrix as V = YWWWVv K Wv, which will replace the K outside of softmax,. Then, we finally obtain
Quew = soft (BQ
which is exactly the transformer attention with 8 = 1//Vd This connection affords us the insightful theoretical ground of the attention mechanism: the update step of the attention mechanism in Transformer layer acts as an inner-loop optimization step, minimizing an implicit energy function that is determined by queries, keys, and values.
3 Energy-based Cross-Attention
Recall that our objective is to generate semantically correct images by harmonizing latent (U-Net) representations and context vectors within the denoising autoencoder. To achieve this, we propose simple but effective Bayesian framework, namely mergy-based cross-attention (EBCA). Specifically, we perform test-time optimization of context vectors within the cross-attention spaces of denoising autoencoder to implicitly minimize a specially designed energy function. Note that this is a significant departure from the conventional text-guided diffusion models which have leveraged fixed context vectors embedded by pre-trained CLIP (29 to all cross-attention layers.
3.1 Energy-based Bayesian Context Update
Energy function. Our focus is the cross cros-aatenntinn space of time-dependent denoising auto- encoder, utilizing the conventional U-Net neural architecture. Here, we refer to latent representations as the representations of intermediate layers in U-Net unless otherwise specified. Let L be the number of cross-attention layers. For each (-th layer time t, we detine the queries matrix Q E 1& xd. and the keys and values matrices K2 E RV N and V E R Nxd, respectively. Here, Pt represents the spatial dimension of latent representations in the (-th layer. Given context vectors C E R Nxxe, we map K, and V. with W k., W W.r E xd, such that K = CW K. and V CW vu-- In the following, time and layer index I are omitted in notations Qt, and Ki for simplicity.
The main goal is to obtain a maximum a posteriori probability (MAP) estimate of context vectors C given a set of observed latent representations of queries Q2 First, based on the energy functions (8) and (9). the posterior distribution of K can be defined by using Bayes' rule: p(K Q pTQ KJ Tp( k))((Q), where 88) and (9) are leveraged to model the distribution p(Q K) and pl Q),
respectively. Then, in order to obtain MAP estimation of C, we approximate the posterior inference using the gradient of the log posterior. The gradient can be estimated as follows:
k log K Q) V log p( K) + V
K log ((K) (V E(Q: + V B(K)). (7)
Motivated by the energy function in 3 we first introduce new conditional energy function w.r.t K as follows:
E(Q:
O 2
diag (KKT)
logsumexp(Q22 (QK)s .)).
(8)
where k; denotes the i-th row vector of K, 2 0, and diag(A) = EN'' A for a given A E RNxN Intuitively, logsumexp (okpsspp(kk,,_ ,9) term takes smooth maximum alignment between latent

---

representations qj. -..... P2 and given text embedding kG. Let = arg max, g,kk Then, the implicit minimization of logsumexp term encourages each k; be semantically well-aligned with a corresponding spatial token representation 9z-. In turn, this facilitates the retrieval and incorporation of semantic information encoded by k; into the spatial region of 4,--
The diag( K K") term in serves as a regularizer that constrains the energy of each context vector k; preventing it from exploding during the maximization of logsumexp( QK: ,,), thereby ensuring that no single context vector excessively dominates the forward attention path. In this regard, we also propose a new Q Q-independent prior energy function E K given by:
N
( diag (K"), ).1) log Σ exp (kAA.)
E (
logsumexp
(9)
Instead of penalizing each norm uniformly, it primarily regularizes the smooth maximum of ||& which improves the stability in implicit energy minimization. We empirically observed that the proposed prior energy E K serves as a better regularizer compared to the original diag( K K")
term (related analysis in (12) and appendix E
Although our energy function is built based on the energy function, in contrast to (3) the proposed one is explicitly formulated for implicit energy minimization w.r.t keys K and the associated context vectors C, which is different from the theoretical settings in Section 2.2 w.r.t Q [30]. It is worth noting that the two energy functions are designed to serve different purposes and are orthogonal in their application.
MAP estimation. Based on the proposed energy functions, we derive the update rule of key K and context C' following (7)
Theorem 1. For the energy functions and 0)) the gradient of the log posterior is given by:
(aI + D softmax (5 diag K kK())) K.
K log Q) = softmax2 (BKQ")
(10)
where D()) is a tittt--dgaaa---nnn operator. Then, by using the chain rule the update rule of context vectors C' is derived as follows:
C.+1 Cn + ( softmax, (BKQ")Q al D softmax ( (kia((KK"))
KK ))) K w
(11)
where 7 > is a step size.
In practice, we empirically observed that the nonzero a in (8) often leads to an over-penalization of contexts, which can ultimately cause some contexts to vanish. To address this, we set = 0. Moreover, we found it beneficial to assign distinct step sizes Yattn and Treg as follows:
Cn+1 = Cn + Tattn softmaxy (3KQ") QW' Yreg D( softmax ( diag (K KW
Attention
Regularization
(12)
where the first and second terms are named as attention and regularization term, respectively.
Our theoretical observations offer valuable insights into implicit energy minimization by modifying the forward path of cross-attention. Inspired by these observations, we have transplanted the energy- based cross-attention layers to pre-trained text-to-image diffusion models. We first illustrate the challenges associated with adopting EBCA in a deep denoising auto-encoder and subsequently demonstrate how to overcome them practice.
If a single recurrent Transformer block were available, single energy function would be minimized for a given cross- attenntot layer by recurrently passing the updated context Cn+1. However, in practical applications, there are multiple layer- and time-dependent energy functions in conventional deep denoising auto-encoder, which makes it infeasible to minimize each energy function individually. To address this challenge, we implicitly minimize a nested hierarchy of energy functions a single forward pass based our theoretical observations. More details are followed.
5

---

Algorithms. At given time step t. we initialize the context vectors C1, with text embeddings CCLI obtained from a pre-trained CLIP. Then, within the n-th cross-attention layer with c,, (1 < n < L.), we compute the updated context vectors Cn+I, using the gradients in Theorem 1 and
(12 We then cascade I,+I,t to the next (n + 1)-th cross-attention layer. We do not forward the nnnal context C 1+ at time t to time t + 1, as the distributions of Q; and Q++1 can be significantly different due to the reverse step of the diffusion model. Instead, we reinitialize C1,++1 with CCLIP.
The pseudo-code for the proposed framework is provided in appendix B
The sequence of energy- based cross- aasss-attentoon layers is designed to minimize nested energy functions. While single-layer update in a single step may not lead to optimal convergence, the proposed layer- cascade context optimization can synergistically improve the quality of context vectors. Notably, our proposed method does not incur additional computational costs in practice since the gradient term in Theorem D relies on the existing keys, queries, and attention maps the cross-attention layer, which can be obtained for free in the forward path.
3.2 Energy-based Composition of Queries
In addition the above key advantages, the cross-attention space EBMs shed new light on the zero-shot compositional generation. Recent studies [8 10] have demonstrated that the underlying distributions of EBMs can be combined represent different compositions concepts. For example, given a data point æ and independent concept vectors Ci...., Cn. the posterior likelihood of æ given conjunction of specific concepts is equivalent the product of the likelihood of each individual concept as follows:
p( (i,..., Cm) = p(x p( c:) xe E.
II
(13)
where each E(x c;) represent independently trained EBM with concept code c.. While is an appealing solution to the controllable generation, it is notoriously difficult to train EBMs (|] [7 in way to make themselves scalable to high-resolution image generation. Instead of directly training EBMs in pixel space, we leverage the cross-attention space EBMs and the generative power of state-of-the-ard pre-trained DMs to achieve high-fidelity compositional generations.
More specifically, assume that we have main context vectors C₁ embedded from a main prompt, e.g. "a castle next to river". and a set of independent editing context vectors, C (C2...., C.). each embedded from different editorial prompt, e.g. "monet style", "boat on river", etc. Then,
define the keys Ki,, for context s within a cross-attention layer of index as Ki, = C.W... K.W.i, Vs E {1,2..... M). The index will be omitted for notational simplicity. Then, for given key K,, s. we consider the energy function in cross attention space w.r.t queries Q:
E(Q; K.) = diag QQ")QQ))
Σ logsumexp(K.qq. K
(14)
2
which is directly motivated by 13 concept conjunction of C' as in
We then introduce the compositional energy function E,, for the and the updated rule as done in (6)
13
M
B(Q; (K, (K,) M
B(Q; K.)
M
(15)
M
p
diag (QQ")
ΣΣ
logsumexp(K' .
= 2
M
a, softmaxg (BQr TF.s) V
(16)
M
where QTF. KTF, and V VTF directly follows the definition in (6) and the degree and direction of s-th composition can be controlled for each concept individually by setting the scalar as, with a, < for concept negation g Note that this is exactly linear combination of transformer cross-attention outputs from different contexts with = 1/ Vdn. We refer to this process as the Energy-based Composition of Queries (EBCQ).

---

Cumulattive posteror energy difference
Layerwiie energy dynamics
Stable-Difuson
Stable-Diffusion
"A monkey and hat"
Figure 3: Energy analysis for multi-concept image generation. The right graph displays E [ K) across 16 cross-attention layers within each sampling step. The red and green lines correspond the Stable Diffusion and ours, respectively. Mean and standard deviation are calculated and displayed. The bottom plot shows the cumulative sum of the posterior energy difference between Stable Diffusion and the proposed method, and intermediate denoised estimates are displayed together.
This update rule implies that the compositional energy can be minimized implicitly through modifica- tions to the forward path of EBCA, thereby guiding the integration of semantic information conveyed by editorial prompts. Notably, this is a significant departure from the existing energy- composi- tional methods [8 26 9] Specifically, no training or fine-tuning is required. Instead, cross-attente cross-attention outputs of the main and editorial prompts are simply obtained in parallel, averaged. and propagated to the next layer. Moreover, the introduction of EBMs in cross-attention space is orthogonal to [22]. which conducts compositional generation by treating a pre-trained diffusion model cø- itself as an implicitly parameterized EBM.
Experimental Results
To verify our claim of energy minimization via modified cross-attention, we conduct various experi- ments in text-guided image generation to verify semantic alignment, namely (1) multi-concept gener- ation [22] (2) text-guided image inpainting [23)44], and (3) compositional generation [1 28
which includes real and synthetic image editing. In this work, while the first and third applications address similar challenges, we categorize them separately based on the implementation derived from the energy- mnase interpretation. Specifically, the Energy-based Bayesian Context Update (EBCU) is applied to every task, and the Energy-based Composition of Queries (EBCQ) is additionally leveraged in the third task. Note that all applications have been done without additional training.
Setup. The proposed framework can be widely mounted into many state-of-the-arr text-to-image DMs due to its unique functionality of context updates and cross-attention map compositions. Here, verify the effectiveness of energy-based cross- cross-attention with Stable Diffusion (SD) [31]. The SD is an LDM that is pre-trained followed by the subsequent fine-tuning on the LAION-aesthetic dataset. For the text embedding, use a pre-trained CLIP (29 model following the Imagen [33]. The pre-trained SD is under the creativeML OpLnRAIL license. Detailed experimental settings are provided in the appendix.
subset of the large-scall scale image-language pair dataset, LAION-SB [4
4.1 Analysis on Energy during the sampling
We perform comprehensive analysis on ( and (9) during the forward path through the modified -ttsstint,,toon, which offers insights into energy behavior for real applications. Specifically, we examine the energy dynamics involved in the multi-concept image generation that is straightforward and could readily applied to other applications with minimal effort. We record computed energy values along layers and sampling steps 30 times with fixed text prompt and then display them in Figure 3 with the generated samples, where red and green lines denote the energy involved with the Stable-Diffusion (SD) and the proposed method, respectively. For each sampling step block that is alternately shaded, the energy values are plotted through 16 cross-attention layers.
Across all layers and sampling steps. the energy associated with the proposed method remains consistently lower than that of the SD. This denoised estimates and the given text prompt. In both cases of the SD and the proposed method, E(Q; K) decreases over sampling steps. This implies that the iterative refinement of the updated query carried over to subsequent sampling steps, resulting in a better match to the given context. Note that the proposed method even achieves lower energy with the EBCU. More analyses are provided in the appendix E including the ablation study.
line with the semantic alignment between intermediate
7

---

"A cat wearing a shirt'
"A lion and crown'
Stade Dff
Stucture Cif
Cura
Figure 4: Multiconcept generation comparison. Each row indicates the Stable Diffusion, the Struc- tured Diffusion, and the proposed method applied the Stable Diffusion, respectively. Generated samples in the same column used the same text prompt and the same random seed.
Input
Stabls Ropainn
Snablo-Inpain
Ours
Masked
Slatio-Repant
Stable-rpairt
Bow to. bythoday Tal
beer
Straworryy dake. candle
Tabla lamp
Figure 5: Text-guided inpainting comparison. The masked region is conditionally generated by given text-prompt positioned on the left-end of each row.
4.2 Multi-Concept Generation
We empirically demonstrate that the proposed framework alleviates the catastrophic neglect and attribute binding problems defined in the existing literature 4 As shown in Figures and the EBCU effectively mitigates these problems by promoting attention to all relevant concepts the prompt. Specifically, the regularization term introduced by the prior energy prevents a single context from dominating the attention, while the attention term updates the context vectors in the direction of input queries, improving semantic alignment and facilitating the retrieval of related information.
4.3 Text-guided Image Inpainting
In addition, we have evaluated the efficacy of the proposed energy-based EBCU text-guided inpainting task. Although existing state-of-the-at DMs such as DALLE and SD can be employed for inpainting by exploiting the ground-truth unmasked area (23), they usually require computationally expensive fine-tuning and tailored data augmentation strategies (3
contrast, as shown in Figure 5 our proposed energy-based framework significantly enhances the quality of inpainting without any fine-tuning. Specifically, we incorporate the energy-based cross-attention into the Stable- Repaint (SR) and Stable-Inpaint (SI) models, which can struggle to inpaint multiple objects (e.g.. birthday hat and bow tie) or unlikely combinations of foreground and background objects (e.g.. Teddy bear the Starry Night painting). In contrast, the proposed approach accurately fills in semantically relevant objects within the target mask region.
4.4 Compositional Generation
We demonstrate that the proposed framework improves the controllability and compositionality of DMs, which significant challenge for generative models. To assess this, we split the task into synthetic and real image editing.

---

DDIM-nvarsion
Synthelic image
Composable-
Image
swap
Plugrand Play
Tannke Oal
Penn
GOUD
boor
Batty fox Good
A photo of
the lace of
worren
photo of
Chicken - Pigeun
faca
random
person
seap
picture of
Dag Loonneyy
a car
[-] nounbe
Figure 6: Image editing comparison. The left three columns show examples of synthetic image editing. while the right four columns show examples of real image editing. To ensure a fair comparison, the same editing prompt, positioned on the left-end of each row, is given to all methods being compared.
Real Image
baby
'in
woman"
"old woman"
"old man'
Real Image
"baar"
'Eon"
"hoise"
"al pastel
*whbo wol
Real image editing with multiple editing prompts. Given real image at the left-end of each row, the proposed method allows robust image editing with different prompts over each sample.
Figure 7:
Synthetic image editing, Text-to-image DMs provide impressive results, but it is difficult to generate images perfectly aligned with user intentions [ Although modifying prompts can guide generation, it can also introduce unintended changes in the generated content. Our results, shown in Figure 5, demonstrate that the proposed method can edit image contents while maintaining the original-prompt- related identity of the generated images thanks to the EBCU, which continuously harmonizes the latent representations and text prompts. While composable-dif' (22 can generate compositions, they often fail to preserve the original properties of images, such as the color in the second row, or fail to compose the desired concept, such as the boat in the first row.
Real image editing. We demonstrate that the proposed framework can also edit real images, which especially challenging existing methods often dramatically alter input content introduce unexpected variations. To do this, we integrate our framework with DDIM inversion [35 25 which inverts images with meaningful text prompts into the domain pre-trained diffusion models. First, we use the image captioning network BLIP 20 to automatically caption the interested image, following 28 Next, we obtain the inverted noise latents of the input image using Diffusion Pivotal Inversion 25 Then, we apply EBCQ and EBCU while denoising the inverted latents. The optimized unconditional textual embedding vector is also targeted for additional EBCUs. The results in Figure 6 demonstrate that our method achieves better editing performance by avoiding undesired changes.

---

Quantitative comparisons. We further conducted comparative analysis of the proposed framework
against several state-of-the-art diffusion- based image editing methods [24 12 25 28 For our evaluations, we focus two image-to-image translation tasks. The first one is an animal transition task, which translates (1) cat - dog. (2) horse - zebra, and (3) adding glasses to cat input images (cat + cat with glasses). The second one is human transition task, which translates (1) woman -> man, (2) woman + woman with glasses, and (3) woman + man with glasses. Source images are retrieved from the LAION 5B dataset (34] and CelebA-HQ [15] Motivated by [39 28), we measure CLIP Accuracy and DINO-ViT structure distance. Experimental details are provided in the appendix including explanations of baselines, datasets, metrics, and hyperparameter configurations.
Table (Animal transition) Comparison to state-of-the-an diffusion--aan basedd editing methods. Dist for DINO-ViT Structure distance. Baseline results are from (28
(a) Cat
Dog
(b) Horse
Zebra
(c) Cat
Cat w/ glasses
Method
CLIP Acc (*)
Dist (U)
CLIP Acc (1)
Dist (L)
CLIP Ace (1)
Dist ({)
SDEdit [24] word swap
71.2%
0.081
92.2%
0.105
34.0%
0.082
DDIM word swap
72.0%
0.087
94.0%
0.123
37.6%
0.085
trompp-lo-prompt [12
66.0%
0.080
18.4%
0.095
69.6%
0.081
pix2pix-zero 28
92.4%
0.044
75.2%
0.066
71.2%
0.028
Stable Diffusion ours
93.7%
0.040
90.4%
0.061
81.1%
0.052
DDIM Inversion
Source
word
PixaPix-zero
Ours
Source
Cycle Diff
Pix2Pix
Ours
Cat
Man
Hons 2
Women
Cit w/
Man
Figure 8: Real image editing on (a) animal transition and (b) human transition tasks.
Table I and in the appendix show that the proposed energy-based framework gets high CLIP-Acc while having low Structure Dist. It implies that the proposed framework can perform the best edit while still retaining the structure of the original input image. While DDIM word swap records remarkably high CLIP-Ace in horse zebra task, Figure & shows that such improvements based on unintended changes in the overall structure. More visualizations are provided in the appendix D
5 Conclusion, Limitations and Societal Impacts
Conclusion. In this work, we formulated the cross-attention with an energy perspective and proposed the EBCU, a modified cross-attention that could implicitly minimize the energy in the latent space. Furthermore, proposed EBCQ which is inspired by energy energy-based formulation of multiple text composition. The proposed method is versatile as shown by multiple applications, theory grounded, easy to implement, and computationally almost free.
Limitations and Societal Impacts. The framework presented in this study generates images based on user intentions, which raises concerns regarding potential misuse for creating deepfakes or other forms disinformation. regulated appropriately. Additionally, while the framework performs well across various tasks, it requires pre-trained deep models, rendering challenging to apply to out-of-domain datasets, such as medical images.
is crucial to ensure that these methods are implemented ethically and
10

---

Acknowledgments
This research was supported by the KAIST Key Research Institute (Interdisciplinary Research Group) Project, National Research Foundation of Korea under Grant NRF-2020R A2B5B03001
Field-oriented Technology Development Project for Customs Administration through National Re- search Foundation of Korea(NRE) funded by the Ministry of Science & ICT and Korea Customs Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safery) (Project Number: 1711137899, KMDFF_RR 20200901 _001)), Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea (ove))
Service(** I I
No.2011-0-0000
Institute of Information & communi- cations Technology Planning & Evaluation (ITTP) grant funded by the Korea governmen((MSIT) (No.2021-0-02068 Artificial Intelligence Innovation Hub), Institute of Information & communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2023-00233251, System3 reinforcement learning with high-level brain functions), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (MRF-2009MM550224400666661)
Artificial Intelligence Graduate School Program(KISTT)
References
[1] M. Brack, Friedrich, D. Hintersdorf, L. Struppek, P. Schramowski, and Kersting, Sega: Instructing diffusion using semantic dimensions. arXiv preprint arXiv:2301. 12247, 2023.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] Caron, H. Touvron, Misra, Jégou, Mairal, Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. pages 9650-9660, 2021.
[4] H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-On Or. Attend-and-excite: Attention basod semantic guidance for text- to-image diffusion models. arXiv preprint arXiv:2301.38806 2023.
[5] Y. Choi, Y. Uh J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8188-8197, 2020.
[6] M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and Vermet. On model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-299, 2017.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[8] Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models, Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
[9] Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models, arXiv preprint arXiv:2012.01316. 2020.
[10] Y. Du and Mordatch. Implicit generation and generalization in energy-hased models. arXiv preprint arxiv: 993.08689, 2019.
[11] W. Feng, X. He, T.J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang. Fraening-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2112.00032, 2022.
[12] A. Hertz, R. Mokady, Tenenbaum, K. Aberman, Pritch, and D. Cohen-Or. Prompt-to-prompl image editing with cross attention control. arXiv preprint arXiv:2208.01622 2022.
[13] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-653 6851, 2020.

---

[14] B. Hoover, Y. Liang, B. Pham, R. Panda, H. Strobelt, D. H. Chau, M. J. Zaki, and D. Krotov. Energy transformer. arXiv preprint 2rXi::3302.02533, 2023.
[15] H. Huang, R. He, Z. Sun, T. Tan, et al. Introvae: Introspective variational autoencoders for photographic image synthesis, Advances in neural information processing systems, 31, 2018.
[16] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of inffusion-based generative models. In Proc. NeurlPS, 2022.
[17] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 1401144100 2019.
[18] D. P. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. arXiv:2107.0066: 2021.
arXiv preprint
[19] D. Krotov and J. Hopfield. Dense associative memory is robust to adversarial inputs. Neural computation, 30( 10112)311113116, 2018.
[20] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language- pre- pre-training for unified vision- language understanding and generation. In International Conference on Machine Learning, pages 12888- 12900. PMLR, 2022.
[21] L. Liu, Y. Ren, 7. Lin, and Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint axXiv2200.009777 2022.
[22] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XVII, pages 423 439. Springer, 2022.
[23] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[24] C. Meng. Y. Song, J. Song, J. Wu, J.-Y. Zhu, and Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0107. 2021.
[25] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprini 1.X19:2211.09794, 2022.
[26] W. Nie, A. Vahdat, and A. Anandkumar. Controllable and compositional generation with latent-space energy- based models. Advances in Neural Information Processing Systems, 34:13497-13510 2021.
[27] E. Nijkamp, M. Hill, S.-C. Zhu, and Y. N. Wu. Learning non-convergent non-persistent short-run meme toward mnergy-bascd model. Advances in Neural Information Processing Systems, 32, 2019.
[28] G. Parmar, K. K. Singh, R. Zhang, Y. J. Lu, and preprint arXiv:2302.0m 03027, 2023.
J.-Y. J.-.. 7hu. Zero-shot image-to-image translation. arXiv
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[30] H. Ramsauer, Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L.. Gruher, M. Holzleitner, M. Pavlović, G. K. Sandve, et al. Hopfield networks is all you need. arXiv preprint 20X.0220 2020.
[31] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684 10695, 2022.
[32] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer- Aasister-Assisted Intervention-MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 231-241. Springer. 2015.
[33] C. Saharia, W. Chan, S. Saxena, L. Li. J. Whang. E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan. T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 01136677334994 2022.
12

---

[34] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, al. Laion-Sb: An open large-scale dataset for training generation image-text models. arXiv preprint urXiv:2210.88002. 2022.
[35] J. Song. C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arX:v:2010.02502. 2020.
[36] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[37] Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint urXiv:2011. 2020.
[38] N. Tumanyan, O. Bar-Tal, S. Bagon, and T. Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision Pattern Recognition, pages 10748 10757, 2022.
[39] Tumanyan, M. Geyer. Bagon, and Dekel. Plug-and-play diffussio features for text-driven image- to-image translation. arXiv preprint arXiv:2211.12522 2022.
[40] C. Wu De Torre. Unitying diffusion models' latent space, with applications to cyclediffusion and guidance. arXiv preprint arXiv:2210.05559, 2022.
[41] S. Xie, Z. Zhang, Lin, T. Hinz, and K. Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.5003 2022.
[42] C. Ye. Geometry of Deep Learning. Springer, 2022.
[43] A. L. Yuille and A. Rangarajan. The concave-convex procedure (ccep). Advances neural information processing systems, 14, 2001.
13

---

A Proof of Theorem 1
In computing derivative of scalar, vector, matrix with respect scalar, vector, matrix, should be consistent with the notation. In this paper, follow the denominator layout convention deserihed in [42 To make the paper self-contained, we briefly introduce denominator layout associate calculus.
main motivation for using denominator layout is from the derivative with respect the matrix. More specifically, for given scalar and a matrix W E I2 1n according to the denominator layout, we have
dc
Bc thri
8we11
dc aW
mm
(17)
Oc Ommm
Bc Surmon
Furthermore, this notation leads the following familiar result:
Oa' x
Ôx a
a.
(18)
dæ
Accordingly, for a given scalar and a matrix W mxx,,, we can show that
dc
Oc
dw = UNVEC
@VEC(W))
E R
(19)
order to consistent with (17). where VEC() and UNVEC() refer vectorization operation its
reverse, respectively. Under the denominator layout notation, for given vectors æ 1 and y E Ie",, the derivative of a vector with respect to a vector given by
8yn
32
dy
E R mxn
(20)
Ōæ
8yn
8ym
8m
Then, the chain rule be specified follows:
@e(g(u))
du Ög(()) ac(g) da
(21)
äg
Finally, the following property of Kronecker delta product is useful throughout paper
VEC(ABC) (C" & A)VEC(B)
(22)
(A@ B) A B"
(23)
Using this, we first prove the following key lemmas.
Lemma 1. For given column vector R,, have
logsumexp(az, ()
softmax(3x)
dx
Proof.
Aaalldd Σ Aapplan
logsumexp(a; 0) 3-1 log DN exp(3x,)
Oæ
Ox
expl pa) E exp(day)
=oftmax(Bar (3x)
Q.E.D.
Lemma Let k. denote the i-th row vector of K Nxd Then, we have
ōk,k!
2e' (Tx) K
0K
where e'y represents a N d-dmennsional column vecior where only the i-th entry is 1, with all other entries set to zero.
14

---

Prouf. First, note that the expression &y can be equivalently rewritten as K'
Then, we have
k = (eix) KK" eiN
Furthermore, using 22 we have K" e'eyx = ((ex)) & (a)Ve((K")) that
OK @VEC(K") (e'v Ia)
where Ja denotes the d d identity matrix. Using the chain rule, have
ōk,k
ōk,k? 2(e Ia)K e/x
(VEC(K") (VEC(K") OK
Thus, we have
UNVEC
(VT)
ōk,k?
2UNVEC (lex @ Ia)k" -KK ((y(e
where we again use 22 Therefore, by taking the transpose, we have
kk, (ex( K. 0K
Q.E.D. Lemma 3.
V (etssumexp(QKf..1 - (ssssttax((3333
(24)
Proof. Note that
Q&? QK e'x QK ((e'v)" V)(CK((
Hence, using 18
have
OQK* e¹ (VEC(KT) = e @ @Q
(25)
Furthermore, using Lemma
we have
logsumexpl QKT
OQK? (VEC(K*)
3 logsumexpp (Qk?.
(VEC(K) KT)
aQk?
= (e'y & Q")
softmax
leading to the following:
0 logsumexp(QKK UNVEC
a Blogsumexp(Q2 Qk? ,8) OVFC(K")
0K"
UNVEC (ee'x @ Q") softmax (3Qhi
= Q" softmax (BQA, (ENN)
where we again use
22
Now, by taking the transpose, we can prove 24 Q.E.D.
Lemma 4.
i diag (KKT)
2K
(26)
aK
[softmax(=)]
0
Olog N exp( k
K
(27)
0K
[softmax(x)]
where diag ()) denotes sum of the diagonal element, i.e. trace of and æs column matrix whose i-th element is given by k.kk." /2, and [)] dentoes the i-th element.
Proof. First,
diag (KT) Σ K?
4,
15

---

where Kij denotes the (i.j)-th element. Therefore, using the denominator layout in 117 it is trivial to show that
de 2K.
0K
Second, let us construct column vector æ: whose i-th element is given by 2s == k, /2. Then, using Lemmas and]aand the chain rule, we have
@ log > 1 exp( k,,k
-
Or. d logsumexp(2;1) 1)
0K
OK
Ou:
Σ (el)) K [sftmax(
Σ (softmmaxxx)))
[sotmaax(1))
K
[oflmax(())]
This concludes the proof.
Theorem For the energy functions
K) diag( K. (KK) K"K
logsumexp(Qke ,9)
(28)
and
N E(K) log Σ exp( k.k?)
(29)
the gradient of the log posterior is given by:
K log Dogp(K K Q) softmaxa (BKQ")Q al ( softmax ( (iag((K')) (ogg(K"'
K,
(30)
Then, by using the chain rule the update rule of context vectors C is derived as follows:
Ca+1 = C. + softmax2 (BKQ") (I D softmax ( diag KK") (2RK ())))
W
(31)
where y is step size, and D()) aveetoo-tt--iigggndddmmrrr---- operatrr
Proof. Based Bayes' theorem, gradient the log posterior is derived as:
V logp(K Q) (V E(Q K) E(K)).
(32)
Using Lemmas 123 and 4 we have
V B(Q:K) aK - Ee' (30tmaxx3333) )
aKK - softmax
(33)
Second, by noting that E(K) log En
Lemma informs
[softmax(I))]
0
K B(K)
K
2oftmax(x))
D softmax 2 diag K k²)))
diag(K
where D(-) is vector-toddiggonal-matt operator that takes N-dimensional softmax vector as an input and returns a N diagonal matrix with softmax values as main diagonal entries. Therefore, one can finally obtain:
V loggl K Q) = softmax, (BKQ")Q ( [
D softmax (5 diag K (K")))
K.
(34)
By using the chain rule with K CW k. the update rule of context vectors C' is derived as in
11
16

---

B Pseudo-code for EBCU and EBCQ
This section provides the deseription of the pscudocode for the proposed Energy-based Bayesian Context Update (EBCU) and anergy-based Composition of Queries (EBCQ). Algorithm outlines ooutiins the cascaded context propagation across cross-attention layers within the UNet model during the sampling step Note that the context is reimitialized at the beginning each sampling step. EBCU implemented in each cross-attention layer. Specifically, the proposed EBCU provides significant computational efficiency by reusing the similarity QK which requires computational cost ( )) to compute E(E K). Consequently, there is only small amount of additional computational overhead associated with the proposed EBCU.
other hand, Algorithm 2 details
Algorithm 1 Context cascade at sampling step t
Require:
Q Colip, UNet
I: Ct Cets // Re-initialize
for layer UNet do
3: if layer is CrossAttention then
4:
Q, C₁ + layer(Q,, C.) // Algorithm 2
5:
else
Q, + IQ.)
7:
end if
8: end for
9: Q Q
10: return Q,+1
Algorithm Energy-based Bayesian Context Update (EBCU)
Require: Q, C, W W. W O. 8 Tain:
I: Q, K, V +- QW q: CW CW.
QK?
3: Q + s3f)
VxE(Q;K) softmaxa (35") Q
5: VKE(K) -aI I D (softmax diag( (iag(KK (KK))) )))
6: AC -Naa E(Q; K) + Trg W
7: C <- C C+AC AC
8: return Q
Algorithm 3 outlines the pseudocode for the EBCQ implemented for M given contexts. For the simplicity, we exclude the EBCU from the algorithm. Nontheless, the EBCU and the EBCQ could be leveraged together.
Algorithm 3 Energy-based Composition of Queries (EBCQ)
Require: Q. [CI,..., CM), W WA, W as
1: Q - QW, 9
2: for in [1 1I....,M. M....,Mer do
3: K,. V. f- C. W.W C.W,
4: S. = QK?
5: end for
6: Q+ M
EM a.softmax
7: return Q
C Experimental setups
In this section. we describe detailed experimental setups for three applications including baseline method, paper-parameter of the proposed method, and dataset if it is the case. Code: httpss//githu: com/ tberg//ttaattio///Eeggggaaaad..t waw.aaaaarrrrreeeetttiiiii
C.1 Common experimental setup
We mainly leverage pre-trained Stable Diffusion v1-5 (except Table | v1-4) which is provided by diffusers, Python library that offers various Stable Diffusion pipelines with pre-trained models. All images sampled for
17

---

50 steps via PNDM sampler [1] using NVIDIA RTX 2080Ti. In every experiment, we set the parameter in Equation (9) zero, focusing solely controlling the values of Tattn and Treg- EBCU is applied every task, and EBCQ is additionally employed in C.4
Different learning rate for each token It worth noting that the Tazto and "/reg could be expressed as vectors. In other words, if the context C' E R. Nxde is given, Tatte and "reg are N N-dimensional vectors. Hence, we have the flexibility to adjust the learning rate Tt-)- allowing us increase or decrease the impact of certain tokens based on the user's intent. Unless otherwise noted, "Ytttn and Treg is set a constant for each text token.
Learning rate scheduling Since the proposed EBCU is leveraged the diffusion model, one can readily introduce scheduling strategies for Yottn and Yreg along the sampling step t. We implement multiple variants such as 'constant', 'step", and 'exponential decay" as follows.
[constant]
?(t) =
[step] 7() = 7/o ReLu(l 7)
(35)
[exp-decay] (() = X
where "/o is the initial value, ReLu(z) = 0 if x < 0, otherwise 1, T denotes the temporal threshold, and A denotes the decay ratio. Unless stated otherwise, the schcduling strategy is set to the "constant'.
C.2 Multi-concept image generation
We compared the performance of the proposed method with Structured Diffusion which does not require additional training as our method. We leveraged the open-sourced official implementation !
For the proposed method, we set the "Yottn and Trey differently for each sample within [le-2, 1.5e-2, 2e-2]. As shown in the following ablation studies E large Yottn tends to generate saturated images while large "reeg results mixed/vanished contents.
We found that using diffcrent learning rates for each context token useful for multi- generation, especially when single concept tends to dominate with a constant learning rate. For example, given the main prompt "A cat wearing a shirt", we set the Tatta for the "shirt" to 3e-2, while Tattn is set to 1.5e-2 for other tokens. We have observed that doubling the Yattn for text token to be emphasized is sufficient to achieve balanced multi-concept image generation for most cases.
C.3 Text-guided image inpainting
Additionally, we conducted performance comparison between our proposed method and two alternative approaches: (a) Stable Inpain which fine-tunes the weights of Stable Diffusion through inpainting training, and (b) Stable Repaing which leverages the work of Lugmayr et al. 22 on the latent space of Stable Diffusion for the inpainting task. In the case of Stable Repaint, the mask is downsized and transferred into the latent space. We applied the Energy-based Bayesian Context Update (EBCU) technique to both methods, resulting in improved results compared to their respective baselines.
Masked EBCU. To further enhance the performance for the inpainting task, we introduce the concept of
masked Energy-based Bayesian Context Update (masked EBCU). Specifically, let M E Pf xPf represent diagonal matrix where the main diagonal values are derived from the downsampled inpainting mask for the I-th
cross-attentin layer, with an output spatial size of Pé In Equation (36). we modify the attention term (2) by incorporating the downsampled mask, effectively covering the query matrix as follows:
C.+1 7C softmax, (3KQ") MQ (al+
aI + D softmax (₂ ((la((X*)))) K W (36)
the attention term updates the context vectors, aligning k; towards q,. 1,..., P?P. while considering the alignment strength between each q, and k. However, in the inpainting task, we have prior knowledge that the context vectors should be most aligned with the semantically relevant masked regions. Therefore, we mask out unrelated background spatial representations, allowing for the context vectors be updated with a specific focus on the masked regions. This approach facilitates the incorporation of semantic information encoded by k; specifically into the spatial mask regions.
As evident in Equation 12
In our proposed method, we set different values for Taten and "Feg for each sample, selected from the set [le-2, 1.5e-2, 2e-2, 2.5e-2], to account for variations in the input samples.
https: htte:///tttaaaooosssssssssattttt
Guidacee
http:s//gggtbbbocooooooggg/go//r http:///oottttrrooomomm///////// comuutis////
18

---

C.4 Image editing via compositional generation
We present empirical evidence demonstrating the effectiveness of our energy-based framework for compositional synthetic and real-image editing. The Energy-based Bayesian Context Update (EBCU) technique can readily applied to both the main context vector (C' in Section 3.2, = 1) and editorial context vectors (C,s1). Each EBCU operation influences the attention maps used in the Energy-based Composition of Queries (EBCQ), enhancing the conveyance of semantic information associated with each context. Note that ax in [6 represents the degree of influence of the s-th concept in the composition. In practice, we fix 0n for the main context, while 0221 is tuned within the range of (0.5, 1.0).
Let "aeten.s and "reg. denote the step sizes for EBCU of the 8-th context vector. If the editing process involves changing the identity of the original image (e.g., transforming a "caa" into a "dog"), ) we set both "faMt,1 and Jreg.1 zero. Otherwise, if the editing maintains the original identity, we choose values for Tasto,1 and "reg. from the range of (5e-4, le-3), similar to (st1n.(>>1) and (1-)) All hyperparameters. including ax, and 7/.. are fixed during the quantitative evaluation process (more details in Section and Table
To ensure consistent results, we maintained fixed random seed for both real and synthetic image editing. For real image editing, we employed null-text pivotal inversion 25
to obtain the initial noise vector.
During the reverse diffusion process in Sections C.2 and C.3 we kept y fixed as a constant value. However, for compositional generation, we utilized step scheduling (Equation 35 for y and a... After converting the initial noise vector for real images or using fixed random seed for synthetic images, EBCU and EBCQ are applied after threshold time T. > 0 for the s-th editorial context. This scheduling strategy helps preserve the overall structure generated images during the editing process. our observations, value of T, E [10, 25] generally produces satisfactory results, considering a total number of reverse steps set to 50, However, one can increase or decrease for more aggressive or conservative editing, respectively.
The exemplary real images presented in Figures 5 and 6 of the main paper were sampled from datasets such as FFHQ 117. AFHQ [5). and ImageNet 77. For detailed quantitative analysis, please refer to Section D
D Quantitative Comparison
In this section, we conducted a comparative analysis of the proposed framework against several state-of-the-art diffusion-based image editing methods (24 12 25 28
following the experimental setup of (28] To ensure
fair comparison, all methods utilize the pre- trained Stable Diffusion v1-4, employ the PNDM sampler with an equal number sampling steps, and adopt the same classifier-fre guidance scale.
D.1 Baseline Methods
In addition to the Plug-and-Pla method discussed in the main paper, we include the following basclines for comprehensive quantitative comparison:
SDEdit [24] word swap. This method introduces the Gaussian noise of intermediate timestep and progressively denoises images using new textual prompt, where the source word (e.g., Cat) is replaced with the target word (e.g.. Dog).
Prompt-to-prompp (P2P) 12]. P2P edits generated images by leveraging explicit attention maps from source image. The source attention maps Mr are used to inject, re-weight, or override the target maps based on the desired editing operation. These original maps act as hard constraints for the edited images.
DDIM + word swap (25). This method applies null-text inversion to real input images, achieving high-fidelity reconstruction. DDIM sampling is then performed using inverted noise vectors and edited prompt generated by swapping the source word with the target.
pix2pix-zero (28). pix2pix-zer -zeoo first derives a text embedding direction vector Aeein from source to the target by using a large bank of diverse sentences generated from a statc-of-the-an art sentence generator, such as GPT-3 2 Inverted noise vectors are denoised with the edited text embodding, e Acaan and cross-attcntion guidance to preserve consensus.
Cycle-diffusion [40]. Cycle-diffusion reformulates diffusion models deterministic mappings from Gaussian latent code to images. presents a DPM-Encoderr that allows for encoding images into this latent space.
D.2 Dataset
For the animal transition task, we focus on three image-to to to-image translation tasks: (1) translating cats to dogs (cat + dog). (2) translating horses to zebras (horse + zebra), and (3) adding glasses to cat input images (cat + cat with glasses). Following the data collection protocol of (28), retrieve 250 relevant cat images and 213
19

---

horse images from the LAION SB dataset [34] using CLIP embeddings of the source text description. We select images with high CLIP similarity to the source word for each task.
For the human transition task, focus three amage-to-image translation tasks: (1)
man, (2)
woman + woman w/ glasses, and (3) woman - man glasses, For this, select source images of women without wearing glasses from CelebA-HQ. To ensure fair comparisons, all methods leverage the same version of Stable Diffusion, sampler, sampling steps, etc.
D.3 Metries
CLIP Accuracy and DINO- VIT structure distance. Specifically, (a) CLIP Ace represents whether the targeted semantic contents are well reflected the generated images. calculates the percentage
Motivated by [39 28
instances where the edited image has a higher similarity to the target text, as measured by
CLIP, than to the original source text [28]. On the other hand, (b) structure distance [39] 33 measures whether the overall structure of the input image is well preserved. It is defined as the difference in self-similarity of the
keys extracted from the attention module the deepest DINO-VIT [3 layer.
D.4 Details
The main context vector Cmainn encoded given a main prompt automatically generated by BLIP (00) In addition, the editorial context vectors Cere and Ctgt are encoded given the text descriptions of the source and larget concept, i.e. source and target prompt. For example, for cat -> dog task (cat - cat w/ glaases)), the source prompt "cat" ("cat wearing glasses "), and the target prompt is "dog" ("without glasses"). Then we apply EBCU EBCQ based on the obtained context vectors. Please refer Table for the hyperparameter configurations.
D.5 Results
Table aad 2 show that the proposed energy-based framework gets high CLIP-Ace while having low Structure Dist. implies that the proposed framework can perform the best edit while still retaining the structure of the original imput image. This is remarkable result considering
proposed framework is not specially designed for the real-image editing task. Moreover, the proposed framework does not rely on the large bank of
prompts and editing vector Ceai 2X which can be casily incorporated into our method.
While DDIM + word swap records remarkably high CLIP-Ace in horse + zebra task, Figure 9 and 16 show
that such improvements based unintended changes in the overall structure, Table 3 summarizes the hyperparameter settings for each task. Examples of results presented in Figure [7]
Table 2: (Human transition) Comparison to state-of-the-arr diffusion-based editing methods. Dist for DINO-VIT Structure distance.
(a) Woman Man
(b) Woman
Woman w/ glasses
(c) Woman + Man w/ glasses
Method
CLIP Ace
Dist (L)
CLIP Acc
Dist
CLIP Ace
Dist
CycleDiffusion
40
57.0%
0.156
74.0%
0.018
63.5% 93.9%
0.237
Pix2pix-zero
93.0%
0.043
66.0%
0.024
0.052
Stable Diffusion + ours
94.0%
0.035
99.0%
0.029
96.0%
0.037
Table 3: Hyperparameter configurations for each editing task. Each task index comes from Table
Mattn,main: and "fregmmain 0 as mentioned in section C.4 Note that Oare for the concept negation (related ablation study in Figure I1). denotes the warm-up up period for step scheduling in
and Section C.4
Task
Ossee
Otg!
M-mmnnn
Nattnnane
Treg.ore
Nattn,Agt
Treg
(a)
-0.65
0.75
0
5e-4
5e-4
6e-4
6e-4
25
-0.5
0.6
0
4c-4
4c-4
Sc-4
5c-4
(c)
-0.6
0.7
lc-3
le-3
lc-3
lc-3
17
20

---

DDIM inversion word swap
Real Image
DDIM inversion
Ours
Figure 9: Image editing comparison with DDIM-inversion. Generated samples by DDIM-inversion with word swap readily deviate the original data contents, while the proposed method avoids undesired changes.
E
Ablation study and more results
"A teddy bear"
Yattn
Yrag
0.0
0.005
0.01
0.015
0.02
0.025
0.03
D0
0.01
0.015
0.02
0.025
Figure 10: Ablation results for Nattn and Treg- All samples are generated from the same random noise.
21

---

Stable Diffusion
Yrea
Yatts 0
Tatte Treo 0
Stable Diffusion
Preq
Fatts
Fartu- * O
cal and dog
Real Image
CACAO only
BCU + CACAO
Real Image
CACAO only
BCU CACAO
Zebra
Cat Dog
Real Image
(+) Dog
(+) Dog (-) Cat
Real Image
(+) glasses
w/ glasses
(-) wio glasses
Cat - Dog
Cat Cat
Figure 11: Ablation study results. The first row shows multi-concept generation examples with varying Yattn and 'Freg while the second row shows real image editing examples with varying the usage of EBCU and EBCQ. The last row shows the effect of negative prompt for the image editing application.
Attention and regularization terms. To access the degree of performance improvement attained by the proposed EBCU, conducted ablation study for the attention and the regularization terms by regulating "aten and "reg for the text-guided image inpainting (Figure Top and the multi-concept image generation (Figure [T] From the Figure TO we can observe that the desired content is generated when proper range "aaton and Treg are given. Specifically, once "Yreg is set to a valid value, the EBCU consistently generate a "teddy bear" with various Taaten otherwise it generates background or imperfect objects. This result emphasizes the role of the introduced prior energy E( K). Furthermore, the Tieton also affects to the context alignment of the generated sample (for instance "ratte 0.025 and "ree 0.02), which highlights the importance of the introduced conditional energy function E Q; K). The same evidences could be found in the first row in Figure I which are the multi-concept image generation examples.
Synergy between EBCU and EBCQ. While both EBCU and EBCQ are designed from the common energy- based perspective, each operation originated from diffcrent energy functions E(K; Q) and É(Q: (K.) 1=), respectively. This fact suggests the synergistic energy minimization by combining the EBCU and EBCQ. which could further improve the text-conditional image generation. To investigate this further, we conducted an ablation study using real image editing application. Specifically, utilizing EBCQ and when combining EBCU with EBCQ. The second row in Figure [] is the result of the ablation study that shows fully compatibiliyy of the EBCU and EBCQ. Importantly, the incorporation of the EBCU improves the quality of the generated images. While the EBCQ alone effectively captures the context of the given editing concept, the addition of EBCU enhances the fine-grained details in the generated outputs.
compared the editing performance when solely
Importance of concept negation. Remark that a negative a, in (16) denotes the negation of given editing prompt. We empirically observed that the concept negation may significantly contribute the performance compositional generation. Specifically, for the image-to-image translation task in Table we apply both positive and negative guidance with the target (e.g. Dog) and source (e.g. Cat) concepts, respectively, following the degree of guidance denoted in Table The third row in Figure [1 shows the impacts of source concept negation in the image-to-ing features, c.g. eyes of the Cat, the negative guidance removes such conflicting existing attributes. This implics that the proposed framework enables useful arithmetic multiple concepts for both real and synthetic image editing.
translation task. While the positive guidance alone may fail to remove the
source-concept-rellle
Prior energy and a. While E <diag(K (KKT) in penalizes norm of each context vectors uniformly, the proposed prior energy function B(K) adaptively regularizes the smooth maximum of || Intuitively, adaptive penalization prevents the excessive suppression of context vectors, potentially resulting images that are semantically aligned with given context. To demonstrate energy function, we conducted a multi-concept image generation task with varying a in (II) from 0 to 1, while fixing other hyperparameters. Figure 12 illustrates the gradual disappearance of salient contextual elements in the generated images depending the change of a. Specifically, the crown is the first diminish, followed by
effectiveness of adaptive penalization in the prior
22

---

subsequent context elements, with the lion being the last to vanish with 1. This result highlights the validity of the adaptive penalization for the context vectors which stems from the prior energy function.
Context shift analysis. Since the proposed framework updates the initial context vector, one may be concerned that the optimized vector may severely shifted from the original context so that loses the intended semantic meaning. Here, we emphasize that the change of context vector is adaptive, not an invalid shift, because the proposed BCU allows an adaptive propagation of the context vector through UNet, resulting in a context vector that is better aligned with Q, on cross-attention space.
To further support our claim, use the updated Or (T' for number of sampling time steps) from the proposed method as fixed context vector instead of Caip and perform multi-concept generation using the conventional cross-attention operation. Since the energy functions are defined differently each sampling time step, using Cr as a fixed context for each sampling time step may result in low www-quaiity samples. However, this approach allows us to evaluate whether the updated context contains the correct semantics of the given textual conditions, in contrast to Celi. Figure 13 demonstrates that the updated context vector does indeed capture the correct semantics of the given textual conditions (c.g., both black horse and yellow room).
Multi-step update. In order better understand the context update, we implement multistep context update for comparison while keeping the propagation of C' to subsequent layers. For the multi-concept generation and inpainting tasks, we observe that this multiple update of the context vector actually improves the quality of generated images (Figure [3p). This is in the same line as our perspective that one forward path of context vector is equivalent to one-step gradient descent for energy function. That being said, we also observe that the multiple context update is relatively computationally expensive and a single-step update is usually sufficient for improved performance. Therefore, we decide to use a single-step update with context propagation to subsoquent layers. We believe these results may lay down foundation for further research on understanding the alignment between latent image representations and context embeddings.
a=0.0
a=0.5
a= 1.0
Figure 12: Generated samples with varying a values. As a increases, the generated images progres- sively deviate from the intended context, "A lion and a crown".
(4)
Fixed Camp
Adaptive C,
Fixed Cy-50
stap
stops
stops
steps
Figure 13: (a) Muti-concept generation results with fixed and adaptive context embeddings. For the last column, updated context via the BCU at the final timestep is given as fixed context embedding. (b) Multi-concept image generation results a different number of context updates. Yattn and Tieg are reduced in proportion to the number of updates.
bawl, tomato Iemon
smile
Monet, Colorful, Sea
-matacch, digarette
Figure 14: Further results for compositional real-image editing.
23

---

"A black horse yellow room"
"A lion and crown"
Sosie OH
State D
Chos
Our
"A blue dog the orange sofa"
"A cat wearing shirt'
Soate DHf
Stade-DR DF
Qura
Our
*A plate blue flower"
"A green apple and red banana"
Saale-Off
Ours
Figure 15: Further results for multi-conen cotcept image generation. Best views are displayed.
24

---

DDIM inversion word swap
DDIM inversion
Real Image
Pix2Pax-zero
Ours
Real Image
word swap
Pix2Pix-zero
Ours
Figure 16: Further results for real image editing: horse to zebra.
25

---

DDIM inversion
DDIM inversion
Real Image
swap
Pix2Pix-zero
Ours
Real Image
swap
Pix2Px-zero
Ours
Figure 17: Further results for real image editing: cat to dog.
26

---

Synthetic Image
"Fall"
"Winter"
"Monet" "M
"Watercolor"
Real Image
"Rat"
"Cat"
"Dog"
"Tiger"
Figure 18: Further results for image editing with varying text prompts. Best views displayed.
27


Reviews:
[{"review_id": "4D40yHEGRJ", "reviewer": "Reviewer_nFGp", "summary": "This paper aims to optimize the context representation through energy based formulation of the cross-attention within the U-Net at test-time to achieve semantic alignment between the textual representation and the image features in the U-Net. \nExperiments are performed for the multi-concept generation, text-guided inpainting and image editing. ", "questions": "Please see weaknesses above. \n\nSpecifically following clarifications are required: \n\n1. The motivation for optimizing keys over queries. \n2. The role of the regularization term in Eq. 12. \n3. In theorem 1 the updates are wrt to the keys, however, eq 14 considers the energy wrt to the queries. The whole idea  of the paper in terms of novelty  from theorem 1, then why use eq. 14? \n\nThe results and comparisons with the baselines can be moved to the main paper. Theorem 1 and Eq 14 need to be better discussed (better combined in one section) and highlighting the contributions and the prior work clearly.", "limitations": "Limitations are adequately discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The motivation of the paper is clear and is supported with adequate experiments. \n+ The approach allows for compositional generation. The qualitative results show that the approach does better than simple DDIM+inversion. \n+ Multiple concepts can be composed together in the generated images with test-time optimization. \n", "weaknesses": "- The formulation of the gradient posterior in Eq. 9 is missing the term on the expectation of the gradient on the right side? \n- The difference and the motivation to optimize the energy wrt the keys instead of queries is not so clear? Ablations with the two approaches can be done to show the benefits and attention visualizations may help to evaluate the difference between the two. \n- The subscripts 1, 2 below the softmax are not defined. \n- In lines 150-151 what is meant by the \"forward path of cross-attention\"?\n- Low sample diversity: Example in figure 2 seems to harm the sample diversity wrt Stable diffusion.\n- What is the computational overhead? How many optimization steps are required to converge to a good solution?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The formulation of the gradient posterior in Eq. 9**. We would like to clarify that the notation E in eq. (9) denotes the energy function defined in eq. (7) and (8), not the expectation. We realize a typo in eq. (1) that should be corrected as \\mathbb{E}. We will fix eq. (1) in our revised paper.\n\n**W2: The difference and the motivation to optimize the energy wrt the keys instead of queries**. We would like to clarify that optimization for the energy w.r.t the queries have already been done by the cross-attention operation as described in proposition 1 and eq. (6). For the difference and the motivation to optimize the energy w.r.t key, please refer to the general comment 1.\n\n**W3, 4: The subscripts 1, 2 below the softmax, Forward path of cross-attention**. The definition for subscripts below the softmax is described in lines 64-68, section 2.2. The “forward path of cross-attention” means one operation of the conventional cross-attention, which is described in eq. (6). While eq. (6) only outputs updated query Q, the proposed BCU means updating the context vector C according to eq. (12) and propagating it to the subsequent UNet layer. Please refer to figure 1 in the rebuttal pdf file.\n\n**W5: Low sample diversity**. Regarding figure 2 in the main paper, it seems that the influence of $\\gamma_{reg}$ is significant, causing the monkeys to appear more grayish because the role of “reg” is to normalize the context. If we were to increase $\\gamma_{attn}$ by two-folds, the result would inherit the diversity of the Stable Diffusion (see figure 6 in the rebuttal pdf). We will include a discussion of this in our revised paper. That said, we would like to point out that the proposed method can generate more diverse images compared to Stable-Diffusion that  may not accurately reflect the context and lead to low diversity as shown in figure 3 in the main paper and figure 11 in the appendix. Thus, the proposed method proves to be more beneficial in generating a wider range of images through textual guidance.\n\n**W6: Computational overhead and optimization steps**. We would like to emphasize that the additional computational cost is negligible. Both Stable Diffusion and the proposed method take 7.0 seconds to generate a single image within 50 sampling steps with NVIDIA GeForce RTX 3090. Specifically, for the modified forward path, we reuse the $QK^T$ term (algorithm 2 in appendix B), which significantly reduces the additional computational cost. Also, we would like to clarify that we have not *explicitly* optimized image representations and context embeddings. The primary contribution of the proposed method is to establish a link between cross-attention operation and energy minimization. This allows us to effectively modify the cross-attention operation, resulting in adaptive propagation of the context embedding that better aligns the image features than a fixed CLIP embedding. Throughout our work, we have demonstrated that a single step of cross-attention, which is equivalent to one-step gradient descent for a defined energy function, is sufficient to improve performance for various tasks. In particular, we can observe the emergence of multiple concepts related to given textual conditions in the early phase of generation, which implies a fast convergence of the energy function (figure 2 in the main paper).\n\n**Q1: The motivation for optimizing keys over queries**. Please see the general comment 1.\n\n**Q2: The role of the regularization term**. The regularization term in Eq. 12 originated from the conditional energy function w.r.t K (7) and the role is described in lines 127-132. In summary, it constrains the energy of each context vector $k_i$, preventing it to explode during the maximization of logsumexp($Qk_i$, $\\beta$). Specifically, $\\gamma_{reg}=0$ in Fig 9 suggests that the norm regularization is necessary for the quality and semantic alignment of generated images. Our regularizer plays a role in preventing a single context vector from excessively dominating the forward attention path. For more details, please see the general comment 1.\n\n**Q3: eq. 14**. As mentioned by the reviewer, the primary contribution of the paper is theorem 1. However, we would like to highlight that the essence of the paper is the energy perspective interpretation of the cross-attention operation. In other words, what we have proposed in the paper are two different forward paths (each for query and key) of cross-attention layers that aim to minimize different energies (eq. 14 and eq. 9). Please refer to figure 1 in the rebuttal pdf.\n \nIn this context, we could readily extend the energy perspective interpretation to compositional generation by formulating the task as sampling from the posterior distribution as described in equation (13), leading to the novel energy (14) and method CACAO (16). It is important to note that CACAO is designed to focus on the query update, allowing for the effective incorporation of multiple textual conditions into image generation, while BCU enables adaptive context propagation. The compatibility of BCU and CACAO for compositional image generation is evident, as demonstrated by the ablation study result in the appendix.\n \nIn summary, the proposed energy perspective interpretation of cross-attention operation is a powerful tool that could be leveraged from a better understanding of the stable-diffusion model to improved performance for various applications.\n\n**Q4: move results to the main paper, better discuss Theorem 1 and Eq 14**. Agreed. Moving the comparison results to the main paper is beneficial and we will revise the manuscript accordingly. Also, we will consider re-organizing the manuscript per reviewers' suggestion to highlight the contribution and main motivations of this work. \n"}, {"review_id": "4SHYTOYjLo", "reviewer": "Reviewer_MvGR", "summary": "This paper proposes  Bayesian Context Update (BCU) and  Compositional Averaging of Cross-Attention Output (CACAO). The main idea is to view the cross-attention between text and image as optimizing an energy-based model, and modify the intermediate outputs according to the energy. A series of examples are shown to demonstrate the effectiveness of BCU and CACAO.", "questions": "Please refer to Weakness.", "limitations": "Limitations are properly discussed at the end of the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The examples are visually intriguing.\n2. The energy viewpoint of the cross-attention mechanism in Stable Diffusion could be enlightening.", "weaknesses": "1. I can understand that Q_t is the H * W * C dimensional feature in the network processing the latent-space image at time step t, and C_t is the context vector which is an optimizable variable added by the authors. C_t is initialized to C_clip, then it is optimized going through the layers. As the layers go deeper, C_t is shifted from C_clip, which means the semantic meaning is shifted from the original user text input. My question is: how severe is the shift? How do the authors control the shift? In fact, it makes more sense to me if Q_t,l are the variables to be optimized.\n\n2. More clarification on the examples in Figure 2 is required. I tried Stable Diffusion by myself, and found Stable Diffusion can generate \"A monkey and a hat\" with the correct semantic meaning. I kindly ask the authors to elaborate how the Stable Diffusion examples are picked. Moreover, Stable Diffusion + Ours degrades the image quality significantly (all the images become black-and-white). Could the authors provide some explanation on why that happens?\n\n3. More quantitative results should be added to prove the effectiveness of the proposed method. In the provided qualitative examples, the advantage of the proposed method over Composable-Diff and Plug-and-Play is marginal. Since the results of generative models are heavily dependent on the random seeds, I suggest the authors to provide systematic quantitative evaluation to benchmark the proposed method. I do notice the quantitative evaluation part in the Appendix, but only animal transition seems limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: How severe is the shift? How do the authors control the shift?**. Thanks for pointing out the important question that is deeply related to the motivation of our work. We would like to emphasize that the change of context vector is adaptive, not an invalid shift, because the proposed BCU allows an adaptive propagation of the context vector through UNet, resulting in a context vector that is better aligned with $Q_t$ on cross-attention space. For more details, please refer to the general comment 1. Moreover, we would like to remark that control of step size ($\\gamma_{reg}$) and a residual path for context update (Eq. (12)) can avoid deviating outside the appropriate semantics. \n\n**Experiment.** To further support our claim, we use the updated $C_T$ ($T$ for a number of sampling time steps) from the proposed method as a fixed context vector instead of $C_{clip}$ and perform multi-concept generation using the conventional cross-attention operation. Since the energy functions are defined differently for each sampling time step, using $C_T$ as a fixed context for each sampling time step may result in low-quality samples. However, this approach allows us to evaluate whether the updated context contains the correct semantics of the given textual conditions, in contrast to $C_{clip}$. Figure 2 in the rebuttal pdf demonstrates that the updated context vector does indeed capture the correct semantics of the given textual conditions (e.g., both black horse and yellow room). We will include this result in the revised paper to further strengthen the evidence for the effectiveness of the proposed method.\n\n**W2: More clarification on the examples in Figure 2, degradation of the image quality**. We would like to note that when the Stable Diffusion can generate correct semantic meaning, the improvement via the proposed method is marginal (refer to figure 7 in the rebuttal pdf). Therefore, we sampled multiple times with random seeds from 0 to 40, and improve the performance of the incorrect Stable diffusion sample cases.\n\nFor Figure 2 in the main paper, we recognize that increasing $\\gamma_{attn}$ is helpful to avoid gray image generation.  Please refer to Figure 6 in the rebuttal pdf. This is related to the role of $\\gamma_{reg}$ which is a regularization term. Finally, we would like to kindly note that our method does not significantly degrade the image quality. Please compare the 1st and 3rd rows of Figure 3. Indeed, the curated black-and-white images are the outcomes inherent to the behavior of the original stable diffusion model upon which our approach is founded.\n\n**W3: More quantitative results**.  Per the requests from reviewers, we further compare the performance of human face editing and measure additional metrics such as LPIPS and human evaluation to evaluate the quality of generated images. Furthermore, we compare the proposed method with additional baseline methods. Please refer to the general comment 2. Lastly, we would like to remark that the provided quantitative evaluations are by following the prior works [1, 2] on this research area.\n\n**References**\n\n[1] Tumanyan, Narek, et al. \"Plug-and-play diffusion features for text-driven image-to-image translation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Parmar, Gaurav, et al. \"Zero-shot image-to-image translation.\" ACM SIGGRAPH 2023 Conference Proceedings. 2023."}, {"review_id": "iGO4g76Hp3", "reviewer": "Reviewer_jCTY", "summary": "This paper proposes an energy-based model (EBM) framework addresses semantic misalignment in text-to-image diffusion models by incorporating EBMs in each cross-attention layer, minimizing a nested hierarchy of energy functions, and achieving highly effective results in diverse image generation tasks. From the shown figures, they demonstrate the effectiveness of the EBM framework.", "questions": "As I stated above, the main drawback of this paper is in the experimental part. There are lack of qualitative comparisons with correct methods. Instead, the original Stable Diffusion model is always serving as the baseline comparison. Furthermore, there are no sufficient quantitative results to show the efficiency of the proposed framework in this paper.", "limitations": "As I stated in the weakness and questions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1, The proposed framework demonstrates general applicability to diverse tasks, such as inpainting, image editing, and multi-concept generation.\n\n2, Theoretical analysis and explanations are comprehensively conducted to enhance the robustness of the experimental results.\n\n3, Notably, this paper stands out as the first diffusion model study observed to incorporate an energy-based perspective into its formulation.", "weaknesses": "1, The experimental results are only including the qualitative comparison with some other methods. In my view, some quantitative numbers should also be shown in the paper to demonstrate the performance of this method compared to others.\n\n2, For multi-concept generation, the Custom Diffusion [1] has been released during the end of last year, it should be considered as a comparison. \n\n3, The same comparison limitations also happened in inpainting (considering Blended Diffusion[2], GLIDE[3]) and image editing (Prompt2Prompt[4], pix2pix-zero[5]).\n\n4, For evaluation metrics, there are also commonly used metrics in different sub-area. For example, the CLIP-score/Lpips/Structure-Dist can be considered in image editing tasks.\n\n\n[1] Multi-Concept Customization of Text-to-Image Diffusion\n[2] Blended Diffusion for Text-driven Editing of Natural Images\n[3] GLIDE: Towards photorealistic image generation and editing with text-guided diffusion model\n[4] Prompt-to-Prompt Image Editing with Cross Attention Control\n[5] Zero-shot Image-to-Image Translation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Some quantitative numbers should also be shown in the paper**. In contrast to your comment, we would like to remind the reviewer that the quantitative comparison results have already been reported in appendix D. We will move the results to the main paper in our revised version to emphasize the effectiveness of the proposed method. Also, per requests from other reviewers, we additionally evaluate the quality of generated images using Lpips, which again reveal that the proposed method outperforms baseline methods. Please refer to the general comment 2.\n\n**W2: more baseline in multi-concept generation**. Thank you for the suggestions. However, Custom Diffusion is for the personalized generation, which requires a few images that contain the desired concepts. In contrast, the proposed method only depends on a given textual condition to generate multiple concepts simultaneously. Thus, it seems not a fair comparison. Instead, we compare the performance with Composable-diffusion [1] which is commonly compared for the multi-concept generation. Please refer figure 8 in the rebuttal pdf. We will add this result to the revised paper.\n\n**W3: more baseline in inpainting and image editing**. Per your suggestion, we conduct further comparison studies with Blended  Latent Diffusion [2]. Please refer to figure 9 in the rebuttal pdf. We will add the results in the revised paper. Also, we would like to kindly remark that the performance of recent algorithms for image editing already has been compared and reported in appendix D.\n \n**W4: evaluation metrics**. We have used CLIP-acc and Structure-Dist by following [3, 4] (see appendix D). We will move the quantitative comparison results to the main paper in our revised version to emphasize the effectiveness of the proposed method. Also, per the requests from reviewers, we further evaluated the quality of generated images from the real-image editing task via Lpips. Please refer to the general comment 2.\n\n**References**\n\n[1] Liu, Nan, et al. \"Compositional visual generation with composable diffusion models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Avrahami, Omri, Ohad Fried, and Dani Lischinski. \"Blended latent diffusion.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-11.\n\n[3] Tumanyan, Narek, et al. \"Plug-and-play diffusion features for text-driven image-to-image translation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Parmar, Gaurav, et al. \"Zero-shot image-to-image translation.\" ACM SIGGRAPH 2023 Conference Proceedings. 2023.\n"}, {"review_id": "6ysjeWfgoe", "reviewer": "Reviewer_2y1g", "summary": "This work tackles the semantic misalignment problem of stable diffusion model using the energy-based model framework.The authors first show that each cross-attention in the diffusion model can be seen as one step optimization of a pre-defined energy function. They then formulate a Bayesian update for the context vector accordingly. The authors demonstrate the effectiveness of their method by showing qualitative samples in multi-concept generation, text-guided image inpainting as well as compositional generation tasks. ", "questions": "1. The understanding of the whole unet: While we may look at each individual cross-attention layer as updating a certain energy function, I would like to hear the authors' insight on how we can understand the unet as a whole. At different cross-attention layers, the weight are different. And there are a batch of linear and non-linear transformations between each two cross-attention layers. Then can we understand the whole unet as updating a certrain energy? (The cascading update of C seems to suggest that the optimization across the whole unet has some consistency.)  And if possible, how? \n\n2. The energy distribution across different attention layers: In figure 2, the authors plot the energy of 16 attention layers across different sampling steps. It seems that at all the steps, the energy will peak at the middle layer and reach bottom at the two ends. Can the authors provide some insight on why this happens?  \n\n3. In section 3.1 algorithm, the authors provide their design of updating context vector C. While this seems to work well on the samples shown in the paper, I'm wondering what if other update algorithms are use. For example, what if we don't cascade $C_{n+1, t}$ to the next (n+1)th layer but updates $C$ for all the layers starting from scratch and update them for more than one step? \n\n4. From figure 8 in the supplementary, it seems that increasing $\\gamma_{reg}$ instead of $\\gamma_{attn}$ encourages pattern to occur.  When $\\gamma_{reg}$ is zero, no matter how to tune $\\gamma_{attn}$, the editing cannot get successful results (no teddy bear here). This can be a bit counter-intuitive. As for my understanding, the attention term should be the one to align the key and query. Are there any explanations on why these happen? \n\n", "limitations": " The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "I love the novelty and theoretical framework of this paper. While there are several other works tackle the more controlable image generation/editing problem based on the modification of the cross attention layers, this paper provides a systematical theory framework for this. This might not only help with the improvement on certain task but also facilicate people to form a deeper understanding of the model itself.", "weaknesses": "1. Lack of quantitative results:\nWhile several qualitative samples are shown in the paper, quantitative evaluation over a large number of different samples can be more convincing to judge the model's performance against the baseline. The authors may consider using pretrained model (for example like [1] does) or using human evaluation.\n\n2.Choice of hyper-parameter:\n Looking at the supplementary, it seems that a successful editing requires one to choose parameters $\\alpha$, $\\beta$, $\\gamma$ case-by-case. Furthermore, it seems that changing hyper-parameters have a great influence over the generation performance.  This might limit the proposed method to be used in real applications.\n\n[1] Training Diffusion Models with Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Lack of quantitative results**. We would like to kindly remind the reviewer that we have already measured CLIP accuracy and DINO-ViT structure distance motivated by [1,2] for the image editing task and compared it with state-of-the-art methods in the appendix. The result shows that the proposed method exhibits the best editing performance while preserving the structure of the original input image. Please refer to the appendix for details.  We will move the quantitative results to the main paper in our revised version.\n  \nFurthermore, per other reviewers' suggestions, we also conducted a human evaluation and measured LPIPS to evaluate image quality. See the general comment 2.\n \n**W2: Choice of hyper-parameter**. \n\n- $\\alpha$: We would like to assure the reviewer that we found $\\alpha$ to be the most stable and effective when set to 0, as we mentioned in lines 146-147 and the ablation study results in figure 10 in the supplementary. Therefore, we used it consistently in all experiments, eliminating the need for separate tuning. \n\n- $\\beta$: In the case of $\\beta$, it plays a role as the temperature of the softmax in the conventional attention operation. Thus, following the lead of previous attention-related studies, we highly recommend setting $\\beta$ to $\\frac{1}{\\sqrt{d}}$, where $d$ denotes the dimensionality of the embeddings. \n\n- $\\gamma$: For the hyperparameters, called $\\gamma_{attn}$ and $\\gamma_{reg}$, we would like to remark that they correspond to step size for one-step gradient descent of energy which is natural to tune to get the better performance. However, we also would like to emphasize that fixed hyperparameters are used in the real-image editing tasks, instead of case-by-case tuning. Remarkably, the proposed method outperforms the state-of-the-art works (see appendix D for the detailed result). This result indicates that the proposed method exhibits robustness within a certain range of hyperparameters. In order to facilitate the use of the proposed method, we have included a guide for selecting hyperparameters for each task in appendix C.\n\n**Q1: The understanding of the whole unet**. Thanks for the constructive question. The paper aims to interpret each individual cross-attention layer from the energy perspective, as the reviewer points out. Formulating a unified energy for the entire UNet architecture is non-trivial, mainly due to the inherent complexities arising from non-linearities. Although it would be out of the scope of this work, we would like to recommend a related work [3, 4] that tries to understand the whole Transformer from an energy perspective.\n \n**Q2: The energy distribution across different attention layers**. Thanks for carefully reading the manuscript. As the reviewer mentioned, there is a tendency for energy along cross-attention layers in Unet. It may be related to the dimensions of feature maps which have not been considered for plotting the energy dynamics (for example, the energy is highest at the bottleneck layer of Unet). However, we would like to note that the key message conveyed by Figure 2 is the relative energy gap before and after applying BCU. The figure highlights the impact of the proposed method on energy reduction. Furthermore, it is evident that the energy difference increases significantly, implying that adaptive context propagation influences the energy of subsequent CA layers, leading to cumulative energy minimization. To ensure clarity and emphasis on the relative energy gap, we will revise Figure 2 in the main paper.\n\n**Q3: Using other update algorithms**. Per the reviewer’s request, we implement a multistep context update for comparison while keeping the propagation of C to subsequent layers. For the multi-concept generation and inpainting tasks, we observe that this multiple update of the context vector actually improves the quality of generated images. This is in the same line as our perspective that one forward path of context vector is equivalent to one-step gradient descent for energy function. Please refer to Figure 3 in the rebuttal pdf for the result. That being said, we also observe that the multiple context update is relatively computationally expensive and a single-step update is usually sufficient for improved performance. Therefore, we decide to use a single-step update with context propagation to subsequent layers.\n\n**Q4: $\\gamma_{reg}$ and $\\gamma_{attn}$ in Figure 8**. The observation is consistent with Eq. (12), where the regularization term, controlled by $\\gamma_{reg}$, plays a role in preventing a single context vector from excessively dominating the forward attention path (lines 127-129). In other words, without proper regularization, other context vectors could dominate the attention path, leading to the neglect of the ‘teddy bear’ concept. We would like to remark that the Stable-Diffusion receives 77 tokens to generate the image. However, with the proper application of the regularization term (step size 0.025 for figure 8 in the supplementary), we can observe that the BCU with non-zero $\\gamma_{attn}$ can correctly generate the teddy bear. This result highlights the significance of the proposed BCU in capturing and expressing concepts given by textual conditions but also emphasizes that both the attention term and the regularization term need to be considered together.\n\n**References**\n\n[1] Tumanyan, Narek, et al. \"Plug-and-play diffusion features for text-driven image-to-image translation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Parmar, Gaurav, et al. \"Zero-shot image-to-image translation.\" ACM SIGGRAPH 2023 Conference Proceedings. 2023.\n\n[3] Yang, Yongyi, and David P. Wipf. \"Transformers from an optimization perspective.\" Advances in Neural Information Processing Systems 35 (2022): 36958-36971.\n\n[4] Hoover, Benjamin, et al. \"Energy transformer.\" arXiv preprint arXiv:2302.07253 (2023).\n"}, {"review_id": "3E2GbL0T0A", "reviewer": "Reviewer_5Nym", "summary": "The paper proposes to formulate cross-attention layers using energy-based models such that by minimizing the cross attention energy with respect to the context latent representation, the method can further alleviate semantic misalignments between generated or edited samples and the input descriptions, and allow zero-shot compositional generalization using combinations of cross attention outputs based on multiple inputs.", "questions": "1. is the method quite sensitive to the hyper-parameters that you use to optimize context vectors for each image? Based on the supplementary material, it seems to me that the method is quite sensitive and often requires a lot of tuning in terms of coefficients $\\gamma_{attn}, \\gamma_{reg}$ for updates, step size, etc.", "limitations": "The authors have included limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a detailed theoretical formulation of cross attentions using energy-based perspective.\n2. The method outperforms existing methods qualitatively on various image generation tasks (e.g., inpainting, multi-concept generation and compositional generation) without additional training.\n3. The authors have conducted relatively comprehensive evaluations on the method.", "weaknesses": "1. **Lack of limitations**. We have seen the good side of such method, which is to improve semantic alignment between generated images and text inputs. However, there are often trade-offs. For example, energy-based optimization is quite unstable, so it requires some tuning in terms of step size and the number of optimization steps. It would be good to mention the limitation of the method, for example, how much the method is slower compared to the standard denoising steps. \n2. **Lack of experiments on Image quality**. Though I think only context vectors are optimized, image generation quality will be unlikely to change much. I would still suggest evaluating generated images would provide a way to help understand whether such algorithms can change the generation performance.  In addition, although semantic alignment seems to improve in qualitative results, we also need to measure it quantitatively. And current metrics (e.g., CLIP) for measuring semantic alignment can be unreliable, thus a human evaluation seems necessary to compare different methods quantitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Lack of limitations**. Thanks for the comment. We only modified the forward path of the cross-attention layer and this is equivalent to a one-step gradient descent of defined energy. Although we did not minimize the energy multiple steps to achieve further convergence, we have shown that the one-step energy update (i.e. one-step forward step) effectively reduces the energy, as shown in figure 2, and it results in better performance on multiple tasks. In addition, for the modified forward path, we reuse the $QK^T$ term (algorithm 2 in appendix B), which significantly reduces the additional computational cost. In fact, both Stable Diffusion and the proposed method take ~7.0 seconds to generate a single image within 50 sampling steps with NVIDIA GeForce RTX 3090. However, we acknowledge the reviewer's comment that the proposed method requires step size as hyperparameters, namely $\\gamma_{attn}$ and $\\gamma_{reg}$, which the user should tune for their specific applications. We address this issue in appendix D by providing guidance on the range of appropriate values for these parameters.\n\n**W2: Lack of experiments on Image quality**. Per the reviewer’s request, we also measured LPIPS to evaluate the quality of the generated image and conducted human evaluation. Please refer to the general comment 2.\n\n**Q1: Sensitivity to the hyper-parameter**. We would like to remark that the hyperparameters, namely $\\gamma_{attn}$ and $\\gamma_{reg}$, correspond to step size for one-step gradient descent of energy which is natural to tune to get the better performance. However, we also would like to emphasize that fixed hyperparameters are used in the real-image editing task. Remarkably, the proposed method outperforms the state-of-the-art works (see appendix D for the detailed result). This result indicates that the proposed method exhibits robustness within a certain range of hyperparameters, which may vary depending on the task. In order to facilitate the use of the proposed method, we have included a guide for selecting hyperparameters for each task in appendix C. Still, there is a need for tuning $\\gamma$ for a newly suggested task so we will add it to the limitation section.\n"}, {"review_id": "RyJ3iKSOk2", "reviewer": "Reviewer_SVoL", "summary": "This paper proposes a novel **energy-based** framework that can automatically **update the context** used in cross-attention **without additional training**. They claim the proposed updating process well solves the **semantic misalignment** issue in text-to-image diffusion models.   ", "questions": "1. The definition of the energy function does not appear reasonable.\n2. Lack of quantitative experimental results.\n3. Lack of explanations of why energy minimization help slove semantic misalignment.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper exhibits a clear, engaging, and concise writing style.\n2. The paper introduces an interesting energy definition and proves the subtle connections with the attention mechanism used in transformers. \n3. The experimental results look promising.", "weaknesses": "1. The definition of the energy function does not appear reasonable. For instance, consider the definition of E(K) in equation 8. E(K) increases as the L2-Norm of k_i, where i = 1, 2, ..., N, increases. The minimum value of E(K) is achieved when all k_i converge to zero. This suggests that the author assumes k_i = 0 has the highest probability (since energy is minimum), which contradicts reality. Additionally, the definition of E(Q;K) in equation 7 also seems unconventional. It resembles E(K;Q) rather than E(Q;K). I have significant doubts regarding the effectiveness of the proposed energy definition. I kindly request the authors to provide a more explicit explanation for why they have defined the energy function in such a manner and clarify the actual significance of these proposed energy functions.\n\n2. There is a lack of quantitative experimental results. While the authors present some promising outcomes, there is a scarcity of extensive quantitative experiments to demonstrate that the proposed framework statistically outperforms the naive cross-attention mechanism. It is imperative to include detailed information about the quantitative experiments conducted to validate the efficacy of the proposed framework.\n\n3. There is no explicit explanation of why energy minimization can help solve semantic misalignment. I can not draw direct connections between them in the paper.\n\nI tend to reject this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In contrast to your concerns, it appears that there are several misunderstandings by the reviewers. To clarify the misunderstanding, we would like to give detailed point-by-point answers below.\n\n**W1 and Q1: The definition of the energy function does not appear reasonable**. Thanks for the important comment. We would like to gently note that the design of proposed posterior energy $E_{posterior}(K; Q) = E_{likelihood}(Q;K) + E_{prior}(K)$ is based on the theoretical foundations of [1] and empirical analyses in figure 9, 10 (appendix).\n \n**Hopfield energy**. Ramsauer et al. [1] propose a modern Hopfield energy $E_{hopfield}(Q; K) = \\frac{1}{2} \\text{diag}(Q^TQ) - \\sum \\text{logsumexp}(Kq_i^T)$ of which query update rule is equivalent to the attention mechanism. It includes two different terms: (a) exponential interaction term $\\text{logsumexp}$, and (b) the regularization term $\\frac{1}{2} \\text{diag}(Q^TQ)$ which ensures that the norm of the queries remains finite and the energy is appropriately bounded.\n \n**Design of E(Q; K) and E(K)**. While $E_{hopfield}(Q; K)$ is proposed for a query update, for a context update, we inherit the principles of [1] and define our likelihood energy as $E_{likelihood}(Q; K) = \\frac{\\alpha}{2} \\text{diag}(K^TK) - \\sum \\text{logsumexp}(Qk_i^T)$, which is symmetric to the $E_{hopfield}(Q; K)$.\n \nThat being said, as Reviewer suggested, employing $E_{likelihood}(Q; K)$ in eq(7) as $E_{posterior}(K; Q)$ appears to be a viable option. However, we have observed inferior performance, and we suspected that the problem stems from the design of regularization term.\n \nSpecifically, $\\frac{\\alpha}{2} \\text{diag}(K^TK)$ uniformly regularizes the context vector which often over-penalizes contexts and causes the context to vanish. To resolve this issue, instead of uniformly regularizing every contexts, we set $\\alpha=0$ and propose $E_{prior}(K) = \\text{logsumexp}(\\frac{1}{2} diag(KK^T))$ to regularize the smooth maximum of context vectors. By summing $E_{likelihood}(Q; K)$ and $E_{prior}(K)$, we finally get the posterior energy $E(K; Q)$ to be minimized by Bayesian Context Update (BCU), Eq. (12).\n \nAblation studies in Fig 9 and 10 support our intuitions. Specifically, $\\gamma_{reg}=0$ in Fig 9 suggests that the norm regularization is necessary for the quality and semantic alignment of generated images. In other words, regularizer plays a role in preventing a single context vector from excessively dominating the forward attention path. Moreover, Fig 10 suggests that increasing $\\alpha$ may over-penalize the contexts which prohibit using $E_{likelihood}(Q; K)$ in eq(7) as $E_{posterior}(K; Q)$.\n\n**W2 and Q2: Lack of quantitative experimental results**. In contrast to your comment, we would like to note that the quantitative comparison results have already been reported in appendix D. We will move the results to the main paper in our revised version to emphasize the effectiveness of the proposed method. Also, per requests from other reviewers, we additionally evaluate the quality of generated images using Lpips, which again reveal that the proposed method outperforms baseline methods. Please refer to the general comment 2.\n\n**W3 and Q3: Lack of explanations of why energy minimization help slove semantic misalignment**. Please refer to the general comment 1. We would like to clarify that the goal of the proposed method is to achieve the *adaptive* context propagation through UNet, which could mitigate the semantic misalignment caused by the use of fixed context embeddings. Here, we would like to remark that minimizing the energy function is equivalent to maximizing the log-likelihood of the probability density function. In this context, minimizing the energy function $E(K|Q)$ corresponds to maximizing the log-likelihood of $p(K|Q)$, which indicates finding the most likely K given Q and finally achieves the *adaptive* context for a better semantic alignment.\n\n**References**\n\n[1] Ramsauer, Hubert, et al. \"Hopfield networks is all you need.\" arXiv preprint arXiv:2008.02217 (2020)."}, {"review_id": "xbrjWtxXDA", "reviewer": "Reviewer_tGPJ", "summary": "This paper proposed a training-free algorithm to modife cross-attention during inference time that could implicitly minimize the energy in the latent space. This paper formulate this idea from a energy perspective. The authors conduct three experiments - multi-concept generation, image inpainting, compositional editing to demonstrate the effectivenns of the new idea.", "questions": "1. You metions \"nested hierarchy of energy functions\" many times? But what is the definition of it and there's any reference papers?\n2. In Eq. (6), what is the definition of softmax_1 and softmax_2? And how to obtain Eq. (6) from Q=softmax_2(\\betaQK^T)K? I am a little confused.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea from energy perspective is innovative and the derivation of the equations are solid.\n2. Figure 2 an the associate caption is straightforward that shows the relation between energy value and multi-concept generation quality.", "weaknesses": "1. In experiment section, the authors provide only some generation cases for subjective-quality analysis without any quantitative results. The shown cases after cherry-pick maybe not adequate to demonstrate the effectiveness of the method proposed in this paper.\n2. A minor typo in Line 83: \"update rule for a state pattern \\epsilon\" -> \"update rule for a state pattern \\zeta\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Lack of quantitative results**. We would like to kindly remind the reviewer that the quantitative comparison results have been reported in appendix D. We will move the results to the main paper in our revised version to emphasize the effectiveness of the proposed method. Also, per requests from other reviewers, we additionally evaluate the quality of generated images using Lpips, which again reveal that the proposed method outperforms baseline methods. Please refer to the general comment 2.\n\n**W2: typo**. We will fix it in the revised paper.\n\n**Q1: nested hierarchy of energy functions**. We have used this term to emphasize that there are many energy functions for each layer, time-step, etc. We do not optimize just a single energy but optimize a nested hierarchy of energy functions during the forward pass of updated context vectors. Specifically, we mentioned “nested hierarchy of energy functions” in order to indicate a unified model-specific energy for the entire UNet. We gently note that it is non-trivial to derive an analytical form of the unified energy for a whole U-Net network, mainly due to the inherent complexities arising from non-linearities.\n\n**Q2: definition of softmax_1 and softmax_2, derivation of Eq. (6)**. The definition for subscripts below the softmax is described in lines 64-68, section 2.2.\neq. (6) could be derived from eq. (4). Eq. (6) is opted for drawing a connection between transformer attention and hopfield energy minimization. For a such connection, the value matrix $V$ is arbitrarily introduced with a mapping $W_V$. For more details, please refer to eq.(10) in [1].\n\n**References**\n \n[1] Ramsauer, Hubert, et al. \"Hopfield networks is all you need.\" arXiv preprint arXiv:2008.02217 (2020).\n"}]

Author rebuttal:
We sincerely thank all the Reviewers for their valuable comments. We are encouraged that the reviewers say that “the motivation of the paper is clear” (nFGp), “love the novelty and theoretical framework” (2y1g), “the idea from energy perspective is innovative” (tGPJ) and “Theoretical analysis and explanations are comprehensively conducted” (jCTY), etc. 
 
We have summarized some of the major concerns that were raised from the reviewers below. Point-to-point responses were also included as a reply to each reviewer.

--- 

**Comment 1**. We further clarify the motivation and contributions of the proposed work in a more concise way. Please refer to the conceptual model figure 1 in the uploaded rebuttal pdf.

**Main motivation**: Our main point is that the conventional use of fixed context embedding is sub-optimal, as illustrated in Figure 1 of the main paper (Stable Diffusion). Such misalignment may be attributed to various causes: error-prone human-designed prompts, limited capacity of pre-trained textual encoders, fundamental gap between CLIP-space and U-Net latent space, etc.  In order to mitigate the misalignment, instead of leveraging fixed contexts, we aim to establish an adaptive context by modeling p(context|representations). Note that this is a significant departure from the previous methods which only model p(representations|context) with frozen context vectors.

**Idea**: To model p(context|representations), we propose an energy-based modeling. Specifically, the energy function in Eq. (3) is closely related to the attention mechanism as shown in [1]. Inspired from these results, we define p(K|Q) in the cross-attention space with the posterior energy $E_{posterior}(K; Q) = E_{likelihood}(Q;K) + E_{prior}(K)$.  Then, one of the primary contributions of the proposed method is to create a correspondence  between the context and respresentation through the cross-attention layers of a diffusion model by minimizing the parameterized energy functions. In contrast to a static and fixed context embeddings, the proposed method enables the adaptive context propagation through an energy minimization such that both Q and C are updated and propagated to subsequent layers.  This test time optimization for context and respresenation alignment through cross attention have never been tried before, which may lay down foundation for further research.

---

**Comment 2**. Reviewers are kindly reminded that, due to the limited space, we reported multiple results in the appendix of the original submission, which includes ***quantitative comparison* (section D.5)**, ablation study on BCU, CACAO (Fig 9), analysis on prior energy $E(K)$ (Fig 10), more visualizations (Fig 11-14), etc.  
In the attached rebuttal PDF file, we now add quantitative comparisons on compositional editing tasks with the CelebA-HQ dataset. Specifically, we focus on three image-to-image translation tasks: woman $\rightarrow$ man, woman $\rightarrow$ woman w/ glasses, and woman $\rightarrow$ man w/ glasses. For this, we select source images of women without wearing glasses from CelebA-HQ. To ensure fair comparisons, all methods leverage the same version of Stable Diffusion, sampler, sampling steps, etc. 

Resp_Table 1 shows that the proposed energy-based framework achieves a high CLIP-Acc while maintaining low Structure Dist and LPIPS. Note that the proposed method works consistently well on the compositional editing, i.e., woman $\rightarrow$ man w/ glasses. Moreover, we conduct a user study to evaluate the perceptual quality of generated samples. Following the protocol in [2], we ask 21 people to rate the scores on a scale from 1(poor) to 5(excellent) on the following questions:  Q1) Does the output align with the intended semantics of the target text? (Text-match), Q2) Do the generated images appear realistic? (Realism), Q3) Do the outputs preserve the content information from source images? (Content). Resp_Table 2 shows that the proposed method consistently scored high across all questions. We would like to remark that, for the cat-to-dog case, while the DDIM inversion produces outputs in alignment with the given text, as evidenced by the highest score for Q1, it is notably limited by a significant loss of original content, reflected by the lowest score on Q3. This observation aligns with our findings reported in section D.5 and figure 7 in appendix. We will include all the quantitative results in the revised version of our main paper.

---
**[Resp_Table1] CelebA-HQ results**

|CelebA-HQ | Woman→Man | | | Woman→Woman w/ glasses| |  | Woman→Man w/ glasses| | | 
|-|-|-|-|-|-|-|-|-|-|
|**Method**| **CLIP Acc (↑)** | **Dist (↓)** | **LPIPS (↓)** | **l CLIP Acc (↑)** | **Dist (↓)** | **LPIPS (↓)** | **l CLIP Acc (↑)** | **Dist (↓)** | **LPIPS (↓)** |
|CycleDiffusion|57.0%|0.156|0.433| l 74.0%|**0.018**|**0.117**| l 63.5%|0.237|0.443|
|Pix2Pix-zero|93.0%|0.043|0.391| l 66.0%|0.024|0.266| l 93.9%|0.052|0.444|
|Ours|**94.0%**|**0.035**|**0.324**| l **99.0%**|0.029|0.292| l **96.0%**|**0.037**|**0.341**|



**[Resp_Table2] User Study**
| |Cat→Dog | | | l Horse→Zebra| | | l Cat→Cat w/ glasses| | |
|-|-|-|-|-|-|-|-|-|-|
|**Method** | **Q1** | **Q2** | **Q3** | **l Q1** | **Q2** | **Q3** | **l Q1** | **Q2** | **Q3**|
|SDEdit|4.15|4.00|4.10 | l 4.00|4.05|4.55| l 4.40|4.40|4.30|
|DDIM-inv|**4.30**|3.85 |2.35| l 4.40|3.50|2.40| l 3.60|3.10|2.10|
|Pix2Pix-zero|3.35|2.80 |2.80| l 3.70|3.05|3.25| l 3.35|3.15|2.80|
|Ours|3.95|**4.00**|**4.40** | l **4.55**|**4.30**|**4.65**| l **4.70**|**4.50**|**4.70**|

---

**References**

[1]: Ramsauer, Hubert, et al. "Hopfield networks is all you need." arXiv preprint arXiv:2008.02217 (2020).

[2]: Kwon, Gihyun, and Jong Chul Ye. "Diffusion-based image translation using disentangled style and content representation." International Conference on Learning Representations. 2023.

Meta-review: This paper proposes a new energy-based framework for text-to-image generation. Extensive experiments are conducted to show that the proposed framework is effective in handling various image generation tasks. In response to the concerns initially raised by all the reviewers in their initial assessments, the rebuttal systematically addressed these issues, including adding additional comparison results, clarification of the motivation and contribution of the paper, and a discussion of computational cost. Following extensive discussions, a consensus was achieved among all seven reviewers, culminating in the decision to accept the paper. This agreement is attributed to the paper's innovation, theoretical justification, and sufficient empirical validation. The AC agrees with all the reviewers and recommends accepting the paper. To further improve the paper quality, the AC urges the authors to revise their paper by taking into account all the suggestions provided by the reviewers, including integrating extra experimental results conducted during the rebuttal phase.
Decision: Accept (poster)