Title: Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion model

Energy-based model

Text-to-image generation
Abstract: Despite the remarkable performance of text-to-image diffusion models in image generation tasks, recent studies have raised the issue that generated images sometimes cannot capture the intended semantic contents of the text prompts, which phenomenon is often called semantic misalignment. To address this, here we present a novel energy-based model (EBM) framework for adaptive context control by modeling the posterior of context vectors. Specifically, we first formulate EBMs of latent image representations and text embeddings in each cross-attention layer of the denoising autoencoder. Then, we obtain the gradient of the log posterior of context vectors, which can be updated and transferred to the subsequent cross-attention layer, thereby implicitly minimizing a nested hierarchy of energy functions. 
Our latent EBMs further allow zero-shot compositional generation as a linear combination of cross-attention outputs from different contexts. 
Using extensive experiments, we demonstrate that the proposed method is highly effective in handling various image generation tasks, including multi-concept generation, text-guided image inpainting, and real and synthetic image editing. Code: https://github.com/EnergyAttention/Energy-Based-CrossAttention.

Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models

Abstract

Despite the remarkable performance of text-to-image diffusion models in image generation tasks, recent studies have raised the issue that generated images sometimes cannot capture the intended semantic contents of the text prompts, which phenomenon is often called semantic misalignment. To address this, here we present a novel energy-based model (EBM) framework for adaptive context control by modeling the posterior of context vectors. Specifically, we first formulate EBMs of latent image representations and text embeddings in each cross-attention layer of the denoising autoencoder. Then, we obtain the gradient of the log posterior of context vectors, which can be updated and transferred to the subsequent cross-attention layer, thereby implicitly minimizing a nested hierarchy of energy functions. Our latent EBMs further allow zero-shot compositional generation as a linear combination of cross-attention outputs from different contexts. Using extensive experiments, we demonstrate that the proposed method is highly effective in handling various image generation tasks, including multi-concept generation, text-guided image inpainting, and real and synthetic image editing. Code: https: //github.com/EnergyAttention/Energy-Based-CrossAttention .

1 Introduction

Diffusion models (DMs) have made significant advances in controllable multi-modal generation tasks [37], including text-to-image synthesis. The recent success of text-to-image diffusion models, e.g., Stable Diffusion [31], Imagen [33], etc., is mainly attributed to the combination of high-fidelity DMs with high-performance large language models (LMs).

Although text-to-image DMs have shown revolutionary progress, recent studies have shown that the current state-of-the-art models often suffer from semantic misalignment problems, where the generated images do not accurately represent the intended semantic contents of the text prompts. For example, [4] discovered the catastrophic neglect problem, where one or more of the concepts of the prompt are neglected in generated images. Moreover, for a multi-modal inpainting task with text and mask guidance, [41] found that the text-to-image DMs may often fail to fill in the masked region precisely following the text prompt.

Therefore, this work focuses on obtaining a harmonized pair of latent image representations and text embeddings, i.e., context vectors, to generate semantically aligned images. In order to mitigate the misalignment, instead of leveraging fixed context vectors, we aim to establish an adaptive context by modeling the posterior of the context, i.e. p ( context | representations ) . Note that this is a significant departure from the previous methods which only model p ( representations | context ) with frozen context vectors encoded by the pretrained textual encoder.

Stable-Diffusion

Ours

(C) Synthetic Image Editing

(B) Inpainting

(A) Multi-concept

'A monkey wearing a kimono'

Stable-Diffusion

Ours

'A horse and a dog'

'A teddy bear'

'A castle next to a river'

'A corgi wearing a bow tie and a birthday hat'

'A photo of an woman'

(+) smile (-) curls

(D) Real Image Editing

Original Image

'colorful cat'

'pink cat'

'dog'

'fox'

'wearing sunglasses'

Specifically, we introduce a novel energy-based Bayesian framework, namely energy-based crossattention (EBCA), which approximates maximum a posteriori probability (MAP) estimates of context vectors given observed latent representations. Specifically, to model p ( context | representations ) , we first consider analogous p ( K l | Q t,l ) in the latent space of the intermediate cross-attention layer of a denoising auto-encoder, i.e., cross-attention space. Here, K l and Q t,l correspond to the key and query in l -th layer at time t that encode the context from the text and the image representation, respectively. Inspired by the energy-based perspective of attention mechanism [30], we then formulate energy functions E ( K l ; Q t,l ) in each cross-attention space to model the posterior. Finally, we create a correspondence between the context and representation by minimizing these parameterized energy functions. More specifically, by obtaining the gradient of the log posterior of context vectors, a nested hierarchy of energy functions can be implicitly minimized by cascading the updated contexts to the subsequent cross-attention layer (Figure 2) .

Moreover, our energy-based perspective of cross-attention also allows zero-shot compositional generation due to the inherent compositionality of Energy-Based Models (EBMs). This involves the convenient integration of multiple distributions, each defined by the energy function of a specific text embedding. In practical terms, this amalgamation can be executed as a straightforward linear combination of cross-attention outputs that correspond to all selected editing prompts.

We demonstrate the effectiveness of the proposed EBM framework in various text-to-image generative scenarios, including multi-concept generation, text-guided inpainting, and compositional generation. The proposed method is training-free, easy to implement, and can potentially be integrated into most of the existing text-to-image DMs.

2 Preliminaries

2.1 Diffusion Model

Diffusion models [36, 13, 37, 18, 16] aims to generate samples from the Gaussian noise by iterative denoising processes. Given clean data x 0 ∼ p data ( x 0 ) , diffusion models define the forward sampling from p ( x t | x 0 ) as x t = √ ¯ α t x 0 + √ 1 -¯ α t z t , where z t ∼ N (0 , I ) , t ∈ [0 , 1] . Here, for the Denoising Diffusion Probabilistic Model (DDPM) [13], the noise schedule β t is an increasing sequence of t , with ¯ α t := ∏ t i =1 α t , α t := 1 -β t . The goal of diffusion model training is to obtain a neural network ϵ θ ∗ that satisfies

ZT

Z-2

(B)

(A)

Query

Self-

Attn

Cross-

Attn

Query

Self-

Attn

EBCQ

EBCU

Context

Context

so that the reverse sampling from q ( x t -1 | x t , x θ ∗ ( x t , t )) is achieved by

where z t ∼ N (0 , I ) and σ 2 is variance which is set as β in DDPM. With iterative process, we can sample the image x 0 from initial sample x T ∼ N (0 , I ) .

Since diffusion models require the iterative sampling on high dimensional space, they are computationally expansive and time consuming. To mitigate this limitation, Latent Diffusion Model (LDM) [31] has proposed diffusion processes on the compressed latent space using pre-trained auto-encoder. Furthermore, by introducing language model-based cross-attention to diffusion model's U-Net neural backbone [32], LDM enables token-based conditioning method such as text-to-image.

2.2 Energy-based Perspective of Attention Mechanism

Definition. Given an N -dimensional vector v , its logsumexp and softmax are defined as logsumexp( v , β ) := β -1 log( ∑ N i =1 exp( v i )) , softmax( v ) := exp( v -logsumexp( v , 1)) .

where v i denotes the i -th element of v . For a given a matrix A , its logsumexp i ( A ) and softmax i ( A ) are understood as taking the corresponding operation along the i -th dimension of A . So, for instance, i = 1 , softmax i ( A ) consist of column vectors that sum to 1 .

Energy model. Following the success of the attention mechanism, several studies have focused on establishing its theoretical foundation. One promising avenue is interpreting the attention mechanism using the Energy-Based Model (EBM) framework. This line of research begins with recent research on modern Hopfield networks [30, 14], which gradually builds up to the self-attention mechanism of the Transformer model.

The Hopfield network is a dense associative memory model that aims to associate an input with its most similar pattern. Specifically, it constructs an energy function to model an energy landscape that contains basins of attraction around desired patterns. Recently, modern Hopfield networks [30, 6, 19] has introduced a new family of energy functions, which aim to improve its pattern storage capacity or make it compatible with continuous embeddings. To this end, [30] proposed the following energy function of a state pattern (query) ζ ∈ R d parameterized by N -stored (key) patterns X = [ x 1 . . . , x N ] ∈ R d × N and β > 0 :

Intuitively, the first term ensures the query remains finite, while the second term measures the individual alignment of the query with every stored pattern. Based on the Concave-Convex-Procedure (CCCP) [43], [30] derives the update rule for a state pattern E and time t as follows:

Proposition 1 ([30]) . Define the update rule f : R d → R d as follows:

Then, the update rule decreases the loss (3) and converges globally. In another word, for ζ ( t +1) = f ( ζ ( t ) ) , the energy E ( ζ ( t ) ) → E ( ζ ∗ ) for t →∞ and a fixed point ζ ∗ .

Note that (4) is equivalent to a gradient descent update to minimize (3) with a step size η = 1 :

Connection to the attention of the transformer. Remarkably, this implicit energy minimization is closely related to the attention mechanism as shown in [30]. To see this, suppose that y i , r i ∈ R d is given as stored (key) and state (query) patterns, respectively. Let W K , W Q ∈ R d × d H represent linear maps for y i and r i , respectively. Then, we introduce x i = W T K y i ∈ R d H and ζ i = W T Q r i ∈ R d H . Let Y = ( y 1 , . . . , y N ) T ∈ R N × d , and R = ( r 1 , . . . , r S ) T ∈ R S × d . We define X T = K = Y W K ∈ R N × d H , and Q = RW Q ∈ R S × d H . By plugging X T = K and ζ = q i into (4) for all i , we obtain that: Q T = K T softmax 1 ( β KQ T ) ∈ R d H × S . By taking the transpose, we obtain Q = softmax 2 ( β QK T ) K . Let V ∈ R N × d V denote the value matrix as V = Y W K W V = KW V , which will replace the K outside of softmax 2 . Then, we finally obtain

which is exactly the transformer attention with β = 1 / √ d H . This connection affords us the insightful theoretical ground of the attention mechanism: the update step of the attention mechanism in a Transformer layer acts as an inner-loop optimization step, minimizing an implicit energy function that is determined by queries, keys, and values.

3 Energy-based Cross-Attention

Recall that our objective is to generate semantically correct images by harmonizing latent (U-Net) representations and context vectors within the denoising autoencoder. To achieve this, we propose a simple but effective Bayesian framework, namely energy-based cross-attention (EBCA). Specifically, we perform test-time optimization of context vectors within the cross-attention spaces of a denoising autoencoder to implicitly minimize a specially designed energy function. Note that this is a significant departure from the conventional text-guided diffusion models which have leveraged fixed context vectors embedded by pre-trained CLIP [29] to all cross-attention layers.

3.1 Energy-based Bayesian Context Update

Energy function. Our focus is on the cross-attention space of a time-dependent denoising autoencoder, utilizing the conventional U-Net neural architecture. Here, we refer to latent representations as the representations of intermediate layers in U-Net unless otherwise specified. Let L be the number of cross-attention layers. For each l -th layer at time t , we define the queries matrix Q t,l ∈ R P 2 l × d l , and the keys and values matrices K l ∈ R N × d l and V l ∈ R N × d l , respectively. Here, P l represents the spatial dimension of latent representations in the l -th layer. Given context vectors C ∈ R N × d c , we map K l and V l with W K,l , W V,l ∈ R d c × d l , such that K l = CW K,l and V l = CW V,l . In the following, time t and layer index l are omitted in notations Q t,l and K l for simplicity.

The main goal is to obtain a maximum a posteriori probability (MAP) estimate of context vectors C given a set of observed latent representations of queries Q t,l . First, based on the energy functions (8) and (9), the posterior distribution of K can be defined by using Bayes' rule: p ( K | Q ) = p ( Q | K ) p ( K ) /p ( Q ) , where (8) and (9) are leveraged to model the distribution p ( Q | K ) and p ( Q ) , respectively. Then, in order to obtain a MAP estimation of C , we approximate the posterior inference using the gradient of the log posterior. The gradient can be estimated as follows:

Motivated by the energy function in (3), we first introduce a new conditional energy function w.r.t K as follows:

where k i denotes the i -th row vector of K , α ≥ 0 , and diag( A ) = ∑ N i =1 A i,i for a given A ∈ R N × N . Intuitively, logsumexp( Qk T i , β ) term takes a smooth maximum alignment between latent

representations q j , j = 1 , · · · , P 2 and a given text embedding k i . Let j ∗ = arg max j q j k T i . Then, the implicit minimization of -logsumexp term encourages each k i to be semantically well-aligned with a corresponding spatial token representation q j ∗ . In turn, this facilitates the retrieval and incorporation of semantic information encoded by k i into the spatial region of q j ∗ .

The diag( KK T ) term in (8) serves as a regularizer that constrains the energy of each context vector k i , preventing it from exploding during the maximization of logsumexp( Qk T i , β ) , thereby ensuring that no single context vector excessively dominates the forward attention path. In this regard, we also propose a new Q -independent prior energy function E( K ) given by:

Instead of penalizing each norm uniformly, it primarily regularizes the smooth maximum of ∥ k i ∥ which improves the stability in implicit energy minimization. We empirically observed that the proposed prior energy E( K ) serves as a better regularizer compared to the original diag( KK T ) term (related analysis in (12) and appendix E).

Although our energy function is built based on the energy function, in contrast to (3), the proposed one is explicitly formulated for implicit energy minimization w.r.t keys K and the associated context vectors C , which is different from the theoretical settings in Section 2.2 w.r.t Q [30]. It is worth noting that the two energy functions are designed to serve different purposes and are orthogonal in their application.

MAP estimation. Based on the proposed energy functions, we derive the update rule of key K and context C following (7):

Theorem 1. For the energy functions (8) and (9), the gradient of the log posterior is given by:

where D ( · ) is a vector-to-diagonal-matrix operator.

Then, by using the chain rule the update rule of context vectors C is derived as follows:

where γ > 0 is a step size.

In practice, we empirically observed that the nonzero α in (8) often leads to an over-penalization of contexts, which can ultimately cause some contexts to vanish. To address this, we set α = 0 . Moreover, we found it beneficial to assign distinct step sizes γ attn and γ reg as follows:

where the first and second terms are named as attention and regularization term, respectively.

Our theoretical observations offer valuable insights into implicit energy minimization by modifying the forward path of cross-attention. Inspired by these observations, we have transplanted the energybased cross-attention layers to pre-trained text-to-image diffusion models. We first illustrate the challenges associated with adopting EBCA in a deep denoising auto-encoder and subsequently demonstrate how to overcome them in practice.

If a single recurrent Transformer block were available, a single energy function would be minimized for a given cross-attention layer by recurrently passing the updated context C n +1 . However, in practical applications, there are multiple layer- and time-dependent energy functions in conventional deep denoising auto-encoder, which makes it infeasible to minimize each energy function individually. To address this challenge, we implicitly minimize a nested hierarchy of energy functions in a single forward pass based on our theoretical observations. More details are followed.

Algorithms. At a given time step t , we initialize the context vectors C 1 ,t with text embeddings C CLIP obtained from a pre-trained CLIP. Then, within the n -th cross-attention layer with C n,t ( 1 ≤ n ≤ L ), we compute the updated context vectors C n +1 ,t using the gradients in Theorem 1 and (12). We then cascade C n +1 ,t to the next ( n +1) -th cross-attention layer. We do not forward the final context C L +1 ,t at time t to time t +1 , as the distributions of Q t and Q t +1 can be significantly different due to the reverse step of the diffusion model. Instead, we reinitialize C 1 ,t +1 with C CLIP . The pseudo-code for the proposed framework is provided in appendix B.

The sequence of energy-based cross-attention layers is designed to minimize nested energy functions. While a single-layer update in a single step may not lead to optimal convergence, the proposed layercascade context optimization can synergistically improve the quality of context vectors. Notably, our proposed method does not incur additional computational costs in practice since the gradient term in Theorem 1 relies on the existing keys, queries, and attention maps in the cross-attention layer, which can be obtained for free in the forward path.

3.2 Energy-based Composition of Queries

In addition to the above key advantages, the cross-attention space EBMs shed new light on the zero-shot compositional generation. Recent studies [8, 10] have demonstrated that the underlying distributions of EBMs can be combined to represent different compositions of concepts. For example, given a data point x and independent concept vectors c 1 , . . . , c n , the posterior likelihood of x given a conjunction of specific concepts is equivalent to the product of the likelihood of each individual concept as follows:

where each E( x | c i ) represent independently trained EBM with concept code c i . While it is an appealing solution to the controllable generation, it is notoriously difficult to train EBMs [9, 27] in a way to make themselves scalable to high-resolution image generation. Instead of directly training EBMs in pixel space, we leverage the cross-attention space EBMs and the generative power of state-of-the-art pre-trained DMs to achieve high-fidelity compositional generations.

More specifically, assume that we have main context vectors C 1 embedded from a main prompt, e.g. " a castle next to a river ", and a set of independent editing context vectors, C = { C 2 , . . . , C M } , each embedded from different editorial prompt, e.g. " monet style ", " boat on a river ", etc. Then, we define the keys K l,s for context s within a cross-attention layer of index l as K l,s = C s W K,l , ∀ s ∈ { 1 , 2 , . . . , M } . The index l will be omitted for notational simplicity. Then, for a given key K s , we consider the energy function in cross-attention space w.r.t queries Q :

which is directly motivated by (3). We then introduce the compositional energy function ˆ E , for the concept conjunction of C as in (13) and the updated rule as done in (6):

where Q TF , K TF,s and V TF directly follows the definition in (6) and the degree and direction of s -th composition can be controlled for each concept individually by setting the scalar α s , with α s < 0 for concept negation [8]. Note that this is exactly a linear combination of transformer cross-attention outputs from different contexts with β = 1 / √ d H . We refer to this process as the Energy-based Composition of Queries (EBCQ).

Cumulative sumof theposteriorenergydifference

Layer-wiseenergydynamics

"A monkey and a hat"

Stable-Diffusion

Stable-Diffusion+Ours

3e5

100

0

t=0

t=15

t=30

t=45

Ecum

2e5

200

-300

1e5

E(Q;K)

400

5e4

500

D

20

Samplingstep 30

40

50

600

700

008-

CAtayers

CAtayers

CAtayers

CAtayers

#1 →#16

#1 →#16

#1 →#16

#1 → #16

This update rule implies that the compositional energy can be minimized implicitly through modifications to the forward path of EBCA, thereby guiding the integration of semantic information conveyed by editorial prompts. Notably, this is a significant departure from the existing energy-based compositional methods [8, 26, 9]. Specifically, no training or fine-tuning is required. Instead, cross-attention outputs of the main and editorial prompts are simply obtained in parallel, averaged, and propagated to the next layer. Moreover, the introduction of EBMs in cross-attention space is orthogonal to [22], which conducts compositional generation by treating a pre-trained diffusion model ϵ θ ∗ itself as an implicitly parameterized EBM.

4 Experimental Results

To verify our claim of energy minimization via modified cross-attention, we conduct various experiments in text-guided image generation to verify semantic alignment, namely (1) multi-concept generation [22, 11], (2) text-guided image inpainting [23, 41], and (3) compositional generation [12, 1, 28] which includes real and synthetic image editing. In this work, while the first and third applications address similar challenges, we categorize them separately based on the implementation derived from the energy-based interpretation. Specifically, the Energy-based Bayesian Context Update (EBCU) is applied to every task, and the Energy-based Composition of Queries (EBCQ) is additionally leveraged in the third task. Note that all applications have been done without additional training.

Setup. The proposed framework can be widely mounted into many state-of-the-art text-to-image DMs due to its unique functionality of context updates and cross-attention map compositions. Here, we verify the effectiveness of energy-based cross-attention with Stable Diffusion (SD) [31]. The SD is an LDM that is pre-trained on a subset of the large-scale image-language pair dataset, LAION-5B [34] followed by the subsequent fine-tuning on the LAION-aesthetic dataset. For the text embedding, we use a pre-trained CLIP [29] model following the Imagen [33]. The pre-trained SD is under the creativeML OpenRAIL license. Detailed experimental settings are provided in the appendix.

4.1 Analysis on Energy during the sampling

We perform a comprehensive analysis on (8) and (9) during the forward path through the modified cross-attention, which offers insights into the energy behavior for real applications. Specifically, we examine the energy dynamics involved in the multi-concept image generation that is straightforward and could be readily applied to other applications with minimal effort. We record computed energy values along layers and sampling steps 30 times with a fixed text prompt and then display them in Figure 3 with the generated samples, where red and green lines denote the energy involved with the Stable-Diffusion (SD) and the proposed method, respectively. For each sampling step block that is alternately shaded, the energy values are plotted through 16 cross-attention layers.

Across all layers and sampling steps, the energy associated with the proposed method remains consistently lower than that of the SD. This is in line with the semantic alignment between intermediate denoised estimates and the given text prompt. In both cases of the SD and the proposed method, E ( Q ; K ) decreases over sampling steps. This implies that the iterative refinement of the updated query carried over to subsequent sampling steps, resulting in a better match to the given context. Note that the proposed method even achieves lower energy with the EBCU. More analyses are provided in the appendix E including the ablation study.

'A cat wearing a shirt'

'A lion and a crown'

Stable-Diff

Structured-Diff

Ours

Stable-Repaint

Stable-Inpaint

Ours

Masked Input

Stable-Repaint

Stable-Inpaint

Ours

Masked Input

Teddy bear

Table lamp

Bow tie,

birthday hat

lit candle

4.2 Multi-Concept Generation

We empirically demonstrate that the proposed framework alleviates the catastrophic neglect and attribute binding problems defined in the existing literature [4]. As shown in Figures 3 and 4, the EBCU effectively mitigates these problems by promoting attention to all relevant concepts in the prompt. Specifically, the regularization term introduced by the prior energy prevents a single context from dominating the attention, while the attention term updates the context vectors in the direction of input queries, improving semantic alignment and facilitating the retrieval of related information.

4.3 Text-guided Image Inpainting

In addition, we have evaluated the efficacy of the proposed energy-based EBCU on a text-guided inpainting task. Although existing state-of-the-art DMs such as DALLE and SD can be employed for inpainting by exploiting the ground-truth unmasked area [23], they usually require computationally expensive fine-tuning and tailored data augmentation strategies [23, 41]. In contrast, as shown in Figure 5, our proposed energy-based framework significantly enhances the quality of inpainting without any fine-tuning. Specifically, we incorporate the energy-based cross-attention into the StableRepaint (SR) and Stable-Inpaint (SI) models, which can struggle to inpaint multiple objects (e.g., birthday hat and bow tie ) or unlikely combinations of foreground and background objects (e.g., Teddy bear on the Starry Night painting). In contrast, the proposed approach accurately fills in semantically relevant objects within the target mask region.

4.4 Compositional Generation

We demonstrate that the proposed framework improves the controllability and compositionality of DMs, which is a significant challenge for generative models. To assess this, we split the task into synthetic and real image editing.

Ours

Composable-Diff

A picture of

a car

(+) mountain road

slow

(-) blur

A photo of

the face of

a random

person

(+) female

smile

(-) male

A photo of

the face of

an woman

(+) smile

(-) curls

A castle

next to a river

boat

Synthetic Image

DDIM-inversion

+ text swap

Plug-and-Play

Real Image

Ours

Tennis → Golf

Baby fox → Goat

Dog → Monkey

Gonn

BoRll

2

2

国

Real Image

'baby'

'girl'

'woman'

'old woman'

'old man'

Real Image

'bear'

'lion'

'horse'

'oil pastel'

Synthetic image editing. Text-to-image DMs provide impressive results, but it is difficult to generate images perfectly aligned with user intentions [1]. Although modifying prompts can guide generation, it can also introduce unintended changes in the generated content. Our results, shown in Figure 5, demonstrate that the proposed method can edit image contents while maintaining the original-promptrelated identity of the generated images thanks to the EBCU, which continuously harmonizes the latent representations and text prompts. While composable-diff [22] can generate compositions, they often fail to preserve the original properties of images, such as the color in the second row, or fail to compose the desired concept, such as the boat in the first row.

Real image editing. We demonstrate that the proposed framework can also edit real images, which is especially challenging as existing methods often dramatically alter input content or introduce unexpected variations. To do this, we integrate our framework with DDIM inversion [35, 25], which inverts images with meaningful text prompts into the domain of pre-trained diffusion models. First, we use the image captioning network BLIP [20] to automatically caption the interested image, following [28]. Next, we obtain the inverted noise latents of the input image using Diffusion Pivotal Inversion [25]. Then, we apply EBCQ and EBCU while denoising the inverted latents. The optimized unconditional textual embedding vector is also targeted for additional EBCUs. The results in Figure 6 demonstrate that our method achieves better editing performance by avoiding undesired changes.

Quantitative comparisons. We further conducted a comparative analysis of the proposed framework against several state-of-the-art diffusion-based image editing methods [24, 12, 25, 28]. For our evaluations, we focus on two image-to-image translation tasks. The first one is an animal transition task, which translates (1) cat → dog, (2) horse → zebra, and (3) adding glasses to cat input images (cat → cat with glasses). The second one is a human transition task, which translates (1) woman → man, (2) woman → woman with glasses, and (3) woman → man with glasses. Source images are retrieved from the LAION 5B dataset [34] and CelebA-HQ [15]. Motivated by [39, 28], we measure CLIP Accuracy and DINO-ViT structure distance. Experimental details are provided in the appendix E including explanations of baselines, datasets, metrics, and hyperparameter configurations.

DDIM inversion

Source

+word swap

Pix2Pix-zero

Ours

Source

Cycle-Diff

Pix2Pix-zero

Ours

Cat→Dog

Man

Horse →Zebra

Woman

w/glasses

w/glasses

Man

Table 1 and 2 in the appendix show that the proposed energy-based framework gets a high CLIP-Acc while having low Structure Dist. It implies that the proposed framework can perform the best edit while still retaining the structure of the original input image. While DDIM + word swap records remarkably high CLIP-Acc in horse → zebra task, Figure 8 shows that such improvements are based on unintended changes in the overall structure. More visualizations are provided in the appendix E.

5 Conclusion, Limitations and Societal Impacts

Conclusion. In this work, we formulated the cross-attention with an energy perspective and proposed the EBCU, a modified cross-attention that could implicitly minimize the energy in the latent space. Furthermore, we proposed EBCQ which is inspired by energy-based formulation of multiple text composition. The proposed method is versatile as shown by multiple applications, theory-grounded, easy to implement, and computationally almost free.

Limitations and Societal Impacts. The framework presented in this study generates images based on user intentions, which raises concerns regarding potential misuse for creating deepfakes or other forms of disinformation. It is crucial to ensure that these methods are implemented ethically and regulated appropriately. Additionally, while the framework performs well across various tasks, it requires pre-trained deep models, rendering it challenging to apply to out-of-domain datasets, such as medical images.

Acknowledgments

This research was supported by the KAIST Key Research Institute (Interdisciplinary Research Group) Project, National Research Foundation of Korea under Grant NRF-2020R1A2B5B03001980, Field-oriented Technology Development Project for Customs Administration through National Research Foundation of Korea(NRF) funded by the Ministry of Science & ICT and Korea Customs Service(**NRF-2021M3I1A1097938**), Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711137899, KMDF_PR_20200901_0015), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2021-0-02068, Artificial Intelligence Innovation Hub), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2023-00233251, System3 reinforcement learning with high-level brain functions), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2019M3E5D2A01066267).

References

M. Brack, F. Friedrich, D. Hintersdorf, L. Struppek, P. Schramowski, and K. Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprint arXiv:2301.12247 , 2023.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826 , 2023.

Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8188-8197, 2020.

M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and F. Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics , 168:288-299, 2017.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems , 33:6637-6647, 2020.

Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316 , 2020.

Y. Du and I. Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689 , 2019.

W. Feng, X. He, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032 , 2022.

A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

B. Hoover, Y. Liang, B. Pham, R. Panda, H. Strobelt, D. H. Chau, M. J. Zaki, and D. Krotov. Energy transformer. arXiv preprint arXiv:2302.07253 , 2023.

H. Huang, R. He, Z. Sun, T. Tan, et al. Introvae: Introspective variational autoencoders for photographic image synthesis. Advances in neural information processing systems , 31, 2018.

T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS , 2022.

T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

D. P. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630 , 2021.

D. Krotov and J. Hopfield. Dense associative memory is robust to adversarial inputs. Neural computation , 30(12):3151-3167, 2018.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation. In International Conference on Machine Learning , pages 1288812900. PMLR, 2022.

L. Liu, Y. Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

N. Liu, S. Li, Y . Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 423-439. Springer, 2022.

A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022.

C. Meng, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022.

W. Nie, A. Vahdat, and A. Anandkumar. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems , 34:13497-13510, 2021.

E. Nijkamp, M. Hill, S.-C. Zhu, and Y. N. Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems , 32, 2019.

G. Parmar, K. K. Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlovi´ c, G. K. Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217 , 2020.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

N. Tumanyan, O. Bar-Tal, S. Bagon, and T. Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1074810757, 2022.

N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel. Plug-and-play diffusion features for text-driven imageto-image translation. arXiv preprint arXiv:2211.12572 , 2022.

C. H. Wu and F. De la Torre. Unifying diffusion models' latent space, with applications to cyclediffusion and guidance. arXiv preprint arXiv:2210.05559 , 2022.

S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.05034 , 2022.

J. C. Ye. Geometry of Deep Learning . Springer, 2022.

A. L. Yuille and A. Rangarajan. The concave-convex procedure (cccp). Advances in neural information processing systems , 14, 2001.

A Proof of Theorem 1

In computing a derivative of a scalar, vector, or matrix with respect to a scalar, vector, or matrix, we should be consistent with the notation. In this paper, we follow the denominator layout convention as described in [42]. To make the paper self-contained, we briefly introduce the denominator layout and associate calculus.

The main motivation for using the denominator layout is from the derivative with respect to the matrix. More specifically, for a given scalar c and a matrix W ∈ R m × n , according to the denominator layout, we have

Furthermore, this notation leads to the following familiar result:

Accordingly, for a given scalar c and a matrix W ∈ R m × n , we can show that

in order to be consistent with (17), where VEC () and UNVEC () refer to the vectorization operation and its reverse, respectively. Under the denominator layout notation, for given vectors x ∈ R m and y ∈ R n , the derivative of a vector with respect to a vector given by

Then, the chain rule can be specified as follows:

Finally, the following property of the Kronecker delta product is useful throughout the paper [42]

Using this, we first prove the following key lemmas.

Lemma 1. For a given column vector x ∈ R N , we have

= softmax( β x )

Proof.

Lemma 2. Let k i denote the i -th row vector of K ∈ R N × d . Then, we have

where e i N represents a N -dimensional column vector where only the i -th entry is 1, with all other entries set to zero.

Proof. First, note that the expression k T i can be equivalently rewritten as K T e i . Then, we have

Furthermore, using (22), we have K T e i N = (( e i N ) T ⊗ I d ) VEC ( K T ) so that

where I d denotes the d × d identity matrix. Using the chain rule, we have

Thus, we have

where we again use (22). Therefore, by taking the transpose, we have

Q.E.D.

Lemma 3.

Proof. Note that

Hence, using (18) we have

Furthermore, using Lemma 1 we have

leading to the following:

where we again use (22). Now, by taking the transpose, we can prove (24). Q.E.D.

where diag( A ) denotes the sum of the diagonal element, i.e. trace of A , and x i is a column matrix whose i -th element is given by k i k T i / 2 , and [ · ] i dentoes the i -th element.

Proof. First,

where K ij denotes the ( i, j ) -th element. Therefore, using the denominator layout in (17), it is trivial to show that

Second, let us construct a column vector x whose i -th element is given by x i := k i k T i / 2 . Then, using Lemmas 1 and 2 and the chain rule, we have

This concludes the proof.

Theorem 1. For the energy functions

the gradient of the log posterior is given by:

Then, by using the chain rule the update rule of context vectors C is derived as follows:

where γ > 0 is a step size, and D ( · ) is a vector-to-diagonal-matrix operator.

Proof. Based on the Bayes' theorem, the gradient of the log posterior is derived as:

Second, by noting that E( K ) = log ∑ N i =1 exp( 1 2 k i k T i ) , Lemma 4 informs us

where D ( · ) is a vector-to-diagonal-matrix operator that takes N -dimensional softmax vector as an input and returns a N × N diagonal matrix with softmax values as main diagonal entries. Therefore, one can finally obtain:

By using the chain rule with K = CW K , the update rule of context vectors C is derived as in (11).

and

B Pseudo-code for EBCU and EBCQ

This section provides the description of the pseudocode for the proposed Energy-based Bayesian Context Update (EBCU) and Energy-based Composition of Queries (EBCQ). Algorithm 1 outlines the cascaded context propagation across cross-attention layers within the UNet model during the sampling step t . Note that the context is reinitialized at the beginning of each sampling step. On the other hand, Algorithm 2 details the EBCU implemented in each cross-attention layer. Specifically, the proposed EBCU provides a significant computational efficiency by reusing the similarity QK T , which requires computational cost O ( N 2 ) , to compute ∇ K E ( Q ; K ) . Consequently, there is only a small amount of additional computational overhead associated with the proposed EBCU.

Algorithm 1 Context cascade at sampling step t

Algorithm 2 Energy-based Bayesian Context Update (EBCU)

Algorithm 3 outlines the pseudocode for the EBCQ implemented for M given contexts. For the simplicity, we exclude the EBCU from the algorithm. Nontheless, the EBCU and the EBCQ could be leveraged together.

Algorithm 3 Energy-based Composition of Queries (EBCQ)

C Experimental setups

In this section, we describe detailed experimental setups for three applications including baseline method, hyper-parameter of the proposed method, and dataset if it is the case. Code: https://github.com/ EnergyAttention/Energy-Based-CrossAttention .

C.1 Common experimental setup

We mainly leverage pre-trained Stable Diffusion v1-5 (except Table 1: v1-4) which is provided by diffusers , a Python library that offers various Stable Diffusion pipelines with pre-trained models. All images are sampled for

50 steps via PNDM sampler [21] using NVIDIA RTX 2080Ti. In every experiment, we set the parameter α in Equation (9) to zero, focusing solely on controlling the values of γ attn and γ reg . EBCU is applied to every task, and EBCQ is additionally employed in C.4.

Different learning rate for each token It is worth noting that the γ attn and γ reg could be expressed as vectors. In other words, if the context C ∈ R N × d c is given, γ attn and γ reg are N -dimensional vectors. Hence, we have the flexibility to adjust the learning rate γ {·} , allowing us to increase or decrease the impact of certain tokens based on the user's intent. Unless otherwise noted, γ attn and γ reg is set to a constant for each text token.

Learning rate scheduling Since the proposed EBCU is leveraged for the diffusion model, one can readily introduce scheduling strategies for γ attn and γ reg along the sampling step t . We implement multiple variants such as 'constant', 'step', and 'exponential decay' as follows.

where γ 0 is the initial value, ReLu ( x ) = 0 if x ≤ 0 , otherwise 1, τ denotes the temporal threshold, and λ denotes the decay ratio. Unless stated otherwise, the scheduling strategy is set to the 'constant'.

C.2 Multi-concept image generation

We compared the performance of the proposed method with Structured Diffusion [11] which does not require additional training as our method. We leveraged the open-sourced official implementation 1 .

For the proposed method, we set the γ attn and γ reg differently for each sample within [1e-2, 1.5e-2, 2e-2]. As shown in the following ablation studies E, large γ attn tends to generate saturated images while large γ reg results in mixed/vanished contents.

We found that using different learning rates for each context token is useful for multi-concept generation, especially when a single concept tends to dominate with a constant learning rate. For example, given the main prompt "A cat wearing a shirt" , we set the γ attn for the "shirt" to 3e-2, while γ attn is set to 1.5e-2 for other tokens. We have observed that doubling the γ attn for a text token to be emphasized is sufficient to achieve balanced multi-concept image generation for most cases.

C.3 Text-guided image inpainting

Additionally, we conducted a performance comparison between our proposed method and two alternative approaches: ( a ) Stable Inpaint 2 , which fine-tunes the weights of Stable Diffusion through inpainting training, and ( b ) Stable Repaint 3 , which leverages the work of Lugmayr et al. [23] on the latent space of Stable Diffusion for the inpainting task. In the case of Stable Repaint, the mask is downsized and transferred into the latent space. We applied the Energy-based Bayesian Context Update (EBCU) technique to both methods, resulting in improved results compared to their respective baselines.

Masked EBCU. To further enhance the performance for the inpainting task, we introduce the concept of masked Energy-based Bayesian Context Update (masked EBCU). Specifically, let M ∈ R P 2 l × P 2 l represent a diagonal matrix where the main diagonal values are derived from the downsampled inpainting mask for the l -th cross-attention layer, with an output spatial size of P 2 l . In Equation (36), we modify the attention term (12) by incorporating the downsampled mask, effectively covering the query matrix as follows:

As evident in Equation (12), the attention term updates the context vectors, aligning k i towards q j , j = 1 , . . . , P 2 l , while considering the alignment strength between each q j and k i . However, in the inpainting task, we have prior knowledge that the context vectors should be most aligned with the semantically relevant masked regions. Therefore, we mask out unrelated background spatial representations, allowing for the context vectors to be updated with a specific focus on the masked regions. This approach facilitates the incorporation of semantic information encoded by k i specifically into the spatial mask regions.

In our proposed method, we set different values for γ attn and γ reg for each sample, selected from the set [1e-2, 1.5e-2, 2e-2, 2.5e-2], to account for variations in the input samples.

C.4 Image editing via compositional generation

We present empirical evidence demonstrating the effectiveness of our energy-based framework for compositional synthetic and real-image editing. The Energy-based Bayesian Context Update (EBCU) technique can be readily applied to both the main context vector ( C 1 in Section 3.2, s = 1 ) and editorial context vectors ( C s> 1 ). Each EBCU operation influences the attention maps used in the Energy-based Composition of Queries (EBCQ), enhancing the conveyance of semantic information associated with each context. Note that α s in (16) represents the degree of influence of the s -th concept in the composition. In practice, we fix α 1 = 1 for the main context, while α s> 1 is tuned within the range of (0.5, 1.0).

Let γ attn,s and γ reg,s denote the step sizes for EBCU of the s -th context vector. If the editing process involves changing the identity of the original image (e.g., transforming a "cat" into a "dog"), we set both γ attn, 1 and γ reg, 1 to zero. Otherwise, if the editing maintains the original identity, we choose values for γ attn, 1 and γ reg, 1 from the range of (5e-4, 1e-3), similar to γ attn, ( s> 1) and γ reg, ( s> 1) . All hyperparameters, including α s and γ s , are fixed during the quantitative evaluation process (more details in Section D and Table 3).

To ensure consistent results, we maintained a fixed random seed for both real and synthetic image editing. For real image editing, we employed null-text pivotal inversion [25] to obtain the initial noise vector.

During the reverse diffusion process in Sections C.2 and C.3, we kept γ fixed as a constant value. However, for compositional generation, we utilized step scheduling (Equation 35) for γ s and α s . After converting the initial noise vector for real images or using a fixed random seed for synthetic images, EBCU and EBCQ are applied after a threshold time τ s > 0 for the s -th editorial context. This scheduling strategy helps to preserve the overall structure of generated images during the editing process. In our observations, a value of τ s ∈ [10 , 25] generally produces satisfactory results, considering a total number of reverse steps set to 50. However, one can increase or decrease τ s for more aggressive or conservative editing, respectively.

The exemplary real images presented in Figures 5 and 6 of the main paper were sampled from datasets such as FFHQ [17], AFHQ [5], and ImageNet [7]. For a detailed quantitative analysis, please refer to Section D.

D Quantitative Comparison

In this section, we conducted a comparative analysis of the proposed framework against several state-of-the-art diffusion-based image editing methods [24, 12, 25, 28], following the experimental setup of [28]. To ensure a fair comparison, all methods utilize the pre-trained Stable Diffusion v1-4, employ the PNDM sampler with an equal number of sampling steps, and adopt the same classifier-free guidance scale.

D.1 Baseline Methods

In addition to the Plug-and-Play method discussed in the main paper, we include the following baselines for comprehensive quantitative comparison:

SDEdit [24] + word swap. This method introduces the Gaussian noise of an intermediate timestep and progressively denoises images using a new textual prompt, where the source word (e.g., Cat ) is replaced with the target word (e.g., Dog ).

Prompt-to-prompt (P2P) [12]. P2P edits generated images by leveraging explicit attention maps from a source image. The source attention maps M t are used to inject, re-weight, or override the target maps based on the desired editing operation. These original maps act as hard constraints for the edited images.

DDIM + word swap [25]. This method applies null-text inversion to real input images, achieving high-fidelity reconstruction. DDIM sampling is then performed using inverted noise vectors and an edited prompt generated by swapping the source word with the target.

pix2pix-zero [28]. pix2pix-zero first derives a text embedding direction vector △ c edit from the source to the target by using a large bank of diverse sentences generated from a state-of-the-art sentence generator, such as GPT-3 [2]. Inverted noise vectors are denoised with the edited text embedding, c + △ c edit, and cross-attention guidance to preserve consensus.

Cycle-diffusion [40]. Cycle-diffusion reformulates diffusion models as deterministic mappings from a Gaussian latent code to images. It presents a DPM-Encoder that allows for encoding images into this latent space.

D.2 Dataset

For the animal transition task, we focus on three image-to-image translation tasks: (1) translating cats to dogs (cat → dog), (2) translating horses to zebras (horse → zebra), and (3) adding glasses to cat input images (cat → cat with glasses). Following the data collection protocol of [28], we retrieve 250 relevant cat images and 213

horse images from the LAION 5B dataset [34] using CLIP embeddings of the source text description. We select images with a high CLIP similarity to the source word for each task.

For the human transition task, we focus on three image-to-image translation tasks: (1) woman → man, (2) woman → woman w/ glasses, and (3) woman → man w/ glasses. For this, we select source images of women without wearing glasses from CelebA-HQ. To ensure fair comparisons, all methods leverage the same version of Stable Diffusion, sampler, sampling steps, etc.

D.3 Metrics

Motivated by [39, 28], we measure CLIP Accuracy and DINO-ViT structure distance. Specifically, ( a ) CLIP Acc represents whether the targeted semantic contents are well reflected in the generated images. It calculates the percentage of instances where the edited image has a higher similarity to the target text, as measured by CLIP, than to the original source text [28]. On the other hand, ( b ) structure distance [39, 38] measures whether the overall structure of the input image is well preserved. It is defined as the difference in self-similarity of the keys extracted from the attention module at the deepest DINO-ViT [3] layer.

D.4 Details

The main context vector C main is encoded given a main prompt automatically generated by BLIP [20]. In addition, the editorial context vectors C src and C tgt are encoded given the text descriptions of the source and target concept, i.e. source and target prompt. For example, for a cat → dog task (cat → cat w/ glasses), the source prompt is "cat" ( "cat wearing glasses" ), and the target prompt is "dog" ( "without glasses" ). Then we apply EBCU and EBCQ based on the obtained context vectors. Please refer to Table 3 for the hyperparameter configurations.

D.5 Results

Table 1 and 2 show that the proposed energy-based framework gets a high CLIP-Acc while having low Structure Dist. It implies that the proposed framework can perform the best edit while still retaining the structure of the original input image. This is a remarkable result considering that the proposed framework is not specially designed for the real-image editing task. Moreover, the proposed framework does not rely on the large bank of prompts and editing vector △ c edit [28] which can be easily incorporated into our method.

While DDIM + word swap records remarkably high CLIP-Acc in horse → zebra task, Figure 9 and 16 show that such improvements are based on unintended changes in the overall structure. Table 3 summarizes the hyperparameter settings for each task. Examples of results are presented in Figure 17 and 16.

Real Image

DDIM inversion

Ours

DDIM inversion

+ word swap

E Ablation study and more results

𝛾

!"#

𝛾

$%%&

0.0

0.005

0.01

0.015

0.02

0.025

0.03

0.0

0.015

0.02

0.025

0.01

'A teddy bear'

Real Image

CACAO only

BCU + CACAO

Cat → Dog

Horse → Zebra

Real Image

CACAO only

BCU + CACAO

Stable Diffusion

𝛾

!"#

= 0

𝛾

$%%&

= 0

𝛾

$%%&

, 𝛾

!"#

≠ 0

A cat and a dog

A lion and a crown

Stable Diffusion

𝛾

!"#

= 0

𝛾

$%%&

= 0

𝛾

$%%&

, 𝛾

!"#

≠ 0

Real Image

(+) Dog

(+) Dog (-) Cat

Cat → Dog

Real Image

(+) w/ glasses

(+) w/ glasses

(-) w/o glasses

Cat → Cat w/ glasses

d.com-337178582

I$371781PF

Attention and regularization terms. To access the degree of performance improvement attained by the proposed EBCU, we conducted an ablation study for the attention and the regularization terms by regulating γ attn and γ reg for the text-guided image inpainting (Figure 10) and the multi-concept image generation (Figure 11). From the Figure 10, we can observe that the desired content is generated when proper range of γ attn and γ reg are given. Specifically, once γ reg is set to a valid value, the EBCU consistently generate a "teddy bear" with various γ attn , otherwise it generates background or imperfect objects. This result emphasizes the role of the introduced prior energy E( K ) . Furthermore, the γ attn also affects to the context alignment of the generated sample (for instance γ attn = 0 . 025 and γ reg = 0 . 02 ), which highlights the importance of the introduced conditional energy function E( Q ; K ) . The same evidences could be found in the first row in Figure 11 which are the multi-concept image generation examples.

Synergy between EBCU and EBCQ. While both EBCU and EBCQ are designed from the common energybased perspective, each operation is originated from different energy functions E( K ; Q ) and ˆ E( Q ; { K s } s M =1 ) , respectively. This fact suggests the synergistic energy minimization by combining the EBCU and EBCQ, which could further improve the text-conditional image generation. To investigate this further, we conducted an ablation study using a real image editing application. Specifically, we compared the editing performance when solely utilizing EBCQ and when combining EBCU with EBCQ. The second row in Figure 11 is the result of the ablation study that shows fully-compatibility of the EBCU and EBCQ. Importantly, the incorporation of the EBCU improves the quality of the generated images. While the EBCQ alone effectively captures the context of the given editing concept, the addition of EBCU enhances the fine-grained details in the generated outputs.

Importance of concept negation. Remark that a negative α s in (16) denotes the negation of given editing prompt. We empirically observed that the concept negation may significantly contribute to the performance of compositional generation. Specifically, for the image-to-image translation task in Table 1, we apply both positive and negative guidance with the target (e.g. Dog ) and source (e.g. Cat ) concepts, respectively, following the degree of guidance denoted in Table 3. The third row in Figure 11 shows the impacts of source concept negation in the image-to-image translation task. While the positive guidance alone may fail to remove the source-concept-related features, e.g. eyes of the Cat , the negative guidance removes such conflicting existing attributes. This implies that the proposed framework enables useful arithmetic of multiple concepts for both real and synthetic image editing.

Prior energy and α . While α 2 diag( KK T ) in (8) penalizes norm of each context vectors uniformly, the proposed prior energy function E( K ) adaptively regularizes the smooth maximum of ∥ k i ∥ . Intuitively, adaptive penalization prevents the excessive suppression of context vectors, potentially resulting in images that are more semantically aligned with a given context. To demonstrate the effectiveness of adaptive penalization in the prior energy function, we conducted a multi-concept image generation task with varying α in (11) from 0 to 1, while fixing other hyperparameters. Figure 12 illustrates the gradual disappearance of salient contextual elements in the generated images depending on the change of α . Specifically, the crown is the first to diminish, followed by

subsequent context elements, with the lion being the last to vanish with α = 1 . This result highlights the validity of the adaptive penalization for the context vectors which stems from the prior energy function.

Context shift analysis. Since the proposed framework updates the initial context vector, one may be concerned that the optimized vector may be severely shifted from the original context so that it loses the intended semantic meaning. Here, we emphasize that the change of context vector is adaptive, not an invalid shift, because the proposed BCU allows an adaptive propagation of the context vector through UNet, resulting in a context vector that is better aligned with Q t on cross-attention space.

To further support our claim, we use the updated C T ( T for a number of sampling time steps) from the proposed method as a fixed context vector instead of C clip and perform multi-concept generation using the conventional cross-attention operation. Since the energy functions are defined differently for each sampling time step, using C T as a fixed context for each sampling time step may result in low-quality samples. However, this approach allows us to evaluate whether the updated context contains the correct semantics of the given textual conditions, in contrast to C clip. Figure 13a demonstrates that the updated context vector does indeed capture the correct semantics of the given textual conditions (e.g., both black horse and yellow room).

Multi-step update. In order to better understand the context update, we implement a multistep context update for comparison while keeping the propagation of C to subsequent layers. For the multi-concept generation and inpainting tasks, we observe that this multiple update of the context vector actually improves the quality of generated images (Figure 13b). This is in the same line as our perspective that one forward path of context vector is equivalent to one-step gradient descent for energy function. That being said, we also observe that the multiple context update is relatively computationally expensive and a single-step update is usually sufficient for improved performance. Therefore, we decide to use a single-step update with context propagation to subsequent layers. We believe these results may lay down foundation for further research on understanding the alignment between latent image representations and context embeddings.

𝛼 =0.0

𝛼 =1.0

𝛼 =0.5

(A)

Fixed Cclip

Adaptive

Ct

FixedCr=50

1 step

3steps

5steps

10 steps

+bowl,tomato

+smile

+Monet,Colorful,Sea

-mustache,cigarette

Stable-Diff

Ours

'A green apple and a red banana'

Stable-Diff

Ours

'A blue dog on the orange sofa'

'A black horse in a yellow room'

Stable-Diff

Ours

Stable-Diff

Ours

'A black plate and a blue flower'

'A cat wearing a shirt'

'A lion and a crown'

Stable-Diff

Ours

Stable-Diff

Ours

福

Real Image

DDIM inversion

+ word swap

Pix2Pix-zero

Ours

Real Image

DDIM inversion

+ word swap

Pix2Pix-zero

Ours

Real Image

DDIM inversion

+ word swap

Pix2Pix-zero

Ours

Real Image

DDIM inversion

+ word swap

Pix2Pix-zero

Ours

Real Image

'Rat'

'Cat'

'Dog'

'Tiger'

Synthetic Image

'Fall'

'Winter'

'Watercolor'

Peer reviews:
comments: SUMMARY:
This paper proposed a training-free algorithm to modife cross-attention during inference time that could implicitly minimize the energy in the latent space. This paper formulate this idea from a energy perspective. The authors conduct three experiments - multi-concept generation, image inpainting, compositional editing to demonstrate the effectivenns of the new idea.

STRENGTHS:
1. The idea from energy perspective is innovative and the derivation of the equations are solid.
2. Figure 2 an the associate caption is straightforward that shows the relation between energy value and multi-concept generation quality.

WEAKNESSES:
1. In experiment section, the authors provide only some generation cases for subjective-quality analysis without any quantitative results. The shown cases after cherry-pick maybe not adequate to demonstrate the effectiveness of the method proposed in this paper.
2. A minor typo in Line 83: "update rule for a state pattern \epsilon" -> "update rule for a state pattern \zeta"

QUESTIONS:
1. You metions "nested hierarchy of energy functions" many times? But what is the definition of it and there's any reference papers?
2. In Eq. (6), what is the definition of softmax_1 and softmax_2? And how to obtain Eq. (6) from Q=softmax_2(\betaQK^T)K? I am a little confused.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel **energy-based** framework that can automatically **update the context** used in cross-attention **without additional training**. They claim the proposed updating process well solves the **semantic misalignment** issue in text-to-image diffusion models.\n\nSTRENGTHS:\n1. The paper exhibits a clear, engaging, and concise writing style.\n2. The paper introduces an interesting energy definition and proves the subtle connections with the attention mechanism used in transformers. \n3. The experimental results look promising.\n\nWEAKNESSES:\n1. The definition of the energy function does not appear reasonable. For instance, consider the definition of E(K) in equation 8. E(K) increases as the L2-Norm of k_i, where i = 1, 2, ..., N, increases. The minimum value of E(K) is achieved when all k_i converge to zero. This suggests that the author assumes k_i = 0 has the highest probability (since energy is minimum), which contradicts reality. Additionally, the definition of E(Q;K) in equation 7 also seems unconventional. It resembles E(K;Q) rather than E(Q;K). I have significant doubts regarding the effectiveness of the proposed energy definition. I kindly request the authors to provide a more explicit explanation for why they have defined the energy function in such a manner and clarify the actual significance of these proposed energy functions.\n\n2. There is a lack of quantitative experimental results. While the authors present some promising outcomes, there is a scarcity of extensive quantitative experiments to demonstrate that the proposed framework statistically outperforms the naive cross-attention mechanism. It is imperative to include detailed information about the quantitative experiments conducted to validate the efficacy of the proposed framework.\n\n3. There is no explicit explanation of why energy minimization can help solve semantic misalignment. I can not draw direct connections between them in the paper.\n\nI tend to reject this paper.\n\nQUESTIONS:\n1. The definition of the energy function does not appear reasonable.\n2. Lack of quantitative experimental results.\n3. Lack of explanations of why energy minimization help slove semantic misalignment.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to formulate cross-attention layers using energy-based models such that by minimizing the cross attention energy with respect to the context latent representation, the method can further alleviate semantic misalignments between generated or edited samples and the input descriptions, and allow zero-shot compositional generalization using combinations of cross attention outputs based on multiple inputs.\n\nSTRENGTHS:\n1. The paper introduces a detailed theoretical formulation of cross attentions using energy-based perspective.\n2. The method outperforms existing methods qualitatively on various image generation tasks (e.g., inpainting, multi-concept generation and compositional generation) without additional training.\n3. The authors have conducted relatively comprehensive evaluations on the method.\n\nWEAKNESSES:\n1. **Lack of limitations**. We have seen the good side of such method, which is to improve semantic alignment between generated images and text inputs. However, there are often trade-offs. For example, energy-based optimization is quite unstable, so it requires some tuning in terms of step size and the number of optimization steps. It would be good to mention the limitation of the method, for example, how much the method is slower compared to the standard denoising steps. \n2. **Lack of experiments on Image quality**. Though I think only context vectors are optimized, image generation quality will be unlikely to change much. I would still suggest evaluating generated images would provide a way to help understand whether such algorithms can change the generation performance.  In addition, although semantic alignment seems to improve in qualitative results, we also need to measure it quantitatively. And current metrics (e.g., CLIP) for measuring semantic alignment can be unreliable, thus a human evaluation seems necessary to compare different methods quantitatively.\n\nQUESTIONS:\n1. is the method quite sensitive to the hyper-parameters that you use to optimize context vectors for each image? Based on the supplementary material, it seems to me that the method is quite sensitive and often requires a lot of tuning in terms of coefficients $\\gamma_{attn}, \\gamma_{reg}$ for updates, step size, etc.\n\nLIMITATIONS:\nThe authors have included limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the semantic misalignment problem of stable diffusion model using the energy-based model framework.The authors first show that each cross-attention in the diffusion model can be seen as one step optimization of a pre-defined energy function. They then formulate a Bayesian update for the context vector accordingly. The authors demonstrate the effectiveness of their method by showing qualitative samples in multi-concept generation, text-guided image inpainting as well as compositional generation tasks.\n\nSTRENGTHS:\nI love the novelty and theoretical framework of this paper. While there are several other works tackle the more controlable image generation/editing problem based on the modification of the cross attention layers, this paper provides a systematical theory framework for this. This might not only help with the improvement on certain task but also facilicate people to form a deeper understanding of the model itself.\n\nWEAKNESSES:\n1. Lack of quantitative results:\nWhile several qualitative samples are shown in the paper, quantitative evaluation over a large number of different samples can be more convincing to judge the model's performance against the baseline. The authors may consider using pretrained model (for example like [1] does) or using human evaluation.\n\n2.Choice of hyper-parameter:\n Looking at the supplementary, it seems that a successful editing requires one to choose parameters $\\alpha$, $\\beta$, $\\gamma$ case-by-case. Furthermore, it seems that changing hyper-parameters have a great influence over the generation performance.  This might limit the proposed method to be used in real applications.\n\n[1] Training Diffusion Models with Reinforcement Learning\n\nQUESTIONS:\n1. The understanding of the whole unet: While we may look at each individual cross-attention layer as updating a certain energy function, I would like to hear the authors' insight on how we can understand the unet as a whole. At different cross-attention layers, the weight are different. And there are a batch of linear and non-linear transformations between each two cross-attention layers. Then can we understand the whole unet as updating a certrain energy? (The cascading update of C seems to suggest that the optimization across the whole unet has some consistency.)  And if possible, how? \n\n2. The energy distribution across different attention layers: In figure 2, the authors plot the energy of 16 attention layers across different sampling steps. It seems that at all the steps, the energy will peak at the middle layer and reach bottom at the two ends. Can the authors provide some insight on why this happens?  \n\n3. In section 3.1 algorithm, the authors provide their design of updating context vector C. While this seems to work well on the samples shown in the paper, I'm wondering what if other update algorithms are use. For example, what if we don't cascade $C_{n+1, t}$ to the next (n+1)th layer but updates $C$ for all the layers starting from scratch and update them for more than one step? \n\n4. From figure 8 in the supplementary, it seems that increasing $\\gamma_{reg}$ instead of $\\gamma_{attn}$ encourages pattern to occur.  When $\\gamma_{reg}$ is zero, no matter how to tune $\\gamma_{attn}$, the editing cannot get successful results (no teddy bear here). This can be a bit counter-intuitive. As for my understanding, the attention term should be the one to align the key and query. Are there any explanations on why these happen?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an energy-based model (EBM) framework addresses semantic misalignment in text-to-image diffusion models by incorporating EBMs in each cross-attention layer, minimizing a nested hierarchy of energy functions, and achieving highly effective results in diverse image generation tasks. From the shown figures, they demonstrate the effectiveness of the EBM framework.\n\nSTRENGTHS:\n1, The proposed framework demonstrates general applicability to diverse tasks, such as inpainting, image editing, and multi-concept generation.\n\n2, Theoretical analysis and explanations are comprehensively conducted to enhance the robustness of the experimental results.\n\n3, Notably, this paper stands out as the first diffusion model study observed to incorporate an energy-based perspective into its formulation.\n\nWEAKNESSES:\n1, The experimental results are only including the qualitative comparison with some other methods. In my view, some quantitative numbers should also be shown in the paper to demonstrate the performance of this method compared to others.\n\n2, For multi-concept generation, the Custom Diffusion [1] has been released during the end of last year, it should be considered as a comparison. \n\n3, The same comparison limitations also happened in inpainting (considering Blended Diffusion[2], GLIDE[3]) and image editing (Prompt2Prompt[4], pix2pix-zero[5]).\n\n4, For evaluation metrics, there are also commonly used metrics in different sub-area. For example, the CLIP-score/Lpips/Structure-Dist can be considered in image editing tasks.\n\n\n[1] Multi-Concept Customization of Text-to-Image Diffusion\n[2] Blended Diffusion for Text-driven Editing of Natural Images\n[3] GLIDE: Towards photorealistic image generation and editing with text-guided diffusion model\n[4] Prompt-to-Prompt Image Editing with Cross Attention Control\n[5] Zero-shot Image-to-Image Translation\n\nQUESTIONS:\nAs I stated above, the main drawback of this paper is in the experimental part. There are lack of qualitative comparisons with correct methods. Instead, the original Stable Diffusion model is always serving as the baseline comparison. Furthermore, there are no sufficient quantitative results to show the efficiency of the proposed framework in this paper.\n\nLIMITATIONS:\nAs I stated in the weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes  Bayesian Context Update (BCU) and  Compositional Averaging of Cross-Attention Output (CACAO). The main idea is to view the cross-attention between text and image as optimizing an energy-based model, and modify the intermediate outputs according to the energy. A series of examples are shown to demonstrate the effectiveness of BCU and CACAO.\n\nSTRENGTHS:\n1. The examples are visually intriguing.\n2. The energy viewpoint of the cross-attention mechanism in Stable Diffusion could be enlightening.\n\nWEAKNESSES:\n1. I can understand that Q_t is the H * W * C dimensional feature in the network processing the latent-space image at time step t, and C_t is the context vector which is an optimizable variable added by the authors. C_t is initialized to C_clip, then it is optimized going through the layers. As the layers go deeper, C_t is shifted from C_clip, which means the semantic meaning is shifted from the original user text input. My question is: how severe is the shift? How do the authors control the shift? In fact, it makes more sense to me if Q_t,l are the variables to be optimized.\n\n2. More clarification on the examples in Figure 2 is required. I tried Stable Diffusion by myself, and found Stable Diffusion can generate \"A monkey and a hat\" with the correct semantic meaning. I kindly ask the authors to elaborate how the Stable Diffusion examples are picked. Moreover, Stable Diffusion + Ours degrades the image quality significantly (all the images become black-and-white). Could the authors provide some explanation on why that happens?\n\n3. More quantitative results should be added to prove the effectiveness of the proposed method. In the provided qualitative examples, the advantage of the proposed method over Composable-Diff and Plug-and-Play is marginal. Since the results of generative models are heavily dependent on the random seeds, I suggest the authors to provide systematic quantitative evaluation to benchmark the proposed method. I do notice the quantitative evaluation part in the Appendix, but only animal transition seems limited.\n\nQUESTIONS:\nPlease refer to Weakness.\n\nLIMITATIONS:\nLimitations are properly discussed at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to optimize the context representation through energy based formulation of the cross-attention within the U-Net at test-time to achieve semantic alignment between the textual representation and the image features in the U-Net. \nExperiments are performed for the multi-concept generation, text-guided inpainting and image editing.\n\nSTRENGTHS:\n+ The motivation of the paper is clear and is supported with adequate experiments. \n+ The approach allows for compositional generation. The qualitative results show that the approach does better than simple DDIM+inversion. \n+ Multiple concepts can be composed together in the generated images with test-time optimization.\n\nWEAKNESSES:\n- The formulation of the gradient posterior in Eq. 9 is missing the term on the expectation of the gradient on the right side? \n- The difference and the motivation to optimize the energy wrt the keys instead of queries is not so clear? Ablations with the two approaches can be done to show the benefits and attention visualizations may help to evaluate the difference between the two. \n- The subscripts 1, 2 below the softmax are not defined. \n- In lines 150-151 what is meant by the \"forward path of cross-attention\"?\n- Low sample diversity: Example in figure 2 seems to harm the sample diversity wrt Stable diffusion.\n- What is the computational overhead? How many optimization steps are required to converge to a good solution?\n\nQUESTIONS:\nPlease see weaknesses above. \n\nSpecifically following clarifications are required: \n\n1. The motivation for optimizing keys over queries. \n2. The role of the regularization term in Eq. 12. \n3. In theorem 1 the updates are wrt to the keys, however, eq 14 considers the energy wrt to the queries. The whole idea  of the paper in terms of novelty  from theorem 1, then why use eq. 14? \n\nThe results and comparisons with the baselines can be moved to the main paper. Theorem 1 and Eq 14 need to be better discussed (better combined in one section) and highlighting the contributions and the prior work clearly.\n\nLIMITATIONS:\nLimitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}