SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the human transition task, we focus on three image-to-image translation tasks: (1) woman  $\rightarrow$  man, (2) woman  $\rightarrow$  woman w/ glasses, and (3) woman  $\rightarrow$  man w/ glasses. For this, we select source images of women without wearing glasses from CelebA-HQ. To ensure fair comparisons, all methods leverage the same version of Stable Diffusion, sampler, sampling steps, etc.
D.3 Metrics
Motivated by [39, 28], we measure CLIP Accuracy and DINO-ViT structure distance. Specifically, (a) CLIP Acc represents whether the targeted semantic contents are well reflected in the generated images. It calculates the percentage of instances where the edited image has a higher similarity to the target text, as measured by CLIP, than to the original source text [28]. On the other hand, (b) structure distance [39, 38] measures whether the overall structure of the input image is well preserved. It is defined as the difference in self-similarity of the keys extracted from the attention module at the deepest DINO-ViT [3] layer.
D.4 Details
The main context vector  $C_{main}$  is encoded given a main prompt automatically generated by BLIP [20]. In addition, the editorial context vectors  $C_{src}$  and  $C_{tgt}$  are encoded given the text descriptions of the source and target concept, i.e. source and target prompt. For example, for a cat → dog task (cat → cat w/ glasses), the source prompt is "cat" ("cat wearing glasses"), and the target prompt is "dog" ("without glasses"). Then we apply EBCU and EBCQ based on the obtained context vectors. Please refer to Table 3 for the hyperparameter configurations.
D.5 Results
Table 1 and 2 show that the proposed energy-based framework gets a high CLIP-Acc while having low Structure Dist. It implies that the proposed framework can perform the best edit while still retaining the structure of the original input image. This is a remarkable result considering that the proposed framework is not specially designed for the real-image editing task. Moreover, the proposed framework does not rely on the large bank of prompts and editing vector  $\triangle c_{\mathrm{edit}}$  [28] which can be easily incorporated into our method.
While DDIM + word swap records remarkably high CLIP-Acc in horse  $\rightarrow$  zebra task, Figure 9 and 16 show that such improvements are based on unintended changes in the overall structure. Table 3 summarizes the hyperparameter settings for each task. Examples of results are presented in Figure 17 and 16.
Table 2: (Human transition) Comparison to state-of-the-art diffusion-based editing methods. Dist for DINO-ViT Structure distance.
Method	(a) Woman → Man	(b) Woman → Woman w/ glasses	(c) Woman → Man w/ glasses
CLIP Acc (↑)	Dist (↓)	CLIP Acc (↑)	Dist (↓)	CLIP Acc (↑)	Dist (↓)
CycleDiffusion [40]	57.0%	0.156	74.0%	0.018	63.5%	0.237
Pix2pix-zero	93.0%	0.043	66.0%	0.024	93.9%	0.052
Stable Diffusion + ours	94.0%	0.035	99.0%	0.029	96.0%	0.037
Table 3: Hyperparameter configurations for each editing task. Each task index comes from Table 1.  $\gamma_{attn,main} = 0$  and  $\gamma_{reg,main} = 0$  as mentioned in section C.4. Note that  $\alpha_{src} < 0$  for the concept negation (related ablation study in Figure 11).  $\tau_s$  denotes the warm-up period for step scheduling in (35) and Section C.4.
Task	αsrc	αtgt	γ.,main	γattn,src	γreg,src	γattn,tgt	γreg,tgt	τs
(a)	-0.65	0.75	0	5e-4	5e-4	6e-4	6e-4	25
(b)	-0.5	0.6	0	4e-4	4e-4	5e-4	5e-4	15
(c)	-0.6	0.7	0	1e-3	1e-3	1e-3	1e-3	17
Figure 9: Image editing comparison with DDIM-inversion. Generated samples by DDIM-inversion with word swap readily deviate the original data contents, while the proposed method avoids undesired changes.
E Ablation study and more results
Figure 10: Ablation results for  $\gamma_{attn}$  and  $\gamma_{reg}$ . All samples are generated from the same random noise.
Figure 11: Ablation study results. The first row shows multi-concept generation examples with varying  $\gamma_{attn}$  and  $\gamma_{reg}$ , while the second row shows real image editing examples with varying the usage of EBCU and EBCQ. The last row shows the effect of negative prompt for the image editing application.
Attention and regularization terms. To access the degree of performance improvement attained by the proposed EBCU, we conducted an ablation study for the attention and the regularization terms by regulating  $\gamma_{attn}$  and  $\gamma_{reg}$  for the text-guided image inpainting (Figure 10) and the multi-concept image generation (Figure 11). From the Figure 10, we can observe that the desired content is generated when proper range of  $\gamma_{attn}$  and  $\gamma_{reg}$  are given. Specifically, once  $\gamma_{reg}$  is set to a valid value, the EBCU consistently generate a "teddy bear" with various  $\gamma_{attn}$ , otherwise it generates background or imperfect objects. This result emphasizes the role of the introduced prior energy  $\operatorname{E}(\pmb{K})$ . Furthermore, the  $\gamma_{attn}$  also affects to the context alignment of the generated sample (for instance  $\gamma_{attn} = 0.025$  and  $\gamma_{reg} = 0.02$ ), which highlights the importance of the introduced conditional energy function  $\operatorname{E}(\pmb{Q};\pmb{K})$ . The same evidences could be found in the first row in Figure 11 which are the multi-concept image generation examples.
Synergy between EBCU and EBCQ. While both EBCU and EBCQ are designed from the common energy-based perspective, each operation is originated from different energy functions  $\mathrm{E}(\pmb {K};\pmb {Q})$  and  $\hat{\mathrm{E}} (\pmb {Q};\{\pmb {K}_s\}_{s = 1}^M)$  respectively. This fact suggests the synergistic energy minimization by combining the EBCU and EBCQ, which could further improve the text-conditional image generation. To investigate this further, we conducted an ablation study using a real image editing application. Specifically, we compared the editing performance when solely utilizing EBCQ and when combining EBCU with EBCQ. The second row in Figure 11 is the result of the ablation study that shows fully-compatibility of the EBCU and EBCQ. Importantly, the incorporation of the EBCU improves the quality of the generated images. While the EBCQ alone effectively captures the context of the given editing concept, the addition of EBCU enhances the fine-grained details in the generated outputs.
Importance of concept negation. Remark that a negative  $\alpha_{s}$  in (16) denotes the negation of given editing prompt. We empirically observed that the concept negation may significantly contribute to the performance of compositional generation. Specifically, for the image-to-image translation task in Table 1, we apply both positive and negative guidance with the target (e.g. Dog) and source (e.g. Cat) concepts, respectively, following the degree of guidance denoted in Table 3. The third row in Figure 11 shows the impacts of source concept negation in the image-to-image translation task. While the positive guidance alone may fail to remove the source-concept-related features, e.g. eyes of the Cat, the negative guidance removes such conflicting existing attributes. This implies that the proposed framework enables useful arithmetic of multiple concepts for both real and synthetic image editing.
Prior energy and  $\alpha$ . While  $\frac{\alpha}{2} \mathrm{diag}(\boldsymbol{K}\boldsymbol{K}^T)$  in (8) penalizes norm of each context vectors uniformly, the proposed prior energy function  $\operatorname{E}(\boldsymbol{K})$  adaptively regularizes the smooth maximum of  $\| \boldsymbol{k}_i\|$ . Intuitively, adaptive penalization prevents the excessive suppression of context vectors, potentially resulting in images that are more semantically aligned with a given context. To demonstrate the effectiveness of adaptive penalization in the prior energy function, we conducted a multi-concept image generation task with varying  $\alpha$  in (11) from 0 to 1, while fixing other hyperparameters. Figure 12 illustrates the gradual disappearance of salient contextual elements in the generated images depending on the change of  $\alpha$ . Specifically, the crown is the first to diminish, followed by
subsequent context elements, with the lion being the last to vanish with  $\alpha = 1$ . This result highlights the validity of the adaptive penalization for the context vectors which stems from the prior energy function.
Context shift analysis. Since the proposed framework updates the initial context vector, one may be concerned that the optimized vector may be severely shifted from the original context so that it loses the intended semantic meaning. Here, we emphasize that the change of context vector is adaptive, not an invalid shift, because the proposed BCU allows an adaptive propagation of the context vector through UNet, resulting in a context vector that is better aligned with  $Q_{t}$  on cross-attention space.
To further support our claim, we use the updated  $C_T$  ( $T$  for a number of sampling time steps) from the proposed method as a fixed context vector instead of  $C_{\mathrm{clip}}$  and perform multi-concept generation using the conventional cross-attention operation. Since the energy functions are defined differently for each sampling time step, using  $C_T$  as a fixed context for each sampling time step may result in low-quality samples. However, this approach allows us to evaluate whether the updated context contains the correct semantics of the given textual conditions, in contrast to  $C_{\mathrm{clip}}$ . Figure 13a demonstrates that the updated context vector does indeed capture the correct semantics of the given textual conditions (e.g., both black horse and yellow room).
Multi-step update. In order to better understand the context update, we implement a multistep context update for comparison while keeping the propagation of  $C$  to subsequent layers. For the multi-concept generation and inpainting tasks, we observe that this multiple update of the context vector actually improves the quality of generated images (Figure 13b). This is in the same line as our perspective that one forward path of context vector is equivalent to one-step gradient descent for energy function. That being said, we also observe that the multiple context update is relatively computationally expensive and a single-step update is usually sufficient for improved performance. Therefore, we decide to use a single-step update with context propagation to subsequent layers. We believe these results may lay down foundation for further research on understanding the alignment between latent image representations and context embeddings.
Figure 12: Generated samples with varying  $\alpha$  values. As  $\alpha$  increases, the generated images progressively deviate from the intended context, "A lion and a crown".
(A)
Fixed  $C_{clip}$
Adaptive  $C_t$
Fixed  $C_{T = 50}$
Figure 13: (a) Multi-concept generation results with fixed and adaptive context embeddings. For the last column, updated context via the BCU at the final timestep is given as a fixed context embedding. (b) Multi-concept image generation results in a different number of context updates.  $\gamma_{attn}$  and  $\gamma_{reg}$  are reduced in proportion to the number of updates.
Figure 14: Further results for compositional real-image editing.
Figure 15: Further results for multi-concept image generation. Best views are displayed.
Figure 16: Further results for real image editing: horse to zebra.
Figure 17: Further results for real image editing: cat to dog.
Figure 18: Further results for image editing with varying text prompts. Best views are displayed.