InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences
Authors: Chenyang Zhu
Kai Li
Yue Ma
Longxiang Tang
Chengyu Fang
Chubin Chen
Qifeng Chen
Xiu Li
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-image generation
Image editing
Customized concept swapping
Primary area: generative models
TL;DR: We propose InstantSwap, a novel training-free customized concept swapping framework, which enables efficient concept swapping across sharp shape differences.

Abstract: Recent advances in Customized Concept Swapping (CCS) enable a text-to-image model to swap a concept in the source image with a customized target concept. However, the existing methods still face the challenges of $\textit{\textbf{inconsistency}}$ and $\textit{\textbf{inefficiency}}$. They struggle to maintain consistency in both the foreground and background during concept swapping, especially when the shape difference is large between objects. Additionally, they either require time-consuming training processes or involve redundant calculations during inference. To tackle these issues, we introduce InstantSwap, a new CCS method that aims to handle sharp shape disparity at speed. Specifically, we first extract the bbox of the object in the source image $\textit{automatically}$ based on attention map analysis and leverage the bbox to achieve both foreground and background consistency. For background consistency, we remove the gradient outside the bbox during the swapping process so that the background is free from being modified. For foreground consistency, we employ a cross-attention mechanism to inject semantic information into both source and target concepts inside the box. This helps learn semantic-enhanced representations that encourage the swapping process to focus on the foreground objects. To improve swapping speed, we avoid computing gradients at each timestep but instead calculate them periodically to reduce the number of forward passes, which improves efficiency a lot with a little sacrifice on performance. Finally, we establish a benchmark dataset to facilitate comprehensive evaluation. Extensive evaluations demonstrate the superiority and versatility of InstantSwap.

Published as a conference paper at ICLR 2025
INSTANTSWAR FAST CUSTOMIZED CONCEPT SWAP- PING ACROSS SHARP SHAPE DIFFERENCES
Chenyang Zhu
Kai Li T, Yue Ma Longxiang Tang', Chengyu Fang', Chubin Chen',
Qifeng Chen Xiu Li
Tsinghua University https://instantssww github..h...... 10/
Meta HKUST
Image
Customized Clancupt
Image
flower
aunglasses
f
f
Source image
Durs
lotux
annelope
Concept Swap ??:
Rerrow
(a) Concept Swap
(b)
Concept Insertion /Renoval
Figure I: Visual results of INSTANTSWAP. Our approach can seamlessly swap source concept with a customized concept in an image, even with great shape differences. Moreover, INSTANTSWAP can be used for other tasks, such as concept insertion and removal.
ABSTRACT
Recent advances Customized Concept Swapping (CCS) enable a text-to-image model to swap a concept in the source image with customized target concept. However, the existing methods still face the challenges of inconsistency and inef- ficiency. They struggle to maintain consistency in both the foreground and back- ground during concept swapping, especially when the shape difference is large between objects. Additionally, they either require time-consuming training pro- cesses or involve redundant calculations during inference. To tackle these issues, we introduce INSTANTSWAP. new CCS method that aims to handle sharp shape disparity at speed. Specifically, we first extract the bbox of the object in the source image automatically based on attention map analysis and leverage the bbox achieve both foreground and background consistency. For background consis- tency, remove the gradient outside the bbox during the swapping process that the background is free from being modified. For foreground consistency. we employ a cross-atted -atssant mechanism inject semantic information into both source and target concepts inside the box. This helps learn semantic-enhancee rep- resentations that encourage the swapping process to focus on the foreground ob- jects. To improve swapping speed, we avoid computing gradients at each timestep but instead calculate them periodically to reduce the number of forward passes, which improves efficiency lot with little sacrifice on performance. Finally, we establish a benchmark dataset to facilitate comprehensive evaluation. Extensive evaluations demonstrate the superiority and versatility of INSTANTSWAP.
INTRODUCTION
We explore the task of Customized Concept Swapping (CCS), a subtask of text-to-image (T2I) gen- eration, which aims to replace concept in a source image with a highly customized new concept. Combined with diffusion models (Dhariwal & Nichol 202T Rombach et al., 2022; Nichol et al., 2021), recent CCS methods demonstrate widespread applicability in areas such selfie enhance- ment, photo blog creation, and comic creation.
Early work (Yang et al.) (003) in CCS primarily relies on copy-paste techniques, which are rough and unreliable. By integrating powerful customization techniques (Ruiz et al. 2023 Kumari et al.)
Equal contribution.
Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Source Taage
Custowlzed Cencept
PrstoSepp
PrPinviRg
S06 306 C36 Drg
Spore Sam. UISKO
boxed
Wecknesa:
Foreground Ttoond!
Roshneso: Foregrunn and Rodigreund Incoretetency
kitten playing pool with balls a kitten playing pool with a aks candle
Figure 2: Our INSTANTSWAP achieves better swapping consistency than the existing methods.
2023) with image editing methods (Hertz et al. 2022 2023), series of works (Gu al. 2024a
Choi et al. 2023 Li et al. 2023b Gu et al. 2024b) have been proposed. Although achieving remarkable success, these approaches still face the problems of inconsistency and inefficiency as shown in Fig. [2] (1) Inconsistency: Attention-
methods such as PhotoSwap (Gu et al. (024a) and P2P Hertt al.] 2022) maintain background consistency well but struggle with shape differ- ences between source and target concepts, resulting in foreground inconsistency. Score distillation based methods such as SDDS (Poole et al. 2022). DDS (Hertz al., 2023), and CDS (Nam al.] 2024) fail to generate foreground concepts precisely and alter the background significantly, causing both foreground and background inconsistency. (2) Inefficiency: Attention-based methods require
an inefficient training phase (Mokady et al., 2023 Ju et al.) 2024) on source image to maintain background consistency. While score distillation based methods are training-free, they still require redundant calculations of forward passes at each timestep, leading to inference inefficiency.
To address the aforementioned issues, we propose INSTANTSWAP, training-free framework that efficiently performs customized concept swapping across shape differences while maintaining both foreground and background consistency. Specifically, we extract the bounding box (bbox) that indicates the position of the source concept from the enhanced cross-attention map of the source im- age. With this bbox, we perform the background gradient masking (BGM) strategy to prevent modi- fications outside the bbox, thus ensuring background consistency. Moreover, to improve foreground consistency, we leverage the semantic information to highlight the cross-attention maps of source and target concepts respectively within the bbox. This strategy leads to the semantic-enhanced concept representation (SECR), which facilitates precise foreground swapping. Finally, we intro- duce the Step-Skipping Gradient Updating (SSGU) strategy, which only performs forward passes at certain timesteps to calculate gradients. For the timesteps without direct gradient computations, we reuse the previously obtained gradients for updates. Through this strategy, we reduce the total number of forward passes and improve the efficiency of our method.
Since the CCS is a recently proposed task, no dedicated evaluation benchmark currently exists. To address this gap, we introduce ConSwapBench. the first benchmark dataset specifically designed for CCS. ConSwapBench comprises two sub-benchmarks: Conceptßench and SwapBench. Concept- Bench contains images representing target concepts, while SwapBench includes images with one or more concepts to be swapped, serving as source images.
Through extensive qualitative and quantitative comparisons, demonstrate the effectiveness and superiority our INSTANTSWAP. We also conduct comprehensive ablation studies to verify the ef- fectiveness of each component of our approach. Additionally, we further extend our INSTANTSWAL WAP to related tasks, proving its efficacy and versatility. Our contributions are summarized as follows:
We propose INSTANTSWAP, novel training- free customized concept swapping (CCS) framework, which enables efficient concept swapping across sharp shape differences.
We design the background gradient masking (BGM) strategy and ierraantc-enhancee cept representation (SECR) to improve the background and foreground consistency respec- tively. Moreover, we adopt step-skipping gradient updating (SSGU) strategy reduce redundant computation and improve efficiency.
To provide a comprehensive evaluation for CCS, we introduce ConSwapBench, the first benchmark for customized concept swapping. Extensive qualitative and quantitative eval- uations demonstrate the effectiveness and superiority of our INSTANTSWAP.
2 RELATED WORK
2.1
DIFFUSION-BASER BBSED IMAGE EDITING
Image Editing is a fundamental and popular topic in computer vision. Previous works based on Gen- erative Adversarial Networks (GAN) (Goodfellow et al. 2020 Li al. 2020 He al. 2023) only
2

---

Published as a conference paper at ICLR 2025
focus specific object domains (Tang et al. 2023a b: Fang et al. 2024; He et al. 2024a c b Zhong
eet al. 2024ble c a). which limits the application. With the emergence of diffusion model (Rombach et al. 2022p, imgge editing is now able to modify various objects through prompts. These methods
are mainly divided into five categories: instruction-based methods, blending- based, attention-based. inversion-based. and score distillation based methods. Instruction-based methods (Brooks et al.
2023 Ma et al., 2024a b 2022, 2023 Geng et al. [024 Huang et al. [022,] Zhang et al. (0224a) typically require an instruction editing dataset train the diffusion model. Blending-based meth- 2022 Li et al. 2024, Zhang et al. 2023 Huang et al. 2023a) merge the source ods (Couairon et al.
and target prompts to guide the editing process, while attention based methods (Hertz et al.) 2022]
Cao et al. 2023 Ma et al., 2024c; TTmanyya et al.] [023 Guo & Lin 2024; Wang et al. 2024a b
inject the attention feature of the source image. Both methods have lower editing costs but poorer background preservation and prompt alignment. Inversion-
methods (Mokady et al. 2023 Ju
itt al., 2024| Miyake et al. 2023 Dong ct al. 2023 Chen et al.) 2024) aim to reverse the fixed tra- jectory generated by the forward pass to reproduce the source image. These methods can serve as an extra training phase to enhance the background consistency of attention-base
methods. Finally, score distillation based methods (Hertz et al.) [2033] Nam et al. 2024 Wang al.] [202a]] |enn et al.)
2024 Xue et al.) 2024 Chang et al.J 2024] Kim et al. (2023) draw on the optimization process of
SDS (Poole al. 2022). using score distillation- bassaaiooo---ddddddee loss to optimize the source image for editing.
These methods are more flexible than the previous ones but still face challenges with background preservation.
2.2 CONCEPT SWAPPING
Concept swapping, a subtask of general image editing, focuses on replacing the source concept in an image with a user-specified target concept. This task is first proposed by PbE (Yang al. 2023)] which employs a CLIP encoder to extract features of the target concept and inject them into the UNet through cross-attention layer. After that, concurrent works (Gu et al.] 2024a Choi et al.; Li et al. 2023b) extend concept swapping into the customization field (Gal et al. 2022 Zhu al. 2024). They combine attention-based editing methods (Hertz et al., [022 Tumanyan et al.
2023
2023 with ouning-based customization methods (Ruiz et al., 2023 Kumari et al. 2023) to achieve
customized concept swapping. Building on Photoswap, SwapAnything (Gu et al., (024b) further obtains masks with external modules to specify the locations of objects in the source image. We improved the existing method in three key aspects. Firstly, we employ bounding boxes instead of masks as spatial indicators of the source concept, allowing greater flexibility for shape variation during concept swapping. Secondly, we use bounding boxes to prevent background changes via gradient masking, thus ensuring background consistency. Third, we augment concept representa- tion with semantic information to maintain foreground consistency. Finally, rather than executing forward passes every timestep, we execute them only at specific intervals enhance efficiency.
METHOD
Given a set of images (lypically fewer than 5) X, = { representing a specific concept O,, along with an image v. and a prompt p. describing a source concept O., the objective of CCS is to "seamlessly" replace O, in v, with O according to a target prompt Pt. resulting in a final target im- age Zt. An ideal customized concept swapping should handle the shape differences between source and target concepts to preserve swapping consistency while maintaining satisfactory efficiency. We introduce INSTANTSWAP to achieve this. INSTANTSWAP is based on Stable Diffusion and extends from the score distillation based image editing methods (Poole et al. 2022
Hertz et al. 2023).
3.1 PRELIMINARIES
3.1.1 STABLE DIFFUSION
In this paper, the foundational model utilized for text-to- t--to-immgee generation is Stable Diffusion (Rom- bach et al. 2022) It takes a text prompt P as input and generates the corresponding image x. Stable Diffusion consists of three main components: an autoencoder(E] (.)), D(-)), a CLIP text encoder r(-)
and U-Net c-). Typically, it is trained with the guidance of the following reconstruction loss:
Cree = Ez,e- [lle - Es (z,, t, (P))
(I)
where e ~ A' (0. 1) is a randomly sampled noise, t denotes the time step. The calculation of 2, is given by 2, = 042 + dre, where the coefficients 0g and o are provided by the noise scheduler.
3

---

Published as a conference paper at ICLR 2025
3.1.2 SCORE DISTILLATION BASED IMAGE EDITING
Different from traditional attention- image editing, score distillation based methods achieve
image editing through iterative optimization with score distillation loss. Given the latent feature
2 of source image and a denoising U-Net (.))), SDS (Poole et al. 2022) can optimize the latent
feature of the image to align with the target prompt Pa by employing the following loss:
LSDS = I (2,, t, r (Pt) - <l12:
where and randomly sampled noise and timestep.
The resulting image SDS is very blurry and only contains foreground objects in the target prompt P,. To address this issue, DDS (Hertz et al., (023) expresses the gradient of Eq. (2
V VgLsDs(Ee, t, T(P)) = óegt + óbias,
(3)
where digt indicates the direction aligned with the target prompt and óviss refers to undesired part that makes the image blurry. Based on this, DDS further utilizes the fixed latent & of the source image and the source prompt Ps to approximate the bias component in Eq. (3):
V VgLsDs(E & t, (P..) Rs dbias R3 òbias.
(4)
Finally, DDS is represented by the difference of Eq. and Eq. 4
V.LoDS = V,LsDs( (2,, t, ((P)) - V (E, = digt.
(5)
Based on Eq. 5 the loss of DDS is given by
LDDS = |Ea (2,, T (Pi)) - és t,
(6)
3.2 INSTANTSWAP
Directly extending score distillation based editing methods to the task of CCS encounters the chal- lenge of inconsistency. These methods optimize the background and foreground simultaneously, causing cross-interference and leading to undesirable inconsistency. To address these limitations, we first propose a strategy to automatically locate objects to be edited, resulting in the object bounding box (bbox). With this bbox, we propose a background gradient masking technique to remove gra- dients in the background region and confine swapping to the foreground region. To further enhance foreground swapping consistency, we propose to learn semantic-enhanced concept representations for both source and target concepts based on an attention map feature injection mechanism. An overview of our method is presented in Fig. 3
3.2. AUTOMATIC BOUNDING BOX GENERATION
We first automatically obtain the bbox to indicate the position of the concept O, in the source image. Given the source image I, and the source prompt P.. we perform a forward pass with the U-Net (.).) and obtain the cross-attention map A and self-attention map A through:
QKT
A = Softmax
V
(7)
VJ
where Q is the query vector projected from the image features, d' represents the output dimension of key and query features. K is the key vector and V is the value vector. For cross-attention maps AS, K and V are projected from the text embeddings T(P). For self-attention maps A', K, and V are projected from the image features. Directly applying a threshold on the A can yield coarse- grained mask, which cannot accurately reflect the location of O.. Inspired by (Nguyen et al. 2024
Zhang et al. 2024bb Tang al. 2024), we modify the A follows:
A =
(A
(8)
Based on Eq. (7). all values in A range between 0 and 1. Therefore, element- exponentiation of A by a can weaken the activation of non-target regions. Additionally, as mentioned in (Liu al. 2024 A contains rich structural information. This information can effectively assist A in better activating the target regions. Finally, we apply the threshold 3 to A to obtain the mask. Subse- quently, we converted the mask into the bbox B, based on the minimum and maximum coordinates of all foreground points within the mask. This strategy allows us to obtain the bbox B, without any additional modules. We intentionally set a relatively loose constraint on the mask to obtain bbox that completely covers the source concept. We discuss the effectiveness our automatically obtained bboxes in Sec. 4.5
4

---

Published as a conference paper at ICLR 2025
Source Branch
"A saza en the tasle"
rodic
U-Nett
Scurce Image
Book Greation
V.C 3 &
(6 (.) BB,
SECR
Torget Branch
500R
V.Cacv
old NodS
U-Net
Optimized
Reazot
ake TREEET on ThE Toble"
Figure 3: Overall pipeline of INSTANTSWAP. We first obtain the bhox of the source concept auto- matically. The obtained bbox is input into SECR in both the source and target branches to enhance the foreground swapping consistency. Additionally, the source and target branches generate the pre- diction of noise for the source and target images based their respective prompts. The predicted noise, along with the bbox, is used for the BGM to preserve background consistency.
3.2.2 BACKGROUND GRADIENT MASKING
With the object bbox, propose background gradient masking (BGM) approach to ensure that the concept swap is confined to the foreground region. Given the latent feature & of the source image and the latent feature z of the target image, where initialized é and is continuously optimized to obtain the final target image Tt. Based on Eq. (6) we first obtain the gradient of z:
7.C = (co (2,, t, (P:) 2s (&,, T(P)) Oc (2,,t, T (P:)) 0z:
99)
Oz
02
2022 the mid term is U-Net Jacobian term and be omitted, and Oe = 0z, /02 is a constant which can be represented as w(t):
As stated in (Poole et al.
V.C (E) (z,, (Pt)) (,,
(10)
This gradient shares the same dimension as which means can update pixel-wise manner. However, this will update the foreground and background simultaneously, producing inconsistent background. To remedy this, we apply the bbox B, on Eq. (TO) mask the gradients related the background before back propagation and obtain our BGM:
V.LEGM = ((t)(Es (2e, T (Pt)) - E ((P)) B.,
(11)
This simple masking strategy prevents the background from being updated and thus ensures back- ground consistency.
3.2.3 SEMANTIC- ENHANCED CONCEPT REPRESENTATION
The BGM module maintains the background consistency during swapping. However, whether the source concept can be replaced with the target concept cannot be guaranteed. This limitation arises because the optimization of Eq. (1)) is still carried at the entire feature map level of both the source latent E, and target latent without distinguishing between the foreground and the back- ground. To address this, we propose to obtain semantic-enhanced concept representations for both source and target concepts and emphasize their locations within the foreground region during con- cept swapping.
Let F, be the source image feature and P. represent the prompt of source concept (e.g. "rose"), the semantic embedding c, can be acquired through c, r(p.). We first resize the previously obtained object bbox to fit the dimensions of the source image feature F., resulting in the feature bbox By- We then crop F. with B, to get a regional image feature f.. With fs, we calculate the query vector through Q. W fs. After that, can obtain the key and value vectors through:
K, W Cs. V, W C..
(12)
5

---

Published as a conference paper at ICLR 2025
Image Feature
Key
Concept Prompt
Value
Attn Output
Query
Feature Output
Grap
Attention
Figure 4: Overview of SECR.
Then the final partial attention output is calculated as follows:
(ook3
= Softmax
Vs.
(13)
Vdl
where d' represents the output dimension of key and query features. In this way, we inject the se- mantic information of the source concept into the cross-attention map. resulting in regional concept representation Sa. We then map S. back to the original feature map F, to get a semantic-enhanced representation F. for the entire source image. Fig. illustrates the process. In the target branch, first convert the target concept into semantic space with DreamBooth (see more details in Ap- pendix N). using specific rare token (e.g.. "sks") to represent the concept. With the target prompt Pe (e.g., sss teapot") and the feature bbox B,, similarly apply this process for the target image feature and obtain the semantic-enhanced representation F. for the target image.
Through proactive injection of semantic guidance, we provide the source and target branches with semantic-enhanced concept representation within the foreground region. Consequently, SECR trans- forms the target branch into target concept adder and the source branch into a source concept re- mover. Their collaboration results in precise and seamless concept swapping. thus enhancing the foreground consistency. Moreover, SECR can also facilitate concept insertion and removal, which further discussed in Sec. 4.6
neuse
9
9--1
9r-2
g--2
9+
(a) Previous Methods
(b) Proposed S56U
Figure 5: Comparison between our SSGU and previous methods.
3.2.4 STEP-SKIPPING GRADIENT UPDATING
After addressing the problem of inconsistency, we turn our attention to the challenge of inefficiency. As illustrated in Fig. [ previous methods calculate the gradient at each timestep. However, the suc- cess of DDIM (Song et al. 2020) in accelerating DDPM (Ho et al.] 2020) motivates us to consider: Can skip the calculation of gradients at certain timesteps? During concept swapping, ob- serve that the effect of gradient updates on the target image is similar across adjacent timesteps (see detailed results in Appendix (T). Based this observation, we propose the Step-Skipping Gradient Updating (SSGU) strategy. The key insight of SSGU is that skipping some gradient calculations does significantly sacrifice the swapping consistency while considerably improving efficiency. As result, our SSGU calculates gradients at interval timesteps and reuses the previously calculated gradients during the intervening timesteps.
We define our entire pipeline as F, given the source image T., the timestep f, and the intermediate target image é; at timestep We can obtain the gradient Se and the output intermediate target image -t-1 at timestep t as follows:
9: = F(x,, Zz, t),
(14)
.1+1 3, 19tt.
(15)
where 17 is the learning rate. Our SSGU periodically retains some anchor gradients and skips the forward passes between two anchor gradients. The step-skipping period is controlled by the SSGU factor X. The set of anchor gradients can be defined as:
G (gxx) k 0, [T/A],
(16)

---

Published as a conference paper at ICLR 2025
Source Image
Customized Concapt
3b5
bas
PhotoGwap
PePIn-R2
Oura
blue bird sits
gks condle on a branch
king leaf covered ground duck walking leaf covered ground
hird sitting with trees the background cke Flower sithing wire with trees the background
bird stonding rock near the ocean gls sunglosses standing on top of rock near the ocean
a Small boat. floating on top of sondy beach slss bort floating on top of a sandy beach
shesp standing on top of a lush green field sks teapot standing on top of lush green field
Figure 6: Qualitative comparisons between INSTANTSWAP and other methods. More qualitative
results as well as the used bboxes can be found in Appendix Q
where T is the ended timestep. For any intervening timestep t, gradient to update the intermediate target image ăt. Taking À = 2 as an example, we assume that an even number and -t-2: 9t G. SSGU updates T;-2 and ž with the anchor gradient 9r-2
its nearest former anchor
Ot-2 F(xs, f2-2, 2),
(17)
3 = 8.-2 119t-2,
(18)
_.-1 119t-2-
(19)
For the next timestep another anchor gradient 9r is used update zt. As result, our SSGU reduces the number of forward passes during the entire concept swapping process to 1/A of the original count. Since the forward pass accounts for approximately 95% of the total inference time (see detailed analysis in Appendix [E). our SSGU can improve the overall inference speed of our method by approximately Furthermore, our SSGU can be transferred to other score distillation based methods to improve their efficiency in the same way, which is further discussed Sec. 4.6
times, with minimal effect on swapping consistency (see Sec. 4.5)
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We conduct the experiments with Stable Diffusion (Rombach al. 2022) RTX3090. We use the customized checkpoint from DreamBooth (Ruiz et al. 2023) to introduce concepts. We the SSGU factor À to 5, a obtained through the first three steps. Subsequently, we use SGD (Robbins & Monro [951 with learning rate of 0.1 to optimize for 550 steps of iterations.
v2.1 1-baee single
0.5 and the guidance scale 7.5. The bbox is

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparisons. Our method outperforms all the compared methods in all the
selected metrics. Red stands for the best result, Blue stands for the second best result.
FG
BG
Overall
Method
CLIP-I
PSNR +
LPIPS &
MSE x10
SSIM x
CLIP-T
Time (s)
SDS
73.70
20.79
339.51
107.53
72.59
23.53
40.37
DDS
71.05
24.07
89.80
53.08
83.44
23.99
66.89
CDS
71.69
23.36
90.35
63.41
83.21
24.17
140.26
PhotoSwap
70.15
24.24
120.62
56.64
80.56
22.38
140.34
PnPInv+P2P
70.74
24.63
108.22
47.49
82.07
24.25
37.02
Ours
75.00
27.39
47.68
27.87
86.58
25.74
19.83
4.2
CONSWAPBENCH
Despite the significant application potential of customized concept swapping, there is currently no dedicated evaluation benchmark. To meet the needs of comprehensive evaluation, we introduce ConSwapBench, the first benchmark dataset specifically designed for customized concept swapping. ConSwapBench consists of two uub-benchmarks: ConceptBench and SwapBench. ConceptBench comprises 62 images covering 10 different target concepts used for customization, while SwapBench includes 160 real images containing one or more objects to be swapped, serving as source images. For each image in SwapBench, we use Grounding SAM (Ren et al. 2024) to acquire the bbox of the foreground concepts as the ground truth for evaluation purposes. We apply each customized concept from ConceptBench perform concept swaps each image in SwapBench, ultimately generating a total of 1,600 images for evaluation. More details can be found in Appendix
4.3 QUALITATIVE COMPARISON
Since customized concept swapping is a relatively novel task, there are limited methods available for direct comparison. Consequently, we include SOTA image editing methods and adapt them for cus- tomized concept swapping. We include the following methods: (1) Score distillation based methods:
SDS (Poole al. 2022). DDS (Hertz al. 2023), and CDS (Nam al.. (002);; (2) Attention -based methods: PhotoS wap (Gu et al. 2024a). PnPInv (Ju et al. 2024), and P2P (Hertz et al. (022). We excluded Snaphnything (Gu et al. 2024b) as it not publicly available. The qualitative results are accommodate shape varia-
illustrated in Fig. 6 We find that score distillation based methods tions during concept swapping. However, they exhibit poor foreground fidelity (3rd and 6th rows) and lead to unnecessary modifications the background (1st and 2nd rows). Attention- meth- ods are unable to manage shape variations (4th row) and also struggle with maintaining background consistency (5th row). In contrast, our method demonstrates superior performance in addressing shape variations and maintaining swapping consistency.
4.4 QUANTITATIVE COMPARISON
We also conduct a thorough quantitative comparison on ConSwapBench. For each generated image, we first use the ground truth bbox SwapBench to obtain their foreground and background respec- tively. We use seven different metrics to evaluate the methods from three aspects: (1) Foreground consistency: We calculate the CLIP Image Score (Radford et al. (021)) between the foreground of generated images and the images of customized concepts. (2) Background consistency: We use the four metrics, PSNR, LPIPS (Zhang et al. 2018), MSE, SSIM (Wang et al., 2004) to evaluate the background consistency.
(3) Overall consistency and efficiency: We calculate the CLIP Text Score (Hessel et al., 2021) between generated images and target prompts to evaluate the overall prompt consistency. We also report the inference time of each method evaluate their efficiency. our method outperforms other methods on all seven metrics.
As shown in Tab. [
4.5
ABLATION STUDY
BGM. To verify the effectiveness of BGM in background preservation, we conduct an abla- tion study by removing BGM. As illustrated in
Table 2: Quantitative ablation results of BGM.
Method
|SNNR I.PPS 4 MSE J. SSIM CLIP-T
the second column of Fig. 7 while our method still achieve concept swapping without
BGM
18.03
249.24
184.19
72.25
23.08
Ours
27.39
47.68
27.87
86,58
25.74
BGM, it causes serious modifications on the background. contrast, our full method not only maintains high foreground fidelity but also effectively preserves the background consistency. We
8

---

Published as a conference paper at ICLR 2025
Amage
atunde
Larget
largel&asurce
Sourde Image
hardwced Fleers and with hordweed Flcere dack
Gastmmeod Cancept
lesslt
lesul: o/ wrs
eddle lopped with stfffe candla toppeci www shaiehals
Asstttn frome neckeg
Figure 7: Qualitative results of the ablation study on: Left: BGM and SECR. Right: different bboxes.
further conduct quantitative analysis of the background consistency and prompt consistency, as shown in Tab. 2 Our full method outperforms in all metrics.
Automatic bounding box detection
Table 3: Quantitative ablation results of different bboxes.
mechanism. We further verify the effectiveness of our bboxes by ing ground truth (GT) bboxes from SwapBench to replace the automati- cally obtained bboxes. As shown in Fig. 7 although GT bboxes accurately indicate the location of the method from fully swapping the source concept. Compared to GT bboxes, our bboxes are relatively larger and can fully cover the source concept, thereby facilitating complete concept swapping. We also provide quantitative comparisons of different bboxes. As shown in Tab. 3 Gen stands for the generation bboxes, while Eva stands for the evaluation bboxes. The two types of bboxes do not significantly affect background preservation in our method, whereas our bboxes perform better than GT bboxes on the foreground metric. More detailed comparisons can be found in Appendix
Gen Eva CLIP-I PSNR LPIPS
MSE & SSIM CLIP-T
Ours GT 75.00
27.39
47.68
27.87
86.58
25.74
GT GT
75.79
31.46
32.61
13.51
87.62
25.53
Ours Ours
77.72
31.64
31.17
13.28
88.21
25.74
concept, they prevent our
SECR. To verify the effectiveness of SECR, we conduct abla- tion studies including removing SECR from (1) source branch (w/o source), (2) target branch (w/o target), (3) both (w/o source & target). The visualization results are illustrated in columns to of Fig. 7 Although all methods preserve the background well, they show reduced foreground fidelity. Additionally, we perform a quantitative analysis of their fore- ground consistency and prompt consistency. The results pre- sented in Tab. jindicate that our full method exhibits superior performance.
Table 4: Quantitative ablation re- sults of SECR.
Method
CLIP-I CLIP-T 1
w/o source
73.70
25.47
w/o target
73.40
25.52
w/o souree&ttrgee
72.42
25.21
Ours
75.00
25.74
SSGU. To verify the effectiveness of our proposed SSGU, first visualize the images generated under different SSGU factors. As shown in Fig. 9 A = indicates that SSGU is used. When À S 9, the SSGU can preserve foreground and background consistency well while improving the effi ciency of our method. As increases,
80
100
CLP-1
Time
80
72
60
images exhibit more artifacts due to excessive neglect of gradients. There- fore, identifying an optimal SSGU factor À is crucial. We further conduct detailed quantitative analysis of different values on foreground consistency and efficiency (see com- plete results in Appendix D). illustrated in Fig. & where the x-axis represents different À values and the y-axis rep- resents the respective metric outcomes. When SSGU not used, our method achieves the best swapping consistency but the lowest efficiency. As À increases, our SSGU sacrifices certain swapping consistency but significantly improves efficiency. To balance consistency and efficiency, we ultimately select À = 5 for our final model.
68
60
25 Number of SSGU factor
30
Figure 8: Quantitative results the ablation study of the SSGU factor X.
4.6 APPLICATIONS OF INSTANTSWAP
Multi-concept swapping. Our INSTANTSWA. WAP be easily extended to facilitate multi-concept swaps by sequentially performing multiple single-concept swaps. As shown in the left of Fig. [00 our method can swap each concept within the image with both foreground and background consistency.

---

Published as a conference paper at ICLR 2025
Source Irage
Gustomized Concept
A=1
A=5
X=
A=15
A
A= 30
couple of birds standing Top of rocky beach duck stonding on top of rocky beach
photo of cake on weoden plate photo of sks on wooden plate
Figure 9: Qualitative results of different SSGU factor A. Excessively high À can lead decline in foreground consistency. Searbo
Tstorradieto lesult
Firal Result
Gust paized
Source
Figure 10: INSTANTSWAP can be extended to other tasks such Left: Multi-Concept Swapping. Middle: Human Face Swapping. Right: Concept Insertion and Removal.
Human face swapping. INSTANTSWAF demonstrates exceptional capabilities in human face swap- ping. As shown in the middle of Fig. I0 with customized face models from CivitAI (Civitai 2024). users can seamlessly replace the face in a source image with customized target face.
Concept insertion and removal. In addition to concept swapping, cept insertion and removal. For concept insertion, we employ the same procedure as concept swap- ping. For concept removal, we adjust the target prompt and the target semantic input pt of SECR to a null prompt. The results in the right of Fig. 10 further demonstrate the versatility of our method.
method also supports
Accelerating other methods. SSGU transferred to other score distillation based methods to enhance their efficiency. We select three representative methods: SDS (Poole et al.] 2022). DDS (Hertz al.) 2023) for image edit- ing, and CoSD Kim et al. (2023) for video edit- ing. As shown in Fig. I1 combining these methods with SSGU can significantly improve their efficiency while almost not altering the generation quality. We further conduct quantitative analy- sis to assess the transferability
be
Table 5: Quantitative results of SSGU extension.
Method w/o SSGU w/ SSGU À 3 w/ SSGU
SDS DDS CoSD
40.37s 66.89
14.12s
8.62s
22.65s
13.90s
344.76s
128.97s
79.15s
the proposed SSGU, as presented Tab,
50G 306-U "ii Ecsitio
500 5D6-53BU
Source Image
Vanillo St6
-Sa Easter
Source Video
Vanillo Co
DOS-506U "Ex toster
005-5339 <5x taster
Source Tmage
Va bos
COSD-ESEU *3k Eascer
COSD+SSRU *SN fester
photo Tiger
Change
yellow
Figure 11: Qualitative results of extending our SSGU to other score distillation based methods.
5 CONCLUSION
This paper introduces INSTANTSWAP, novel framework for precise and efficient customized con- cept swap. Our BGM and SECR collaborate to maintain both background and foreground consis- tency. Furthermore, ciency. Finally, we introduce ConSwapBench, a comprehensive benchmark dataset for customized concept swapping. The impressive performance of INSTANTSWAP demonstrates its effectiveness.
propose the SSGU to eliminate redundant computation and improve effi-
Acknowledgment. This work was supported by the STI 2030-Major Projects under Grant 2021Z.D02144
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael S Albergo and Eric Vanden-Fijnden. Building normalizing flows with stochastic inter- polants. arXiv preprint arXiv:220./5557/ 2022.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: funing-frre mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22560-22570 2023.
Hangeol Chang, Jinho Chang, and Jong Chul Ye. Ground-a-score: Scaling up the score distillation for multi-attribute editing. arXiv preprint arXiv:2403.13551, 2024.
Qihua Chen, Yue Ma. Hongfa Wang. Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang. Shaobo Min, Qifeng Chen, and Wei Liu. Follow-your-canvas: Higher-resolution video outpainting with extensive content generation. arXiv preprint arXiv:2409. 00009, 2024.
Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-edit: Text-guided image editing with customized diffusion models. arXiv preprint arXiv:2305.15779 2023.
Civitai. Civitai.
come///aacmmm// 2024.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11227, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 88944, 2021.
Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text-driven image editing using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7430-7440, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Chengyu Fang, Chunming He, Fengyang Xiao, Yulun Zhang, Longxiang Tang, Yuelin Zhang, Kai Li, and Xiu Li. Real-world image dehazing with coherence-based pseudo labeling and cooperative unfolding network. In The eighth- Annual Conference on Neural Information Processing Systems, 2024.
pwt. wert:/ffummmiccm wwwerrmmmii---611 2 IIt
Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Diffusion transformer for image editing. arXiv preprint urXiv:2411.00 1.00226, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.016/1 2022.
Zigang Geng, Binxin Yang. Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang. Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, al. Instructdiffusson: generalist modeling interface for vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12709-12720, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-11 2020.
Jing Gu, Yilin Wang. Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al. Photoswap: Personalized subject swapping in images. Advances in Neural Information Processing Systems, 36, 2024a.
11

---

Published as a conference paper at ICLR 2025
Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, and Xin Eric Wang. Swapanything: Enabling arbitrary object swapping in personalized visual editing. arXiv preprint arXiv:2404.05717, 2024b.
Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi inltiiinsttucttinn image editing by attention modulation. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6986-6999 6996, 2024.
Chumming Hc, Kai Li, Yachao Zhang, Yulun Zhang, Chenyu You, Zhenhua Guo. Xiu Li, Martin Danelljan, and Fisher Yu. Strategic preys make acute predators: Enhancing camoullaged ob- ject detectors by generating camouflaged objects. In The Twelfth International Conference on Learning Representations.
Chunming He, Kai Li, Guoxia Xu, Jiangpeng Yan, Longxiang Tang, Yulun Zhang, Yaowei Wang, and Xiu Li. Hqg-net: Unpaired medical image enhancement with high-quality guidance. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Chunming He, Chengyu Fang, Yulun Zhang, Tian Ye, Kai Li, Longxiang Tang, Zhenhua Guo, Xiu Li, and Sina Farsiu. Reti-diff: Illumination degradation image restoration with retinex-based latent diffusion model, 2024a. URL httthtts::/ htttpp::/aaxii.orrtr www.ooggg/b////ooooooooooo..eere aaaaabs//22 2211 138
Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li. Weakly-supervised concealed object segmentation with sam and multi-scal feature grouping. Advances in Neural Information Processing Systems, 36, 2024b.
sam-based based pseudo labeling
Chumming He. Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang, Yulun Zhang, Wang- meng Zuo, Zhenhua Guo, and Xiu Li. Diffusion models low-level vision: A survey, 2024c. URL https: /ttts::t::///// /ttps/////aivv.i/a
org/ wbwp///................... 11138
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. coompt-to-proppe image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprini arXiv:2104.08718 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Wenjing Huang, Shikui Tu, and Lei Xu. Pfb-diff: Progressive feature blending diffusion for text- driven image editing. arXiv preprint arXiv:2306. 16894, 2023a.
Yukun Huang. Jianan Wang, Yukai Shi. Boshi Tang. Xianbiao Qi, and Lei Zhang. Dreamtime: An improved optimization strategy for diffusion-guidee 3d generation. In The Twelfth International Conference on Learning Representations, 2023b.
Yuzhou Huang, Liangbin Xie, Xintao Wang. Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-basse image editing with multimodal large language models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8362-8371, 2024.
Xuan Ju, Ailing Zeng. Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pup inversion: Boosting diffusion-based editing with 3 lines of code. In The Twelfth International Conference on Learning Representations, 2024.
Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Col- laborative score distillation for consistent visual synthesis. arXiv preprint arXiv:2307.0478: 2023.
12

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1944 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-tra pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 PMLR, 2023a.
Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu. Adversarial feature hallucination networks for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13470-13479, 2020.
Pengzhi Li, Qiang Nie, Ying Chen, Xi Jiang, Kai Wu, Yuhuan Lin, Yong Liu, Jinlong Peng, Chengjie Wang, and Feng Zheng. Tuning-free image customization with image and text guidance. arXiv preprint arXiv:2403. 120.. 2024.
/2658,
Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing, arXiv preprint arXiv:2306. 220..224....1244, 2023b.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamn Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7817-7826, 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2022.
Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao. Visual knowledge graph for human action reasoning in videos. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4132-4141, 2022.
Yue Ma, Xiaodong Cun, Yingqing He, Chenyang Qi, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen, Magicstick: Controllable video editing via control handle transformations. arXiv preprint 1rXiv:2312.03047, 2023.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: fose-guided text-to-video generation using pose-free videos. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4117 2024a.
Yue Ma, Yingging He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Beeug--Yenns Shum, Wei Liu, al. Follow-your-clies Open-domain regional image anima- tion via short prompts. arXiv preprint arXiv:2403. 208.00888, 2024b.
Yue Ma, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Y Yeung Shum, Wei Liu, et al. Follow-your-emoy Fine-controllable and expressive freestyle portrait animation. arXiv preprint urXiv:2406. 000000000000... 2024c.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Regative-prompe inversion: Fast image inversion for editing with text eext-guided diffusion models. arXiv preprint arXiv:2305.1688 16807, 2023.
13

---

Published as a conference paper at ICLR 2025
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cr. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6038-6047) 2023.
Hyelin Nam, Gihyun Kwon, Geon Yeong Park, and Jong Chul Ye. Contrastive denoising score for text-guided latent diffusion image editing, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9192-9201, 2024.
Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based syn- thetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems, 36, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 /2. 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Yuang Peng. Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tat Xia. Dreambench++:: A numann-liigned benchmark for personalized image generation. arXiv preprint arXiv:2406. 16855, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 9748-8763. PMLR, 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEF/CVF conference on computer vision and paltern recognition, pp. 22500- 22510, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02520 2020.
Zhihao Sun, Haipeng Fang, Juan Cao, Xinying Zhao, and Danding Wang. Rethinking image editing detection in the era of generative ai revolution. In ACM Multimedia 2024, 2023.
Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, and Xiu Li. Consistency regularization for generalizable source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4323-4333, 2023a.
Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, and Xiu Li. Source-free domain adaptive fundus image segmentation with cllasebalanced mean teacher. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pp. 684-694, Springer, 2023b.
Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li. and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. arXiv preprint arXiv:2407.05342 2024.
14

---

Published as a conference paper at ICLR 2025
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to to-image translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Unsplash. Unsplash. https://ansplasthn com/ 2024.
Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, and Xiu Li. Cove: Unleashing the diffusion feature correspondence for consistent video editing. arXiv preprint arXiv:2406.0885g) 2024a.
Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing. arXiv preprint 1rX1v:241..04746, 2024b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Jingyun Xue, Hongfa Wang, Qi Tian, Yue Ma, Andong Wang. Zhiyuan Zhao, Shaobo Min, Wenzhe Zhao, Kaihao Zhang, Heung- Yeung Shum, et al. Follow-your-pose v2: Multiple-condition guided character image animation for stable pose control. arXiv preprint wxXiv2446.030033 2024.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-base image editing with diffusion models. In Pro- ceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 18391, 2023.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guidee image editing. Advances in Neural Information Processing Systems, 36, 2024a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-59 595, 2018.
Shiwen Zhang, Shuai Xiao, and Weilin Huang. Forgedit: Text guided image editing via learning and forgetting, arXiv preprint arXiv:2309. 2023.
Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang. Attention calibration for disentangled text-to-image personalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4764-4774, 2024b.
Xinhao Zhong, Bin Chen, Hao Fang, Xulin Gu. Shu-Tao Xia, and En-Hui Yang. Going beyond fea- ture similarity: Effective dataset distillation based on class- conditional mutual information. arXiv preprint arXiv:24/2 http::2//2.09944444 2024a.
Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Tao Dai, Meikang Qiu, and Shu-Tao Xia. Hicrar- chical features matter: A deep exploration of gan priors for improved dataset distillation. arXiv preprint arXiv:2406.057700 057044 2024b.
Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Jianlong Wu, and Bin Chen. Efficient dataset distillation via diffusion-driven patch selection for improved general- ization. arXiv preprint arXiv:2412.099533 2024c.
Chenyang Zhu, Kai Li, Yue Ma, Chunming He, and Li Xiu. Multibooth: Towards generating all your concepts in an image from text. arXiv preprint arXiv:2404. 14239, 2024.
15

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY STATEMENT
To ensure the reproducibility and comprehensiveness of our method, the Appendix includes the fol- lowing sections: Appendix B offers detailed explanation of our method, along with the source code and parameter settings for each comparative method. Appendix [C reviews relevant bench- marks from previous works and provides detailed information and visual examples of the two sub- benchmarks in ConSwapBench Bench. Appendix D provides complete quantitative results of the ablation study on SSGU. Appendix F analyzes the time of each component in method during the infer- ence process. Appendix Fa compares our automatically obtained bounding boxes with the ground truth bounding boxes. Appendix [ adapts our method to other Stable Diffusion models. Ap- pendix HH] explores more challenging concepts for concept swapping. Appendix swapping process of our method. Appendix includes the ablation study of automatic bbox gen- eration. Appendix K discusses the impact of the number of target images.Appendix, [ shows the time cost of the customization process.Appendee tual Inversion (Gal et al., 2022) for customization. Appendix Nexplains the detailed usage of the customization methods.Appendix [ analyses how our SECR can be applied to DiT-based archi- tecture. Appendix P discusses the impact of our method on the naturalness of generated images. A presents the bounding boxes used in the main paper and includes a gallery of additional qualitative results. Appendix |
[ visualizes the
providess comprehensive results of using Tex-
discusses the limitations of method and future works. In Ap- pendix [S] we discuss the potential social impact brought by our method. The code is available in https: //t::///ttthbb com/ chenyangzhul//////o/a InstantSway
B DETAILED IMPLEMENTATIONS
In this section, provide experimental details of our method as well all the comparison methods.
SDS (Poole et al. 2022). We utilized a third-party implementation"] for SDS, adhering to its recom- mended configurations: iteration number of 200, guidance scale of 7.5, and a learning rate of 0.1 with SGD (Robbins & Monro 1951).
DDS (Hertz et al. 2023). We used the official implementation for DDS. We followed its recom- mended settings, setting the iteration to 200, the guidance scale to 7.5, and using SGD (Robbins & Monro 1951) with learning rate of 0.1.
CDS Nam et al. 2024). We utilized the official implementation for CDS, adhering to its recom- mended settings: iteration number of 400, guidance scale of 7.5, wnon of SGD (Robbins) & Monro 1951) is employed with a learning rate of 0.1.
PhotoSwap (Gu et al. 2024a). We utilized the official implementation of PhotoSwap. adhering to its recommended settings. Specifically, we employed the DDIM (Song et al. 2020 sampling method with 50 denoising steps and classifier-free guidance scale of 7.5. The default step for cross-attention map replacement is set at 20, while the default steps for self-attention map and feature replacements are 25 and 10, respectively.
PnPInversion (Ju et al. 2024). We utilized the official implementation of PnPInversion, adhering to its recommended settings: a step number of 50, a reverse guidance scale of and forward guidance scale of 0.
Ours. We set the SSGU factor 5, to 2 and 0.5. We use the first three steps to obtain the bbox and then use SGD (Robbins & Monro, 1951) with a learning rate of 0.1 to optimize for 550 steps of iteration. Besides, we follow Huang et al. (2023b) to adopt a non-increasing timestep strategy. To ensure a fair comparison, we set the iteration steps for SDS, DDS, CDS, and our method to the same value when evaluating inference time.
http:///wkkkmoooooooooooooooyomm hhtpp:///ithuu.cco/// httpa//ww....cccccoooocooootattn zerosott.py
16

---

Published as a conference paper at ICLR 2025
C MORE DETAILS OF CONSWAPBENCH
Customized concept swapping relatively new task, lacking a mature and comprehensive bench- mark for evaluation. Previous approaches have introduced separate benchmarks for image cus- lomization and image editing tasks. For image customization, DreamBooth (Ruiz et al. 2023) in- troduces DreamBench, which includes 30 subjects such as sunglasses, backpacks, dogs, and cats. Custom Diffusion Kumari et al. (2023) proposes CustomConcept100. comprising 101 concepts, in- cluding toys, pets, landscapes, and faces. For image editing, PnP (Tumanyan et al.) 2023) first introduces benchmark with 55 image-prompt pairs. PnPInversion (Ju et al. (2024) proposes PIE- Bench, consisting of 700 images, each with five annotations: source prompt, target prompt, editing type, editing content, and corresponding editing mask. Although PIE-Bench covers various editing types, fewer than 80 images are designed for concept swapping, which is insufficient for comprehen- sive evaluation. To systematically assess our proposed method, we construct a benchmark named ConSwapBench, consisting of ConceptBench and SwapBench.
C.I CONCEPTBENCH
ConceptBench consists of 62 images representing 10 different concepts from the DreamBench (Ruiz jt al. 2023) and CustomConcept110) (Kumari et al. 2023). These concepts include barn, candle, cat, cup, dog, duck, flower, sunglasses, teapot, and vase. We use SAM (Kirillov et al.) 2023) to
segment the subjects in ConceptBench, ensuring that each target concept is accurately translated
into semantic space customized concepts. We visualize our ConceptBench in Fig. 12
Figure 12: Visualization of ConceptBench.
C.2 SWAPBENCH
SwapBench comprises 160 images obtained from Unsplash (Unsplash) 2024 and other related benchmarks including PlE-Bench (Ju al. (004) and Visual Genome (Krishna et al. 2017). Each image contains one or more objects, serving as the source concept for concept swap. For images with captions, the caption is used as the source prompt. For images without captions, we employ BLIP2 (Li et al., 2023a) to generate a caption. We also use Grounding SAM (Ren et al., 2024) to acquire the bbox of the foreground concepts as the ground truth for evaluation purposes. During the evaluation phase, each concept from ConceptBench is applied each image in SwapBench for concept swapping, generating 1,600 images for evaluation. We visualize our SwapBench in Fig. 13
D COMPLETE QUANTITATIVE RESULTS OF THE ABLATION STUDY ON SSGU
We provide detailed quantitative results of different À values across all the metrics. As shown in Fig. 14 overall, as À increases, the consistency of our method decreases while its efficiency increases.
E TIME ANALYSIS OF DIFFERENT INFERENCE COMPONENTS
The inference process of method can be divided into three main components: forward pass, backward pass, and other processes. We illustrate the time allocation for these components in our method with the SSGU factor set to 1. As shown in Fig. [5] the forward pass accounts for ap- proximately 95% of the total inference time. Based on this, our SSGU significantly improves the efficiency of method by eliminating redundant forward passes.
17

---

Published as a conference paper at ICLR 2025
Source Image
bbox
Source Image
Image
Image
A antelops walking on road
A photo of vase on table near window
photo of a basket of books and cup
A bird stonding in the grads
some trees
photo lamp
person holding in front of the ocean
white water lily floating on Top of body of water
gronoe cat sitting sidewalk in front of a building
Figure 13: Visualization of SwapBench.
CLIP-1
PSNR
LPIPS 4
27.5
27.0
72.0
26.5
26.0
59.0
25.5
52.0
45.0
MSE
SSIM
CLIP-T
Time
47.0
87.0
43.0
86.5
25.5
25.0
30.0
86.0
35.0
31.0
23.5
84.5
23.0
10
15
20
25
30
15
20
25
30
Figure 14: Complete quantitative results the ablation study of the SSGU factor X.
COMPARISON OF DIFFERENT BOUNDING BOXES
In Fig. 16 we show our automatically obtained bounding boxes and the ground truth bounding boxes obtained through Grounding SAM (Ren et al. (2024). Compared to the ground truth bounding boxes, our bounding boxes are larger and can fully cover the source concept. This allows our method to completely swap the source concept with the target concept.
G ADAPT INSTANTSWAP TO OTHER STABLE DIFFUSION MODELS
We combine method with Stable Diffusion v1.5 and conduct comprehensive experiments. We present the quantitative results Tab. 6 (the inference time metric is obtained on a single A100).
Total
Other
Backward
Forward
20
40
60
80
100
Time(s)
Figure 15: Time allocation of different components during inference.
18

---

Published as a conference paper at ICLR 2025
Source Image
Our bbox
6T bbox
Source Inage
Our bbox
6T bbox
Figure 16: Comparison of the ground truth bounding boxes and automatically obtained bounding boxes.
We also provide qualitative results in Fig. 17 The results show that method can integrate well with Stable Diffusion v1.5.
Source Image
505
D06
0D5
PhotoSwop
PrPInw-1gg
Durs w/ 501.5
Duns
person sitting in the snow looking at the mountains sks dog sitting in the snow looking ot the mountains
volleyboll sitting in the middle of pool sks duck sitting the middle of
Figure 17: Qualitative results of our method with other Stable Diffusion models.
Table 6: Quantitative results of our method with other Stable Diffusion models.
Methods
CLIP-1
PSNR
LPIPS
MSE
SSIM t
CLIP-T
Inference Time
Ours w/ SD
74.27 72.70
27.98 15.25
47.04 327.23
25.24 350.01
86.01 64.80
26.10 21.23
15.09s 13.38s
Ours2
H MORE CHALLENGING CONCEPTS
We include more uncommon challenging concepts from DreamBench++ (Peng et al. 2024). We provide qualitative results in Fig. 18 The results show that these challenging concepts indeed reduce the foreground consistency our method. Nevertheless, our approach still faithfully completes the concept swapping and surpasses all compared methods all metrics.
19

---

Published as a conference paper at ICLR 2025
Source
Gustoeized Corcept
PhotoSwap
PrPTnviFO
Ours
photo of basket books cup photo basket of books
E
photo of plant
table photo plant
piacy on white table
cartoon goet big horns
robot big harns small
single pink lotus flower in the middle of a green leaf a sks teddy bear the middle green leaf
bind stonding metol fence sks tree-like character stonding metal fence
Figure 18: Qualitative results of more challenging concepts.
VISUALIZATION OF THE SWAPPING PROCESS
We visualize the swapping process of our INSTANTSWAP to intuitively demonstrate how manages shape differences between the source and target concepts. As depicted in Fig. [9 our method can flexibly swap the source concept for the target concept through an optimization approach. Addition- ally, we visualize the images of each swapping step along with their corresponding gradients. As shown in Fig. 20 we select three different stages of swapping timesteps: 0-4, 100-104, and 200-204 with the SSGU factor set to It can be observed that the gradients adjacent timesteps update the target image towards similar direction which does not result in significant changes within each update. This observation supports the use of proposed SSGU.
J ABLATION STUDY ON AUTOMATIC BOUNDING Box GENERATION
To comprehensively analyze how the combination of self-attention and cross-attention for automatic bbox generation affects the performance, we first set the dementt-wise exponentiation of a'and A in Eq. (8) to as and a₂ respectively:
Å (A*)0- (A9)²
(20)
where A E [0, 1].1 A E [0, [0,1) H and W are height and weight of the image latent, A is the cross-attention map corresponding to the source concept. We apply the threshold to normalized Å e H to obtain the bbox B., We exclude the trivial case where both as and Oa are zero and explore different combinations of self-attention and cross-attention. The qualitative results is shown in Fig. 21 (1) a, 0, a 1, 2: In this case, A is an all-ones matrix, so each element in A has the same value, which is equal to the sum of all elements A After normalization and applying threshold, the entire bounding box is activated. (2) a, a. = 0: In this case, A is vector of all ones, each item in Å is the sum of the elements the corresponding
20

---

Published as a conference paper at ICLR 2025
Source Inage
Iteratio 50
Iteration 180
Iteration 158
Iteration 200
Iteration 250
Iteration 300
Iteration 350
Iteration 400
Iteration 458
Iteration 500
Final Result
couple of birds standing top of rocky beach- sks duck standing top of rocky beach
Source Inage
Iteration
Iteration Iteration 150
Iteration 200
Iteration 250
Iteration 380
Iterationn 350
Iteration 400
Iteration 458
Iteration 580
Final Result
person holding shell in front of the ocean-> person holding sks flower front the ocean Figure 19: Visualization of the whole swapping process of INSTANTSWAP.
JFn
24
Tino
Qum
Zhot
9hm
Z1oz
9um
Z1a1
9hz
Ahos
Figure 20: Visualization of the target image and the corresponding gradient adjacent timesteps.
row of A Since A the output of softmax, the sum of each row's elements in A Therefore, A is also a matrix of all ones, which ultimately results in the entire bounding box being activated. (3) a, = 1, a. 1: In this case, the target region in the image cannot be highlighted, resulting in a larger bbox, which causes the background of the source image to also be unnecessarily altered. (4) a, 2, ae 0: In this situation, using self-attentio map alone cannot effectively highlight the foreground region, resulting in a very imprecise bounding box. (5) a, 2, a, = 1, 2: this
case, the self-attention map takes leading role in the bbox generation process, producing smaller
21

---

Published as a conference paper at ICLR 2025
Source Image
C2 0,0.
Oy 0,0, - O, -1,0,- a, a,m1,a," m1 Oy 11,0 G 0,02 0, -2,00 0, 2.0, -
Coneap
sitting
sks sitting on top
Figure 21:
Qualitative results the ablation study on automatic bounding box generation. We add black borders to the bboxes for better visibility.
bbox, thereby reducing foreground consistency. (6) = 1, = (Ours): Our setting achieves proper balance between foreground and background consistency, fully covering the source concept while minimizing background modification. The quantitative in Tab. results in the table further corroborate our analysis.
Table 7: Quantitative ablation results of automatic bbox generation.
CLIP-1 f PSNR
LPIPS
MSE
SSIM
CLIP-T 1
0,1
72.70
15.25
327.23
350.01
64,80
21.23
0,
72.70
15.25
327.23
350.01
64.80
21.23
1,0
72.70
15.25
327.23
350.01
64.80
21.23
1.
74.18
23.83
93.12
82.44
82.95
25.19
2 (Ours)
75.00
27.39
47.68
27.87
86.58
25.74
2,
73.66
26.49
65.56
56.81
84.81
24.97
73.67
19.15
25.12
2.2
72.66
30.86
34.38
15.38
87.54
24.78
Source Image
M=3
bird standing in the grass near some trees sks teapot standing in the grass near some trees
bird sitting on top of rock in the water- sks teapot sitting top of rock in the water
Figure 22: Qualitative results of the ablation study the number of customization target images.
K ANALYSIS OF THE NUMBER OF CUSTOMIZATION TARGET IMAGES
Before concept swapping, our method utilizes DreamBooth (Ruiz et al. 2023) and set of images (typically fewer than 5) X; {x,)MM for customizing certain concepts, where M is the number of target images used in the customization process. this section, we discuss how this number M affects the final performance of our method. When customizing the teapot concept method
22

---

Published as a conference paper at ICLR 2025
effectively perform concept swapping with just one target image for customization shown in Fig. 22 As the number of target images increases, the customization becomes more detailed, result- ing more natural outcomes, such as better alignment of shadows and highlights with the lighting angle. Additionally, we conduct quantitative experiments on foreground and prompt consistency metrics. The results in Tab. g indicate that the performance of our method improves M increases.
Table 8: Quantitative ablation results of the number of customization target images.
M
M = 3
CLIP-11 CLIP-T 1
74.78 28.94
75.40 28.93
75.57 29.13
75.65
29.44
L DISCUSSION ON THE TIME COST OF THE CUSTOMIZATIOON PROCESS
Before concept swapping, our method utilizes DreamBooth (Ruiz al. 2023) for customizing cer- tain concepts. As shown in Tab. 9 DreamBooth only requires about 8.5 minutes for each concept. Meanwhile, our method is not limited using DreamBooth for customization. can be combined with any other more efficient customization methods such as Custom Diffusion (Kumari et al. 2023) which only requires about 3.5 minutes for each concept improve customization efficiency. We sualize the results of our method combined with Custom Diffusion in Fig. 23
Table 9; Training time of different customization methods.
DreamBooth RRuiz et al. 2023
Custom Diffusion (Kuman et al. 2023
Training Time
8.5min
3.5min
Ours a/ Gustom
Outomiced
PhetoSmp
carrying surfboord beach ddunglastes> carrying surfboard wet beach
small animal standing top of grass field sducio standing top dry grass field
Figure 23: Visualization of method with Custom Diffusion.
M UTILIZING TEXTUAL INVERSION FOR CUSTOMIZATION
Our method is compatible with any customization techniques for concept swapping. To demonstrate its generalization capability, we replace DreamBooth (Ruiz et al., 2023) with Textual Inversion (Gal al. 2022) and conduct comprehensive experiments. As shown in Fig. 24 method integrates effectively with Textual Inversion to perform customized concept swapping across shape differences, highlighting its generalization ability. We also provide detailed quantitative results. As shown in the Tab. 10, ours with Textual Inversion achieves similar performance in background consistency and inference time to our original method. Due the limited customization capabilities of Textual Inversion, its foreground consistency is not as high as with our original method.
N DETAILED USAGE OF THE CUSTOMIZATION METHODS
In this section, we detail the use of customization methods in our method. Taking DreamBooth (Ruiz al. 2023) example, given set of images (typically fewer than five) X, (xx) represents
23

---

Published as a conference paper at ICLR 2025
Ours of Testual
Custanized
valleyball
middle
bird sitting top
aitting on top
Figure 24: Qualitative results of our method with Textual Inversion.
Table 10: Quantitative results of our method with Textual Inversion.
FG
BG
Overall
Method
CLIP-1
PSNR
LPIPS x10* 4
MSE
x10
SSIM x10"
CLIP-T t
Time (s)
Ours w/ Textual Inversion
72.20
28.16 27.39
45.91
24.20 27.87
86.76 86.58
24.57
19.92
Ours
75.00
47.68
25.74
19.83
specific concept O,. DreamBooth utilizes these images, combined with text prompt containing rare token and the name of O, (e.g., "a sks teapot"), to fine-tune a text-to- image diffusion model. During the finetuning process, DreamBooth is trained using the reconstruction loss of Eq. and prior preservation loss, which leverages the model's semantic prior encourage diverse generation results. After customization with DreamBooth, we obtain a checkpoint the concept O and token that semantically represents it. During concept swapping, this checkpoint is loaded, and the rare token is used the target branch to activate the concept, enabling customized concept swapping. Furthermore, DreamBooth (e.g.. Custom Diffusion (Kumari et al.) 2023) and Textual Inversion (Gal et al. 2022 Related discussions can be found in Appendices and
method be integrated with other customization methods besides
O APPLY SECR TO DIT-BASED ARCHITECTURES
In our method, SECR leverages semantic information to enhance the cross-attention maps of con- cepts. This technique can be applied to any generative model with cross-attention layer enhance semantic information in regions of interest. However, current DiT-based models, such as SD3 (Esser
al. 2024), typically do include cross-attention layer, as the original paper (Peebles & Xie
FOG
cake on wooden plate a sks duck on wooden plate
Two beats are docked on the shore of lake a sks
duck is docked on shore of a lake
lotus flower growing leaf sks
bird is standing in the middle of field - sks Teapot is standing in the middle of field
teapot is growing on green leaf
Figure 25: Qualitative results of image naturalness.
24

---

Published as a conference paper at ICLR 2025
Table 11: Quantitative results image naturalness.
SDS
DDS
CDS
PhotoSwap
PaPlnv+P2P
Ours
Human Preference Score
-1.41
-0.21
-0.01
-0.86
-0.14
0.51
2023) of DiT shows that using an adaptive layer norm (adaLN) block yields better results than cross-altention layer. Consequently, most subsequent DiT-based models, including SD3, prioritized using adaLN over employing gross-attenting
Thus, we explore the architecture of SD3 investigate how our SECR could be applied SD3. SD3 consists of multiple blocks named MM-DiT Blocks, where each block contains only a self-attention layer. We focus how images and text interact within these blocks. Within the MM-DiT Block, the image features and text features are processed separately by two separate branches to obtain image latent 2y and text latent 2t- Before inputting these latent into the attention layer, the corresponding Qu. K2. V of and Qt, Kr, V2 of 2 concatenated together and undergo self-attention whole. After the attention layer, the concatenated output is split back into the corresponding z' and 2/, returning their independent branches. Specifically, the input of the attention layer in MM-DiT block can be represented as:
Q Q-
Q: Q.
K₁ Kr
V V
(21)
The attention output can be calculated through:
QKT
Softmax
V.
(22)
Vd
where
Qt
TeK? Q.KT
Q,K?
QK
[KT
KJ
(23)
Qx
Q
Combing Eq.
21
Eq.
12
and Eq.
23
we can obtain the attention output:
QKT
(Q(K)
S(Q(
Softmax
V
(Q,Kk?)
S (QQ
V
(24)
V (Q.K3 Vn S (QoK7)
S (Q.K?)
+ (Q,Kz) V
where 5(.) stands for Softmax (aaammax(;(aa)) (Q,K? of Q, K and V, the upper part in Eq. latent output. For image latent output Zz. we have:
Vd'), = 1, x. Corresponding to the concatenation method
24 is the text latent output, and the lower part is the image
Q2K? Vd
QuK Vd
= Softmax
V + Softmax
Vg-
(25)
cross attention
self attention
Eq. 25 demonstrates that the attention layer in the MM-DIT block is similar to cross-atten layer plus self-attention layer. The first term in Eq. (55) the cross-attention operation between the image feature and text feature. The section term is the self-attention operation of the image feature. Consequently, our SECR can be applied to the cross-attention part of Eq. 25 to enhance seman- tic information in regions of interest. However, since SD3 is trained based rectified flow (Liu et al. 2022 20222] Albergo Vanden- Handen-Eijnden 2022 Lipman et al, (022), existing score distillation meth- ods (Poole al. [2022] Hertz 2023) cannot be directly integrated with it. In future work, will continue to explore this potential direction.
P IMPACT OF SECR ON THE NATURALNESS OF IMAGES
In this section, we discuss the impact of our method on the naturalness of generated images. We apply the SECR strategy foreground concepts within the cross-attention layers obtain the
25

---

Published as a conference paper at ICLR 2025
semantic-enhanced representation of the source/target object. This enhanced representation of the foreground region interacts effectively with the features of other surrounding objects and the back- ground in the self-attention layers, mitigating inconsistencies and unnaturalness in the target image. We present comprehensive qualitative and quantitative results to support our analysis. As shown in method successfully enables
Fig. [25] although apply the SECR strategy to the foreground, the correct interaction of concepts with the whole scene. Specifically: (1) Our method seamlessly inpaints objects in the background that the foreground concept occludes (upper left examples). (2) It generates reflections of the target concept in water (upper right examples), (3) It accurately places the target concept on background objects (lower left examples). (4) also generates natural shadows of the target concept that are consistent with the environmental lighting angles (upper left and lower right examples). Additionally, considering that image naturalness is subjective human assessment, we use the human preference metric (Xu et al. 2024) to quantify the naturalness of images. A higher score indicates a better alignment with human preferences. As shown Tab. [ our method the only one that receives positive human evaluation score, demonstrating sigmificant advantage over other methods.
ADDITIONAL QUALITATIVE RESULTS Q
Due to page limitations, we included only a limited number of visualizations in the main paper. To clearly and intuitively demonstrate the effectiveness of method, shown in Fig. [26] first provide the bounding boxes used in the qualitative comparisons of the main paper. Subsequently, provide more qualitative comparisons in Figs. 28 and 29
Finally, present gallery (see Figs. t0 35) of additional qualitative results to comprehensively showcase the performance of our proposed INSTANTSWAP.
Custendoed
506
PhotoSwop
PrPInwrF3
Dure
Bounding Box
kitten playing pool with balls- kitten playing pool with condle
walking leaf covered ground duck walking leaf covered ground
bind sitting with trees background abs flower sitting wire with trees the background
gronge sitting
building Bunglasscs sitting on building
polac bsoc standing on top sandy barn standing on top sondy
sheep stonding on top a lush green field sks teapot stonding on top green field
Figure 26: Visualization of the bounding boxes used in Fig.
26

---

Published as a conference paper at ICLR 2025
R LIMITATION AND FUTURE WORKS
Although the features the entire image have been fully interacted within the self-attention layers, the base diffusion model (e.g., Stable Diffusion (Rombach et al. 2022)) is unable accurately perceive the relative sizes of specific concepts. We visualize this phenomenon in Fig. [27] the first row shows an excessively large cup, the second row depicts toy duck large as the little girl, and the third row displays pair of sunglasses nearly as large as the table. Similar issues occur in all compared methods. Nevertheless, our method still achieves the best concept swapping results.
We hope our INSTANTSWAP can inspire future research, particularly in efficiently managing concept swapping with obvious shape variance. Future work could foeus on (1) extending image-based customized concept swapping to the video domain; (2) developing metrics that more accurately and comprchensively reflect the characteristics of the customized concept swapping task; and (3) achieving more lightweight and precise concept swapping.
Source Image
3DS
DOS
CD6
PhotoSwap
P2
Ours
dog sitting the grass with a tay in the background sks cup sitting in the grass with toy in the background
a little girl walking on beach with a dog a little girl walking on a beach with sks duck
painting on wooden desk and chair a white room sks sunglosses on wooden desk and chair white room
Figure 27: Failure cases.
POTENTIAL SOCIAL IMPACT
The purpose of our approach is to provide users with customized swapping tool that can perform efficient and precise customized concept swapping. However, the outstanding performance of our method also presents certain risks, including the potential for malicious third parties create decep- tive fake images. These risks are not unique to our approach; all image editing methods face similar challenges. Thanks to (Sun et al. 2023), more and more studies are focusing on detecting images that have been altered by generative models, which prevent the misuse of related methods.
27

---

Published as a conference paper at ICLR 2025
Source Image
505
005
CD5
PhotoSwap
PPPInv-PP
Ours
a wooden frome mockup on white marble background a sks duck on a white marble background
woman is playing golf on grassy field sks duck is playing golf on a grassy field
single pink lotus flower the middle a green leaf sks teapot the middle a green leaf
brown and white bird standing on top a dirt field sks teapot standing on top of dirt field
horned animal walking on a road sks teapot walking on a
Figure 28: More qualitative comparisons.
28

---

Published as a conference paper at ICLR 2025
Source Image
Sb5
b26
CDS
PhotoSwop
Oura
Bounding Box
photo of butterfly on leaf photo of sks condle on a leaf
blue and white bird sits branch sks candle sits branch
little boy sitting ground
condle sitting ground front barn
teddy beors sitting a sks cup sitting on
blue wooden lifeguard house on beach during daytime sks sunglesses on beach during daytime.
Figure 29: More qualitative comparisons.
29

---

Published as a conference paper at ICLR 2025
Cow
cat
condle
Concept Swap
Concept Swap
house
barn
mushroom condle
Concept Swap
Concept Swop
boat
barn
bird
condle
Concept Swop
Concept Swop
whole
cat
condle
Concept Swap
Concept Swap
antelope
barn
bird
candle
Concept Swap
Concept Swap
birds
candle
Concept Swap
Concept Swap
antelope
barn
cup
condle
Concept Swap
Concept Swop
Figure 30: More qualitative results of the barn and candle.
30

---

Published as a conference paper at ICLR 2025
cup
Concept Swap
Concept Swap
chair
cat
mug
cup
Concept Swop
Concept Swap
dog
figurine
cup
Concept Swap
Concept Swap
bird
cat
duck
cup
Concept Swap
Concept Swap
duck
cat
bird
cup
Concept
Concept Swap
bird
vase
cup
Concept Swop
Concept Swap
painting
cat
cat
cup
Concept Swap
Concept Swap
Figure 31: More qualitative results of the and cup.
31

---

Published as a conference paper at ICLR 2025
bird
dog
flower
VOSE
Concept Swap
Concept Swop
stone
dog
Concept Swap
Concept Swap
antelope
dog
vase
Concept Swap
Concept Swap
elephant
bird
dog
Concept Swap
Concept Swap
vase
bird
dog
Concept Swap
Concept Swap
person
dog
cat
vose
Concept Swap
Concept Swop
bird
dog
vose
Concept Swap
Concept Swop
Figure 32: More qualitative results of the dog and vase.
32

---

Published as a conference paper at ICLR 2025
vase
flower
cat
sunglasses
Concept Swap
Concept Swap
bird
flower
bird sunglasses
Concept
Concept Swap
*
bird
flower
orange
sung losses
Concept Swop
Concept Swap
bird
sunglasses
flower
Concept Swap
Concopt Swap
boat
sunglasses
bird
flower
Concept Swap
Concept Swap
animal
flower
bird
sunglasser
Concept Swep
Concept Swop
ontelope
flower
pointing sunglasses
Concept Swap
Concept Swap
Figure 33: More qualitative results of the flower and sunglasses.
33

---

Published as a conference paper at ICLR 2025
vase
flower
vase
flower
Concept Swop
Concept Swop
bird
flower
bird
flower
Concept Swap
Concept Swap
bird
flower
bird
flower
Concept Swop
Concept Swap
flower
cup
flower
Concept Swap
Concept
bird
flower
bird
flower
Concept Swap
Concept Swap
animal
flower
animal flower
Concept Swap
Concept Swap
ontelope flower
ontelope flower
Concept Swop
Concept Swap
Figure 34: More qualitative results of the teapot.
34

---

Published as a conference paper at ICLR 2025
cat
duck
cup
duck
Concept Swap
Concept Swap
valleyball
duck
chair
duck
Concept Swap
Concept Swap
animal
duck
cat
duck
Concept Swop
Concept Swap
bear
boat
duck
Concept Swap
Concept Swap
child
duck
bird
duck
Concept Swop
Concept Swap
figurine
duck
cat
duck
Concept Swap
Concept Swap
horse
duck
apple
Concept Swop
Concept Swap
Figure 35: More qualitative results of the duck.
35


Reviews:
[{"review_id": "ZjZ4BB1ooO", "reviewer": "Reviewer_nPYy", "summary": "This paper proposes InstantSwap, a training-free framework for Customized Concept Swapping(CCS). CCS works on transfers the target concept described by target images and target prompt to the location of source concept in the source image. This paper utilizes the cross attention map and the self-attention map of U-Net in diffusion model for source image and source prompt to extract bounding box of the source concept automatically. Then they apply the bounding box to filter the gradients in background from a refined SDS loss. In this way, they can achieve an optimization preserving the background information. To emphasize the concepts in the images, they also use the semantic information of corresponding prompts and the estimated bbox to augment representation of concepts. Additionally, this paper also presents a step-skipping gradient update strategy which reuse previous gradients for current iteration to increase the inference speed. Experiments present the advantages of the proposed method over previous works.", "questions": "1. How does the proposed method work for multi-object scenario?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method presents a complete pipeline for improve effectiveness and efficiency in Customized Concept Swapping(CCS) task.\n2. The proposed method obtains state-of-the-art performance compared to previous works.\n3. This paper also contributes benchmark for Customized Concept Swapping(CCS) task.", "weaknesses": "1. Theoretical analysis about why we can directly apply mask on gradient computing is missing. Masking will produce a distribution shift, why it can converge to a reasonable solution requires some analysis.\n2. How combining self-attention and cross-attention for automatic bbox generation affects the performance seems not be discussed.\n3. How the number of target images affect the performance is not mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OOoqb9Xyo6", "reviewer": "Reviewer_iqew", "summary": "Recent advances in Customized Concept Swapping (CCS) enable text-to-image models to swap concepts, but existing methods struggle to maintain foreground and background consistency, particularly with large shape disparities, and often require time-intensive processes. InstantSwap addresses these challenges by using bounding box analysis and cross-attention mechanisms to enhance both foreground and background consistency while limiting modifications to the background and enhancing foreground focus. This method reduces computation time by periodically calculating gradients, enhancing efficiency with minimal performance loss, and extensive evaluations show InstantSwap's effectiveness and adaptability.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation is clear.\n\nThe results appear promising and solid.\n\nThe experiments are thorough.\n\nThe writing is easy to follow.", "weaknesses": "For each concept replacement, the method first needs to train a DreamBooth model and then perform score distillation, which is time-intensive.\n\nBoth the source and reference branches use DreamBooth-tuned UNet. It would be beneficial to validate the method using text inversion to demonstrate its generalization capability.\n\nWhat about the failure cases?\n\nIt's interesting that the method can handle concepts with significant shape changes. If the original image's concept is very small, resulting in a small bounding box, how does the target image’s foreground region expand without additional processing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o5VrotDUUJ", "reviewer": "Reviewer_VFJh", "summary": "This paper proposes a training-free customized concept swapping framework. It derived bounding boxes from the Attention to map to help preserve the background information during optimizing the latents. The gradient is updated periodically for a better tradeoff between quality and inference time. A semantic enhanced module is further proposed to improve foreground consistency. Both quantitative and qualitative experiments are conducted to validate the effectiveness of this approach.", "questions": "Please see the questions in weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The experimental results are comprehensive and promising.\n\n- A benchmark dataset designed for CCS task is proposed.\n\n- The writing is fluent and easy to understand.", "weaknesses": "- It would be better to give an introduction on the customization methods. This introduction could also help readers to understand the difference brought by integrating customization method into image editing.\n\n- Since this approach needs to modify the cross attention, I wonder if it could be applied to the DiT-based architectures, like SD3.\n\n- It seems that the semantic-enhanced operation enhances the semantic of the source/target object while mitigates the object’s interaction with the surrounding objects and background. Will it make the image unnatural?\n\n- I am curious why P2P fails completely at this task. From the results displayed in the paper of P2P, I would expect P2P to be able to fulfill the task of changing objects. Is it because the customization method does not fit well with P2P?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)