InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences

Authors: Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, Xiu Li
Primary area: generative models
Keywords: Text-to-image generation, Image editing, Customized concept swapping

Abstract:
Recent advances in Customized Concept Swapping (CCS) enable a text-to-image model to swap a concept in the source image with a customized target concept. However, the existing methods still face the challenges of $\textit{\textbf{inconsistency}}$ and $\textit{\textbf{inefficiency}}$. They struggle to maintain consistency in both the foreground and background during concept swapping, especially when the shape difference is large between objects. Additionally, they either require time-consuming training processes or involve redundant calculations during inference. To tackle these issues, we introduce InstantSwap, a new CCS method that aims to handle sharp shape disparity at speed. Specifically, we first extract the bbox of the object in the source image $\textit{automatically}$ based on attention map analysis and leverage the bbox to achieve both foreground and background consistency. For background consistency, we remove the gradient outside the bbox during the swapping process so that the background is free from being modified. For foreground consistency, we employ a cross-attention mechanism to inject semantic information into both source and target concepts inside the box. This helps learn semantic-enhanced representations that encourage the swapping process to focus on the foreground objects. To improve swapping speed, we avoid computing gradients at each timestep but instead calculate them periodically to reduce the number of forward passes, which improves efficiency a lot with a little sacrifice on performance. Finally, we establish a benchmark dataset to facilitate comprehensive evaluation. Extensive evaluations demonstrate the superiority and versatility of InstantSwap.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730648627554

thread_type:
official_review

<thread_object>
creator: reviewer_vfjh

<content>
summary:
This paper proposes a training-free customized concept swapping framework. It derived bounding boxes from the Attention to map to help preserve the background information during optimizing the latents. The gradient is updated periodically for a better tradeoff between quality and inference time. A semantic enhanced module is further proposed to improve foreground consistency. Both quantitative and qualitative experiments are conducted to validate the effectiveness of this approach.

strengths:
- The experimental results are comprehensive and promising.

- A benchmark dataset designed for CCS task is proposed.

- The writing is fluent and easy to understand.

weaknesses:
- It would be better to give an introduction on the customization methods. This introduction could also help readers to understand the difference brought by integrating customization method into image editing.

- Since this approach needs to modify the cross attention, I wonder if it could be applied to the DiT-based architectures, like SD3.

- It seems that the semantic-enhanced operation enhances the semantic of the source/target object while mitigates the object’s interaction with the surrounding objects and background. Will it make the image unnatural?

- I am curious why P2P fails completely at this task. From the results displayed in the paper of P2P, I would expect P2P to be able to fulfill the task of changing objects. Is it because the customization method does not fit well with P2P?

questions:
Please see the questions in weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730772129518

thread_type:
official_review

<thread_object>
creator: reviewer_iqew

<content>
summary:
Recent advances in Customized Concept Swapping (CCS) enable text-to-image models to swap concepts, but existing methods struggle to maintain foreground and background consistency, particularly with large shape disparities, and often require time-intensive processes. InstantSwap addresses these challenges by using bounding box analysis and cross-attention mechanisms to enhance both foreground and background consistency while limiting modifications to the background and enhancing foreground focus. This method reduces computation time by periodically calculating gradients, enhancing efficiency with minimal performance loss, and extensive evaluations show InstantSwap's effectiveness and adaptability.

strengths:
The motivation is clear.

The results appear promising and solid.

The experiments are thorough.

The writing is easy to follow.

weaknesses:
For each concept replacement, the method first needs to train a DreamBooth model and then perform score distillation, which is time-intensive.

Both the source and reference branches use DreamBooth-tuned UNet. It would be beneficial to validate the method using text inversion to demonstrate its generalization capability.

What about the failure cases?

It's interesting that the method can handle concepts with significant shape changes. If the original image's concept is very small, resulting in a small bounding box, how does the target image’s foreground region expand without additional processing?

questions:
Please see the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730868655057

thread_type:
official_review

<thread_object>
creator: reviewer_npyy

<content>
summary:
This paper proposes InstantSwap, a training-free framework for Customized Concept Swapping(CCS). CCS works on transfers the target concept described by target images and target prompt to the location of source concept in the source image. This paper utilizes the cross attention map and the self-attention map of U-Net in diffusion model for source image and source prompt to extract bounding box of the source concept automatically. Then they apply the bounding box to filter the gradients in background from a refined SDS loss. In this way, they can achieve an optimization preserving the background information. To emphasize the concepts in the images, they also use the semantic information of corresponding prompts and the estimated bbox to augment representation of concepts. Additionally, this paper also presents a step-skipping gradient update strategy which reuse previous gradients for current iteration to increase the inference speed. Experiments present the advantages of the proposed method over previous works.

strengths:
1. The proposed method presents a complete pipeline for improve effectiveness and efficiency in Customized Concept Swapping(CCS) task.
2. The proposed method obtains state-of-the-art performance compared to previous works.
3. This paper also contributes benchmark for Customized Concept Swapping(CCS) task.

weaknesses:
1. Theoretical analysis about why we can directly apply mask on gradient computing is missing. Masking will produce a distribution shift, why it can converge to a reasonable solution requires some analysis.
2. How combining self-attention and cross-attention for automatic bbox generation affects the performance seems not be discussed.
3. How the number of target images affect the performance is not mentioned.

questions:
1. How does the proposed method work for multi-object scenario?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732260427582

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We extend our sincere gratitude to all the reviewers (**R1-nPYy**, **R2-iqew**, and **R3-VFJh**) for their insightful and considerate reviews, which help us to emphasize the contributions of our approach. 

We are very encouraged to hear that the reviewers recognized the **clear motivation** (R2) and **effectiveness** (R1, R2) of our method, our **comprehensive and solid experiments** (R2, R3), the **advantageous results** we presented (R1, R2), the **benchmark** we proposed (R1, R3), and the **fluent writing** of our paper (R2, R3).

We would also like to express our sincere gratitude to the reviewers for their insightful identification of areas where our manuscript could be strengthened. 
We have taken all the suggestions carefully and updated our previous version. In the revised manuscript, we have made the following **changes** and highlighted them in different colors for different reviewers (**R1-red**, **R2-blue**, **R3-brown**):

1. Added the ablation study on automatic bounding box generation in **Appendix H**. (R1)
2. Added the analysis of the number of target images in **Appendix I**. (R1)
3. Added the discussion on the time cost of the customization process in **Appendix J**. (R2)
4. Added the results of combining our method with Textual Inversion in **Appendix K**. (R2)
5. Added more failure cases and analysis in **Appendix P**. (R2)
6. Refined **Section 3.2.3** and added a detailed introduction of the customization methods we used in **Appendix L**. (R3)
7. Added a detailed analysis of how our SECR can be applied to DiT-based architecture (e.g., SD3) in **Appendix M**. (R3)
8. Added more qualitative and quantitative results to demonstrate the naturalness of our resulting image in **Appendix N**. (R3)

We sincerely hope to **engage in further discussion with the reviewers to ensure all concerns have been fully addressed**. If any aspects of our work remain unclear, we welcome **any further feedback** to help improve our manuscript. Thank you very much again!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a training-free framework for Customized Concept Swapping(CCS). It transfers a target concept described by images and a prompt to the location of a source concept in the source image. 

The paper is well written, and the method achieves state-of-the-art performance across various thorough experiments. The work additionally contributes a benchmark for Customized Concept Swapping(CCS) task.

After the post-rebuttal discussion, all reviewers unanimously agree that the paper should be accepted. Remaining minor concerns are the inclusion of more challenging concepts to the proposed benchmark. We encourage the authors to address this before submitting the camera-ready version of the paper.

Overall, this is solid work that is relevant to the community and should be presented at ICLR.