PHOTOSWAP: Personalized Subject Swapping in Images
Authors: Jing Gu
Yilin Wang
Nanxuan Zhao
Tsu-Jui Fu
Wei Xiong
Qing Liu
Zhifei Zhang
HE Zhang
Jianming Zhang
HyunJoon Jung
Xin Eric Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: image editing
diffusion model
text to image generation
TL;DR: Subject swapping with a personal item

Abstract: In an era where images and visual content dominate our digital landscape, the ability to manipulate and personalize these images has become a necessity.
Envision seamlessly substituting a tabby cat lounging on a sunlit window sill in a photograph with your own playful puppy, all while preserving the original charm and composition of the image. 
We present \emph{Photoswap}, a novel approach that enables this immersive image editing experience through personalized subject swapping in existing images.
\emph{Photoswap} first learns the visual concept of the subject from reference images and then swaps it into the target image using pre-trained diffusion models in a training-free manner. We establish that a well-conceptualized visual subject can be seamlessly transferred to any image with appropriate self-attention and cross-attention manipulation, maintaining the pose of the swapped subject and the overall coherence of the image. 
Comprehensive experiments underscore the efficacy and controllability of \emph{Photoswap} in personalized subject swapping. Furthermore, \emph{Photoswap} significantly outperforms baseline methods in human ratings across subject swapping, background preservation, and overall quality, revealing its vast application potential, from entertainment to professional editing.

PHOTOSWAP:
Personalized Subject Swapping in Images
Jing Gu'
Yilin Wang
Nanxuan Zhao² Tsu-Jui Fu
Wei Xiong
Qing Liu Xin Eric Wang!*
Zhifei Zhang?
He Zhang?
Jianming Zhang² HyunJoon Jung?
University of California, Santa Cruz
Adobe
University of California, Santa Barbara
https://photoswe/ siteaaa..gtttbb github. io/
reference image
source image
Phatoswap
reference image
image
000
Figure 1: Photoswap can effortlessly replace the subject in a source image, which could be either synthetic (first two rows) real (bottom row), with personalized subject specified in reference images, while preserving the original subject pose and the composition of the source image.
Abstract
In an era where images and visual content dominate our digital landscape, the abil- ity to manipulate and personalize these images has become necessity. Envision seamlessly substituting tabby cat lounging graph with your own playful puppy, all while preserving the original charm and composition of the image. We present Photoswap, a novel approach that enables this immersive image editing experience through personalized subject swapping in existing images. Photoswap first learns the visual concept of the subject from reference images and then swaps it into the target image using pre-trained diffusion models training-fire frre manner. We establish that well-conceptuualized visual subject can be seamlessly transferred to any image with appropriate self-attention and cross-attention manipulation, maintaining the pose of the swapped subject and the overall coherence of the image. Comprehensive experiments underscore the efficacy and controllability of Photoswap in personalized subject swapping. Fur- thermore, Photoswap significantly outperforms baseline methods in human ratings across subject swapping, background preservation, and overall quality, revealing its vast application potential, from entertainment to professional editing.
sunlit window sill in photo-
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

Introduction
Imagine a digital world where the boundaries of reality and creativity blur, where photograph of a tabby cat lounging on sunlit window sill can effortlessly be transformed feature your playful puppy in the same pose. Or envision yourself as a part of a famous movie scene, replaced seamlessly with the original character while preserving the very essence and composition of the scene. Can we achieve this level of personalized image editing, not just with expert-level photo manipulation skills, but in an automated, user-friendly manner? This question lies at the heart of personalized subject swapping, the challenging task of replacing the subject in image with user-specified subject, while maintaining the integrity of the original pose and composition. opens up a plethora of applications in areas such as entertainment, advertising, and professional editing.
Personalized subject swapping is a complex undertaking that comes with its own set of challenges. The task requires profound comprehension of the visual concept inherent to both the original subject and the replacement subject. Simultancously, it demands the seamless integration of the new subject into the existing image. One of the critical objectives in subject swapping is to preserve the similar pose of the replacement subject. It is crucial that the swapped subject seamlessly fits into the original pose and scene, creating a natural and harmonious visual composition. This necessitates careful consideration of factors such as lighting conditions, perspective, and overall aesthetic coherence. By effectively blending the replacement subject with these elements, the final image maintains a sense of continuity and authenticity.
Existing image editing methods fall short in addressing these challenges. Many of these techniques restricted to global editing and lack the finesse needed to seamlessly integrate new subjects into existing images. For example, for most text-to-image (T2I) models, a slightly prompt change could
lead to totally different image. Recent works (Nichol er al. [2022] Meng et al. [2022] Couairon et al. 2022 Cao el al. 2023 Zhang el al. 2023] allow user to control the generation with an additional input such as user brush, semantic layout, or sketches. However, it is still challenging to guide the
generation process to follow users' intent on the generation of object shape. texture, and identity. Other approaches (Hertz al. 2022 Tumanyan ef al. 2023 Mokady el al. 2023) have explored the potential of using text prompts to edit image content in the context of synthetic image generation.
Despite showing promise, these methods are not yet fully equipped to handle the intricate task of swapping subjects in existing images with user-specified subjects.
Therefore, we present Photoswap, novel framework that leverages pre-trained diffusion models for personalized subject swapping in images. In our approach, the diffusion model learns to represent the concept of the subject (O2). Then the representative attention map and attention output saved in the source image generation process will be transferred into the generation process of the target image to generate the new subject while keeping non-subject pixels unchanged. Our extensive experiments and evaluations demonstrate the effectiveness of Photoswap. Not only does method enable the seamless swapping of subjects in images, but it also maintains the pose of the swapped subject and the overall coherence of the image. Remarkably, Photoswap outperforms baseline methods by a large margin in human evaluations of subject identity preservation, background preservation, and overall quality of the swapping (e.g.. 37.3% vs. 27.1% in terms of overall quality). The contributions of this work are as follows: 1) We present new framework for personalized subject swapping in images. 2) We propose a training-free attention swapping method that governs the editing process. 3) The efficacy our proposed framework is demonstrated through extensive experiments including human evaluation.
2 Related Work
2.1 Text-to-Image Generation
In the early stages of text-based image generation, Generative Adversarial Networks (GANs) (Good. fellow es al. 22020; Brock et al.) 2018 Karras et (0199) were widely used due to their exceptional ability to produce high-quality images. These models aimed to align textual descriptions with synthe- sized images through multi-modal vision-language learning, achieving impressive results on specific domains (c.g., bird, chair and human face). When combined with CLIP (Radford et al. (202I), a large pre-trained model that learns visual- visual-textual representations from millions of caption-image pairs, GAN models (Crowson et al. (2022) have demonstrated promising outcomes in cross-domain text-to-image (T2l) generation. Recently, T21 generation has seen remarkable progress with auto-
regressive (OpenAl 2021 Ding et al. 202 and diffusion models (Nichol et al. 2022 Gu et al.
2

---

2022
2022), offering diverse outcomes and can synthesize high- qugh-quality images closely aligned with textual descriptions in arbitrary domains.
OpenAl 2022
Saharia et al.
Rather than focusing on T2I generation tasks without any constraints, subject-driven T2I genera- tion (Nitzan el al., 2022 Casanova el al.) 2021: Ruiz el al. 2023) requires the model to identify the specific object from set of visual examples and synthesize novel scenes incorporating them based on the input text prompts Building upon modern diffusion techniques, recent approaches such as Dream-
Kumari et al. 2023] Mokady et al. 2023) learn to invert special tokens from given set of images. By combining these tokens with text prompts, they generate personalized unseen images. To improve data efficiency, retrieval
Booth (Ruiz el al. 2023) and Textual Inversion (Gal el al. 2023a b
augmentation techniques (Sheynin el al.) 2023 Blattmann el al. 2022 Chen el al.) (023) leverages external knowledge bases to overcome limitations posed by rare entities, resulting in visually relevant appearances and enhanced personalization. In our work, we aim to tackle personalized subject swapping, not only preserving the identity of subjects in reference images, but also maintaining the context of the source image.
2.2 Text-guided Image Editing
Text-guided image editing manipulates existing image based the input textual instructions, while preserving certain aspects or characteristics of the original image. Early works based on GAN models (Karras el al. (2019) only limits to a certain object domain. Diffusion-based methods (Zhang et al. Nichol al. [2022] Feng et al. (023) break this barrier and support text- eext-guided image editing. Though these methods generate stunning results, many of them suffer from conducting local editing, and additional manual masks (Meng ef al. 2022 Zeng et al.) 2023 Meng et al.) (002) are required to constrain the editing regions, which often tedious draw. By employing cross- coss-attention (Hertz et al., 2022) or spatial characteristics (Tumanyan el al. 2023), the local editing can be achieved but struggles with non-rigid transformations (e.g., changing pose) and retaining the original image layout structure. While Imagic (Kawar et al. 2023) addresses the need for non-rigid transformations by fine-tuning a pre-trained diftusion model to capture image-specific appearances, it requires test- time finetuning. which is not time-efficient for deployment. Moreover, relying solely on text as input lacks precise control. In contrast, propose a novel training-free attention swapping scheme that enables precise personalization based on reference images, without the need for consucing finetuning.
2.3 Exemplar-guided Image Editing
Exemplar-guie image editing covers a broad range of applications, and most of the works (Wang et al. 2019; Huang al. 2018; Zhou al. (021) can be categorized exemplar based image
translation tasks, conditioning on various information, such as stylized images (Liu el al. 2021 Deng
er al. 2022 Zhang et al. 2022), layouts (Yang et al. 2023b Li et al. |023c] Jahn et al. [021]) skeletons (Li al. 2023c), sketches/edges (Seo el al.
2023). With the convenience of stylized images,
image style transfer (Liao et al.) 2017] Zhang el al. 2020) receives extensive attention, replying on
methods to build a dense correspondence between input and reference images, but it cannot deal with local editing. To achieve local editing with non-rigid transformation, conditions like bounding boxes and skeletons are introduced, but require drawing efforts from users, which sometimes are hard to obtain. A recent work (Yang el al. 2023a) poses exemplar-guided image editing task as an inpainting task with the mask and transters the semantic content from the reference image to the source one, with the context intact. Unlike these works, we propose a more user- weer-friendly scenario by conducting personalized subject swapping with only reference images and obtain high-quality editing results. DreamEdit (Li et al. (003b) uses iterative inpainting to achieve subject replacement. Nevertheless. the existing approach fails to establish a comprehensive correlation between the source and target subjects. Conversely, our technique ensures that pivotal attributes, such as body gestures and facial expressions, remain unaltered.
3 Preliminary
Diffusion models are a type of generative model that operates probabilistically. In this process, an image is created by gradually eliminating noise from the target that is characterized by Gaussian noise. In the context of text-to-image generation, diffusion model typically involves a process where an initial random image is gradually refined step by step, with each step guided by a learned model, until it becomes realistic image. The changes to the image spread out and affect many pixels over
3

---

Encode Input
T steps of Diffusion
Decode to output
P3 photo o cat sitting green bucket on tuble
If continnu (T-) steps...
Original Image
2
Z
A
DOIM Ewersion
Initial noise copy
A.
x (T-A)
O
0"
Z
27
A
Z
Generated Result
Object
Source Image Diffuiion process
Image Encoder
Pe: A photo sitting in bucket table.
New Concept
Target Image Diffusion process
Image Decoder
sileeff attention map,
Prompt with new concept
attention map, d attention output
Figure 2: The Photoswap framework. Given several images of new concept, the diffusion model first learns the concept and converts it into a token. The upper part is the generation process of the source image, while the bottom part is the generation process of target image. The initial noise feature 24. is copied from zÿ of the source. The attention output and attention map the source image generation process would be transferred to the target image generation process. The final feature =f decoded to output the target image. Refer to Sec. for more details.
time. Given an initial random noise 2T ~ I), the diffusion model gradually denoise 2,, which gives 2t-1-
Diffusion models are probabilistic generative models that learn to generate images by simulating a random process called a diffusion process. In the image generation process, the diffusion model gradually predicts the noise at the current diffusion step and denoises to get the final image. In this study, utilize pre-trained text-to-image diffusion model, Stable Diffusion (Rombach ef 2022). which encodes the image into latent space and gradually denoises the latent variable to generate a new image. Stable Diffusion is based on U-Net architecture (Ronneberger et al. 2015), which generates latent variable zt-1 conditioned on given text prompt P and the latent variable 2, from the previous step t:
-c-1 EØ ( P,t)
(I)
The U-Net consists of layers that include repetition of celf-attention and cross-attention blocks. This study focuses on manipulating self self-attention and cross- aatsettttntionn to achieve the task of personalized subject swapping.
4
The Photoswap Method
Providing a few reference images of a personalized target subject O,, Photoswap can seamlessly swap it with another subject O, in a given source image I.. The Photoswap pipeline is illustrated in Figure 2 To learn the visual concept of the target subject O, we fine-tune diffusion model with reference images and do object inversion to represent O using special token # Then, to substitute the subject in the source image, first obtain the noise 27 that can be used reconstruct the source image I.. Next, through the U-Net, we obtain the needed feature map and attention output in the self-attention and cross-attentin Sec. 4.2 Finally, during the target image generation process that conditioned target text prompt P., the first steps, those intermediate variables (M, A, and Φ) would be replaced with corresponding ones obtained during the the source image generation process. In the last
layers, including M, A, and ó (which we will introduce in
the noise zT and
'For a synthetic image, 2 is the initial noise used to generate it. For real image, we utilize an improved version of DDIM inversion (Song et al. 2020) get the initial noise and re-generate the source image. See Sec. 5.1 for details.

---

Figure SVD visualization of sell-attention maps. Each image's attention map is resized to 64x64 at every layer, and we calculate the average map across all layers for all diffusion time steps. Most significant components are extracted with SVD and visualized. Remarkably, the visualized results demonstrate strong correlation with the layout of the generated image. The top rows visualization about synthetic images while the bottom two rows are about real images.
(T - X) steps, no attention swapping is needed and we can continue the denoising process usual to obtain the final resulting image. Sec. discusses the visual concept learning technique we used, and Sec. 4.2 details the training-free attention swapping method for controllable subject swapping.
4.1 Visual Concept Learning
Subject swapping requires a thorough understanding of the subject's identity and specific character- istics. This knowledge enables the creation of accurate representations that align with the source subject. The subject's 's's identity influences the composition and perspective the image, including its shape, proportions, and textures, which affect the overall arrangement of elements. However, existing diffusion models lack information about the target subject (O₂) in their weights because the training data for text-to-image generation models does not include personalized subjects. To overcome this limitation and generate visually consistent variations of subjects from a given reference set, we need to personalize text-to-image diffusion models accurately. Recent advancements have introduced various methods, such as fine-tuning the diffusion model with distinct tokens associated with specific subjects, to achieve this "pensonalization" (Gal et al. [202aa] Ruiz et al.) 2023) Kumari er al. 2023). In our experiments, method. It's worth noting that alternative concept learning methods can also effectively employed with our framework.
primarily utilize DrcamBooth (Ruiz et al. 2023) visual concept learning
4.2 Controllable Subject Swapping via Training-free Attention Swapping
Subject swapping poses intriguing challenges, requiring the maintenance of the source image's 's spatial layout and geometry while integrating new subject concept within the same pose. This necessitates preserving the critical features the source latent variable, which encapsulates the source image information, and leveraging the influence of the target image text prompt P,, which carries the concept token, to inject the subject into the image.
The central role of the attention layer in orchestrating the generated image' layout has been well- established in prior works (Hertz et al. 2022 CCo ét al. 2023) TTumanyan et al.) 2023). To keep non-subject pixels intact, we orchestrate the generation of the target image 1; by transferring vital
5

---

Diffusion Steps
Figure 4: Self-attention map visualization across diffuusio time steps. This representation reveals that the layout of the generated image is intrinsically embedded in the self-attention map from the initial steps. Consequently, to assert control over the layout, it is imperative to commence the attention swap at the carliest stages of the process.
variables to the target image generation process. Here, explore how distinct intermediate variables within the attention layer can contribute to controllable generation in the context of subject swapping.
Within the source image generation process, we denote the cross-attention map as A the self- attention map as M,, the cross-attention output as v and the self-attention output as of The corresponding variables in the target image generation process are denoted as A M vi %% where i represents the current diffusion step.
In the self-attention block, the latent feature z; is projected into queries qa. keys k;, and values v.- We obtain the afffattennton block's 's output Φ; using the following equation:
Ma where M, = Softmax
(q,k:²)
(2)
where M, is the self-attention map, and d; is the feature output from the self-attention layer. The cross-attention block's 's output 35 is:
v, A, where A; Softmax (q.k:²'
where A; is the cross-attention map. In both self-attention and cross-attention, the attention map M. and A, are correlated the similarity between qi and k; acting as weights that dictate the combination of information in 2/- In this work, the manipulation of the diffusion model focuses on self-attention and cross-attention within U-Net, specifically, swapping Φ, M, and A, while keeping unchanged.
Self-attention map M, as it calculates the similarity within spatial features after linear projection, plays a pivotal role in governing spatial content during the generation process. As visualized in Figure 3 capture M during the image generation and highlight the leading components via Singular Value Decomposition (SVD). This visualization reveals a high correlation between M and the geometry and content of the generated image. Further, when visualizing the full steps of the diffusion process (Figure 4 we discern that the layout information is mirrored in the self-attention from the initial steps. This insight underscores the necessity initiating the swap early on to prevent the emergence of a new, inherent layout.
Cross-attention map A is determined by both latent variable and text prompt, as in Equation and
a can be viewed as weighted sum of the information from a text prompt. Copying A; to A during the target image generation process improves the layout alignment between the source image and the target image.
Self-attention output o, derived from the self-attention layer, encapsulates rich content information from the source image, independent of direct computation with textual features. Hence, replacing of with oi enhances the preservation of context and composition from the original image. Our observations indicate that & exerts more profound impact on the image layout than the cross- attention map A.

---

Algorithm The Photoswap Algorithm
source image J, reference images O,, source image text prompt P.. target image text prompt Pt, diffusion model
Inputs:
0r < 0, O, D. Finetune diffusion model to include the new concept mage Encoder(Is)) P.)o Using DDIM to guarantee re-construction 2T <- 24 > Using the same starting noise for i T,
27 + DDIMInver
1,... do
<², 0, M ; A + (a, Denoise to get the attention output and map for source
image
M 13 A + Eg.(2f, Pt. > Denoise to get the attention output and map for target image
M A + $ M A s M A!,
61-(2 ir, Φ M ; A?) 1) & Denoise the updated attention map and output
DDIM Sampler x7. E*) (') Sample next latent variable for source image
+ DDIM Sampler( 2 e*) > Sample next latent variable for source image
end for 1 = ImageDecoder((5) return 4
function SWAP(Ø 8 .M", A", ,)
6- - (i< >6) o² Control self-attention feature swap
M <i< 7M) M' & Control self-attention Map swap A' (i< AA)?AA A' Control cross-attention map swap
return Φ², "'M, A' end function
Cross-attention output v), emanating from the Pross-attention layer, embodies the visual concept of the target subject. It is vital to note that substituting tross-attention output v/g" with of would obliterate all information from the target text prompt P., as illustrated in Equation Given that kf and of are projections of target prompt embeddings, we retain u/ unchanged to safeguard the target subject's 's identity.
Algorithm I provides the pseudo-code of our full Photoswap algorithm.
5
Experiments
5.1
Imlementation Details
For the implementation of subject swapping on real images, we require an additional process that utilizes an image inversion method, specifically the DDIM inversion (Song el af.) 2020)) to transform the image into initial noise. This inversion method relies on reversed sequence of sampling to achieve the desired inversion. However, there exist inherent challenges when this inversion process is applied in text- text-guided synthesis within a plassifier-free guidance setting. Notably, the inversion can potentially amplify the accumulated error, which could ultimately lead subpar reconstruction outcomes. To fortify the robustness of the DDIM inversion and to mitigate this issue, we further optimize the null text embedding, detailed in Mokady et al. (2023). The incorporation of this optimization technique bolsters the effectiveness and reliability of the inversion process, consequently allowing for a more precise reconstruction. Without further notice, the DDIM inversion in this paper is enhanced by null text embedding optimization.
During inference, we utilize the DDIM sampling method with 50 denoising steps and classifier- free guidance of 7.5. The default step AA for cross-attention map replacement is 20. The default step As for self-attention map replacement is 25, while the default step for self-attention feature A replacement is 10. Please refer Appendix [A] for analysis on attention swapping step, and Appendix
[E] for detailed Unet layer swapping. Note that the replacement steps may change to some specific checkpoint. As mentioned in Sec. g the target prompt P; is just source prompt P. with the object token being replaced with the new concept token. For concept learning, we mainly utilize DreamBooth (Ruiz el af. 2023) to finetune a stable diffusion 2.1 to learn the new concept from 3 images. The learning rate set to 1e-6. We use Adawm optimizer with 800 hundred training steps.
7

---

reference image
source image
Photoswap
reference image
source image
Photoswap
Figure 5: Photoswap results across various object and image domains, demonstrating its wide applicability. From everyday objects to cartoons, the diversity in subject swapping tasks has showcased the versatility and robustness of our framework across different contexts.
We finetune both the U-net and text encoder. The DreamBooth training takes around 10 minutes on machine with A100 GPU cards.
5.2 Personalized Subject Swapping Results
Figure 5 shoocases the effectiveness of our Photoswap technique for subject swapping. Our approach excels preserving crucial aspects such as spatial layout, geometry, and the pose of the original subject while seamlessly introducing reference subject into the target image. Remarkably, even in cartoon images, method ensures that the background remains intact during the subject change process. A notable example is the "cat" image, where our technique successfully retains all the intricate details from the source image, including the distinctive "Whiskers." " This demonstrates our framework's ability to accurately capture and preserve fine-grained information during subject swapping. Please refer to Appendix
further demonstrate the versatility of Photoswap by showcasing its effectiveness in multiple subject swap and occluded object swap scenarios. As depicted in Figure 6a we present a source image featuring sunglasses, which successfully replaced with reference glass while preserving original layout of the sunglasses. Similarly, in Figure 6b we observe a source image with dog partially occluded by suit. The resulting swapped dog wears suit that closely matches the occluded region. These examples serve to highlight the robustness of proposed Photoswap method in handling various real- world cases, thereby enabling users to explore a broader range of editing possibilities. Refer Appendix F for identity control.
5.3 Comparison with Baseline Methods
Personalized object swap is new task and there is no existing benchmark. However, we could modify the existing attention manipulation based methods. More specifically. we used the same concept learning method DreamBooth to finetune the same stable diffusion checkpoint to inject the new concept. To fairly compare with our results, we modified the existing prompt-based editing method P2P (Hertz et al., 2022). an editing method based on diffusion models. Note that origin P2P only works on a pair of synthetic images, in our setting we use the same concept learning DreamBooth and fix the seed to allow concept swapping. On the other hand, PnP (Tumanyan et al. 2023) could

---

reference image
source image
Photoswap
reference image
source image
Photoswap
(a) Multi-subject swap.
(b) Occluded subject swap.
Figure 6: Photoswap results on multi-subject and occluded subject scenarios. The results show that Photoswap can disentangle and replace multiple subjects at once. Also, Photoswap can identify the target object while avoiding influencing the non-subject pixels.
reference image
source image
P2P+ E2P+DreaBBoott
Photoswap
Figure 7: Qualitative comparison between Booth and Photoswap. We can observe that Brot is capable of achieving subject swapping. However, it faces challenges in preserving both the background and the reference subject accurately, while for Photoswap, is robust to handle various cases.
also be implemented in a similar setting, however, we found PnP usually can not lead to satisfactory object swapping and may lead to a huge difference between the source image and the generated image. We suspect that it is because PnP is designed for image translation so it does not initiate the attention manipulation step from the beginning step. The qualitative comparison between Photoswap and Boot shown in Figure ? Check Appendix for performance on other concept learning method. We observe that P2P with DreamBooth could achieve basic object swap, but it still suffers from background mismatching issues.
Human Evaluation. We conduct human evaluation to study the editing quality by (1) Which result better swaps the subject as the reference and keeps its identity; (2) Which result better preserves the background; (3) Which result has better overall subject-driven swapping. We randomly sample examples and adopt Amazon MTurk to compare between two results. Please refer to Appendix D for details. To avoid potential bias, we hired 3 Turkers for each sample. Table demonstrates the comparison between our Photoswap and P2P. Firstly, more turkers (42.4%) denote that our Photoswap better swaps the subject yet keeps its identity at the same time. Moreover, we can also preserve the background in the source image (39.3% vs. 30.2%), which is another crucial goal of this editing. In summary, Photoswap precisely performs subject swapping and preserves the remaining part from the input, leading to an overall superiority (37.3%) to P2P (27.1%).
Ku et al. (023) shows Photoswap
Amazon Mechanical Turk (MTurk): https: htops:///ww.... www.mmtuk..co

---

Table I: Qualitative comparison between Photoswap and DoooDremmBooth, PnP+DreamBooth. MasaCut--DreamBoohh separately. includes 2000 comparisons on 400 image pairs, with 200 real images and 200 synthetic images. Each image pair contain 5 ratings from different human annotator in Amazon Turk. SS means Subject Swapping, BP means Background Preservation, and OQ means
Overall Quality.
Photoswap
P2P
Tie
Photoswap
PnP
Tie
Photoswap
MasaCtrl
Tie
SS
42.4%
30.0%
27.6%
52.7%
22.1%
25.2%
79.1 %
10.3%
10.6%
BP 39.3%
30.2%
30.5%
49.1%
20.7%
30.2 %
72.8%
10.2%
17.0%
OQ
37.3%
27.1%
35.6%
55.1 %
22.4%
22.5%
83.3% %
10.3%
6.4%
achieve SOTA performance by comparing with DreamEdit (Li et al. et al. 2023a).
2023b) and BLIP-Diffusion
5.4
Ethics Exploration
image
source image
Photoswap
reference image
source image
Photoswop
Figure 8: Results real human face images across different races. Evidently, the skin colors also successfully transferred when swapping white person with black person, and vice versa.
Like many Al technologies, text-to-image diffusion models can potentially exhibit biases reflective of those inherent in the training data (Sasha Luccioni al.) 2023 Perera and Patel 2023). Given that these models are trained on vast text and image datasets, they might inadvertently learn and perpetuate biases, such as stereotypes and prejudices, found within this data. For instance, should the training data contain skewed representations or descriptions of specific demographic groups, the model may produce biased images in response to related prompts.
However, Photoswap has been designed to mitigate bias within the generation process of a text-to- image diffusion model. It achieves this by directly substituting the depicted subject with the intended target. In Figure [ we present our evaluation of face swapping across various skin tones. It is crucial to note that when there is a significant disparity between the source and reference images, the swapping results tend homogenize the skin color. As a result, advocate for the of Photoswap on subjects of similar racial backgrounds to achieve more satisfactory and authentic outcomes. Despite these potential disparities, the model ensures the preservation of most of the target subject's specific facial features, reinforcing the credibility and accuracy of the final image.
6 Conclusion
This paper introduces Photoswap, novel framework designed for personalized subject swapping in images. To facilitate seamless subject photo swapping, we propose leveraging self-attention control by exchanging intermediate variables within the attention layer between the source image and reference images. Despite its simplicity, our extensive experimentation and evaluations provide compelling evidence for the effectiveness of Photoswap. Our framework offers a robust and intuitive solution for subject swapping, enabling preferences. In the future, we plan further advance the method address those common failure issues to enhance the overall performance and versatility personalized subject swapping.
to effortlessly manipulate images according to their
References
Blatmannn, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B. (2022). Retrieval-Augment Diffusson Models. NeurIPS.
Brock, A., Donahue, J., and Simonyan, K. (2018). Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations.
10

---

Cao. M., Wang, X.. Qi. Z., Shan, Y., Qie. X., and Zheng, Y. (2023). Masactrl: Tuning- free mutual self-attention control for consistent image synthesis and editing. In ICCV.
Casanova, A., Careil, M., Verheek, J., Drozdzal, M., and Romero-Soriano, A. (2021). Instance-Conditioned GAN. In NeurlPS.
Chen, W., Hu, H., Saharia, C., and Cohen, W. W. (2023). Re- Re-Imagen: GatrivallAAgmenne Text-to tol-tto--mgee Generator. In ICLR.
Couairon, G., Verbeek, J., Schwenk, H., and Cord, (2022). Diffedit: Diffusion- semantic image editing with mask guidance. The Eleventh International Conference on Learning Representations.
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E.,, Castricato, L.,, and Raff, E. (2022). Vggan-clip: Open domain image generation and editing with natural language guidance. In ECCV.
Deng. Y., Tang, F., Dong, W., Ma, C.,, Pan, X., Wang, I.., and Xu, C. (2022). Stytr2: Image style transfer with transformers. In CVPR.
Ding, M., Yang, Z., Hong, W. Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021). Cogview: Mastering lext-to-image image generation via transformers. NeurIPS.
Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang. X. E., and Wang. W. Y. (2023).
Training-Free FFree Structured Diffusion Guidance for Compositional Text-to-lmage Synthesis. In ICLR.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, H., Chechik, G., and Cohen-Or. (2023a). An Image is Worth One Word: Personalizing Text-to-lmage Generation using Textual Inversion. In ICLR.
Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., and Cohen-Or, D. (2023b). Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 42(4), 1-13.
Goodfellow, I., Pouget-Abasie J., Mirza, M., Xu, B., Warde-Farles Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2020). Generative adversarial networks. Communications of the ACM.
Gu, S., Chen, D., Bao. J., Wen, F., Zhang, B., Chen, D., Yuan, L., and Guo, B. (2022). Vector quantized diffusson model for texi-lo-image synthesis. In CVPR.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-or, D. (2022). Prompt-to-promp image editing with cross-attention control. In The Eleventh International Conference on Learning Represen- tations.
Huang, X., Liu, M.-Y., Belongie, S., and Kautz, J. (2018). Multimodal unsupervised image-to-imant translation. In ECCV.
Jahn, M., Rombach, R., and Ommer, B. (2021). High-resolution complex scene synthesis with transformers. arXiv.
Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In CVPR.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Trani, M. (2023). Imagic: Text- based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6007-6017.
Ku, M. Li, T.. Zhang, K., Lu, Y., Fu, X.. Zhuang, W., and Chen, W. (2023). Imagenhub: Standardizing the evaluation of conditional image generation models.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. (2023). Multi-Concept Customization of Text-to-lmage Diffusion. In CYPR.
Li, D., Li, J., and Hoi, S. C. (2023a). Blip-diffusion: Pre-trained subject representation for controllable eext-to-image generation and editing. arXiv preprint arXiv:2305. 14720.
Li, T., Ku, M., Wei, C., and Chen, W. (2023b). Dreamedit: Subject-driven image editing. arXiv preprint arXiv:2306.12624
Li Y. Liu, H., Wu, Q. Mu, F., Yang, J.. Gao, J.. Li. C., and Lee, Y. (2023c). Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12511-22521.
Liao, J. Yao, Y., Yuan, L., Hua, G., and Kang. S. B. (2017). Visual atribute transfer through deep image analogy. ACM Transactions Graphies.

---

Liu, S., Lin, T., He. D., F.. Wang. M.. Li. X., Sun, Z., Li. Q. and Ding, E. (2021). Adaatin: Revisit attention mechanism in arbitrary neural style transfer. In ICCV.
Meng. C.. He, Y., Song. Y., Song. J. Wu, J. J.u. and Ermon, S. (2022). SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations.
Mokady, R., Hertz, A., Aherman, K., Pritch, Y., and Cohen-Or, D. (2023). Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 6038-6047,
Nichol, A. Q. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I,, and Chen, M. (2022). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804, PMLR.
Nitzan, Y., Aberman, K., He, Q. Liba, O., Yarom, M., Gandelsman, Y., Mosseri, I., Pritch, Y., and Cohen-or, D. (2022). MyStyle: A Personalized Generative Prior. In Special Interest Group on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia).
OpenAl (2021). DALL-E: Creating images from text. http:ppp//pppaa..cccc//eeeeraaaa
OpenAl (2022). DALL-E2. https://openai. http:/pppaai..coopooooeeeeo cowmppooouuc/tallleeeee
Perera, M. V. and Patel, V. M. (2023). Analyzing bias diffusion-based face generation models, arXiv preprint arXiv:2305.06 06402.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learming transfcrable visual models from natural language supervision. In ICML.
Rombach, R., Blatmann. A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In CVPR.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In MICCAL. Springer.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman. K. (2023). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In CVPR.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. (2022). Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS.
Sasha Luccioni, A., Akiki, C., Mitchell, M., and Jernite, Y. (2023). Stable bias: Analyzing socictal representations in diffusion models. arXiv e-prints, pages arXiv-2303.
Seo. J.. Lee, G.. Cho, S., Lee, J. and Kim, S. (2023). Midms: Matching interleaved diffusion models for exempaa-bbsed image translation. In Proceedings the AAAI Conference Artificial Intelligence. volume 37, pages 2191-2199.
Sheynin. S., Ashual, O., Polyak, A.. Singer, U., Gafni, O.. Nachmani, E., and Taigman, Y. (2023). KNN- Diffusion: Image Generation via Large-Scale Retrieval. In ICLR.
Song, J., Meng, C., and Ermon, S. (2020). Denoising diffusion implicit models. In International Conference on Learning Representations.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. (2023). Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IERF/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930.
Wang. M.. Yang, G., Li, R.. Liang, R. Zhang, S.H.., Hall, P. M., and Hu, S.MM. (2019). Example-guide style-consistent image synthesis from semantic labeling. In CVPR.
Yang. B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., and Wen, F. (2023a). Paint by example: Exemplar- based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18381-18391
Yang. Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., al. (2023b). Reco: centtollonttolled text-to- -magtt generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14246-14255
Zeng, Y., Lin, Z., Zhang, J., Liu, Q. Collomosse, J., Kuen, J., and Patel, M. (2023). Scenecomposer: Any-level semantic image synthesis. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages
12

---

Zhang, L., Rao, A., and Agrawala, M. (2023). Adding conditional control to lext-to-image diffusion models.
Zhang, P., Zhang, B., Chen, D., Yuan, L., and Wen, F. (2020), Cross-domain correspondence learning for exemplar-base image translation. In CVPR, pages 5143-5153
Zhang. Y., Huang. N., Tang, F., Huang, H., Ma, C., Dong, W., and Xu, C. (2022). Inversion-based creativity transfer with diffusion models. arXiv.
Zhou, X., Zhang, B., Zhang, T., Zhang. P., Bao, J., Chen, D., Zhang, Z., and Wen, F. (2021). Cocosnet v2: Full-resolution correspondence learning for image translation. In CVPR.
13

---

A Attention Swapping Step Analysis
In this section, we visualize the effect of the influence of swapping steps of different components. As discussed in the main paper, self-attention output o, and self-attention map M, derived from the self-attention layer, encompasses comprehensive content information from the source image, without relying on direct computation with textual features. Previous works such as Hertz al. (2022) did not explore the usage of & and M in the object-level image editing process.
Attention map and attention output swap step
Changing do steps of self-attention output swap
Reference Images
Changing d steps of self-a aelf-attention map swap
Source Image
Changing λ., steps cross-attention map swap
Figure 9: Results at different swapping steps. With consistent steps, swapping the self-attention output provides superior control over the layout, including the subject's gestures and the background details. However, excessive swapping could affect the subject's identity, as the new concept introduced through the text prompt might be overshadowed by the swapping of the attention output or attention map. This effect is more clear when swapping the self-attention output XĄ. Furthermore, that replacing the attention map for an extensive number of steps can result in an image with significant noise, possibly due to a compatibility issue between the attention map and the e vector.
observed
Figure 9 provides visual representation of the effect of incrementally increasing the swapping step for one À hyperparameter while maintaining the other two at zero. Although all of them can be utilized for subject swapping, they demonstrate varying levels of layout control, At the same swapping step, the self-attention output á offers more robust layout control, facilitating better alignment of gestures and preservation of background context, In contrast, the self-attention map M and cross-attention map A demonstrate similar capabilities in controlling the layout.
However, extensive swapping can affect the subject's identity, the novel concept introduced via the text prompt might be cclipsed by the swapping of the attention output or attention map. This effect becomes particularly evident when swapping the self-attention output. This analysis further informs the determination of the default As, Am. and AA values. While the cross-attention map A facilitates more fine-grat -nn-ggainedd generation control, given its incorporation of information from textual tokens, we discovered that & offers stronger holistic generation control, bolstering the overall output's 's quality and integrity.
B Results of Other Concept Learning Methods
We mainly use DreamBooth as the concept learning method in the experiments, primarily due to its superior capabilities in learning subject identities (Ruiz al., (023). However, method is not strictly dependent on any specific concept learning method. In fact, other concept learning methods could be effectively employed to introduce the concept of the target subject.
To illustrate this, we present the results of Photoswap when applying Text Inversion (Gal et al. 2023a). We train the model using 8 A100 GPUs with a batch size of 4, a learning rate of 5e-4, and set the training steps to 1000. Results in Figure TO indicate that Text Inversion also proves to be an effective concept learning method, it successfully captures key features of the target object.
14

---

reference image
source image
Photoswap
reference image
source image
Photoswap
Figure 10: Results of Text Inversion (Gal al., 2023a) as the concept learning module. It can successfully capture key subject features, but its performance drops when representing complex structures such as human faces.
Nevertheless, we observe that Text Inversion performance notably underwhelming when applied human faces. We postulate that this is because Text Inversion focuses on learning new embedding for the novel concept, rather than finetuning the entire model. Consequently, the capacity to express the new concept becomes inherently limited, resulting its less than optimal performance in certain areas.
C Failure Cases
reference image
source image
Photoswap
reference image
source image
Photoswop
Figure 11: Failure cases. The model sometimes struggles to accurately reconstruct hand details and complex background information such as formula on a whiteboard.
Here we highlight two common failure cases. First, the model struggles to accurately reproduce hands. When the subject includes hands and fingers, the swapping results often fail to precisely mirror the original hand gestures or the number of fingers. This issue could be an inherited challenge from Stable Diffusion. Moreover, Photoswap can encounter difficulties when the image comprises complex information. As illustrated in the lower row of Figure 11, Photoswap fails to reconstruct the complicated formula on whiteboard. Therefore, while Photoswap exhibits strong performance across various scenarios, it's crucial to acknowledge these limitations when considering its application in real-world scenarios involving intricate hand gestures complex abstract information.
D Evaluation Details
For real images, we sourced all our images from internet searches. We employed the search prompt: a photo of <target>'. Here, the <target> variable could be specific celebrity (e.g., 'Elon Musk') or descriptive scene (e.g.. cute yellow cat running in the forest'). ) The celebrity names were identified through a Google search with the prompt "top celebrities 2023." For scene descriptions, we curated a list of 100 distinct search prompts to source images from the internet. In total, aggregated 1 1,000 images using these prompts. All prompts, along with the collected image, will made available in our next revision.
For synthetic images, we generated 1,000 images using text prompts with the text-to-image diffusion model version 2.1. These prompts spanned range, including those centered humans (e.g.. "A photo woman looking left and smiling, Van Gogh style") and those focused on non-hume non-human subject (e.g., "An old car in the middle of the road, flanked by trees during autumn") "). All prompts used in synthetic image generation will also be released too. For the human evaluation exhibited in this rebuttal and in the paper, we utilized the "random" package in Python to sample 200 images from both the real and synthetic datasets, respectively. Each image underwent evaluation by five distinct individuals on Amazon Turk. In total, this resulted in a comprehensive 6,000 ratings. as compared our model against P2P, PnP. and MasaCtrl. Our findings unequivocally indicate that our model surpasses all other methods in performance.
15

---

reference image
source image
Photoswap
identity
Figure 12: Ablation results of M in Photoswap. With M value increase, the generated one is more similar to the style and identity of source image and dissimilar to the reference subject, and vice versa.
For source image processing, all we do is resize the image into standard 512x512 pixels. is worth noting that there is also no postprocessing needed for generated images.
E Unet Swapping Layer
There are 16 Unet layers in the Stable Diffusion backbone. We tested the effect of different layers according to their position (up. middle, or lower) in the Unet and their latent size. Through hundreds of experiments on the layer combination, we found that while the latent size of the layer plays a minor role, the position of the layer to be swapped matters. More specifically, we find the essential part do swapping operations in all the decoder layers in the Unet.
F
Controlling Subject Identity
The effectiveness the proposed mutual self-attention is demonstrated through both synthetic image synthesis and real image editing. Additionally, we perform an analysis of the control strategy with varying values of M during the denoising process. Figure 12 provides insights into this analysis. is observed that when applying self-attention control with large swapping step Am for M, the synthesized image closely resembles the source image in terms of both style and identity. In this scenario, all contents from the source image preserved, while the subject style learned from the reference subject is disregarded. As the value of M decreases, the synthesized image maintains the subject from the reference image while retaining the layout and pose of the contents from the source image. This gradual transition in the control strategy allows for balance between subject style transfer and preservation of the original image's ' contents.
16


Reviews:
[{"review_id": "cN1UcMCRl7", "reviewer": "Reviewer_z7Bi", "summary": "In this paper, they propose Photoswap, which could seamlessly swap personalized subjects into source images. The swap process is training-free and only leverages the manipulation of self-attention and cross-attention. The swapped object could maintain the pose of the source image without hurting the coherence of the image. The results are promising on both synthetic images and real images.", "questions": "refer to Weaknesses", "limitations": "Authors have stated the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The task of personalized subject swapping in images is fancy and interesting. This is the first work that could handle such challenging swapping task.\n- The paper excels in providing thorough illustrations to explain the design approach, making it easier for readers to understand the intricacies of the proposed method. Furthermore, the clear determination of parameters is commendable, as it enhances the reproducibility and applicability of the research.\n- The results presented in the paper are impressive. Even when dealing with challenging scenarios such as multi-subject swap and occluded subject swap, the proposed method performs remarkably well.", "weaknesses": "- In the user study, the comparisons are conducted on 99 examples. Although the number of test images and total votes (3 votes per example) is small, it may introduce bias into the results. I recommend addressing this concern by either increasing the number of test images. Additionally, it is unclear from the paper whether the user study was conducted solely on synthetic images. It would be valuable to explore the performance of your proposed method on real images as well, as this would provide further insights into its practical applicability and generalization.\n- In Fig. 7, the presented results are not convincingly better than those of the P2P+dreambooth method. To strengthen your claims, I suggest providing additional comparisons, especially on real images. \n- When discussing the swapping of the self-attention layer, it is unclear which self-attention layer in the U-Net architecture you are referring to. Please provide clarification on which specific self-attention layer is being swapped, as this information is essential for replicating your results and understanding the significance of this modification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer on the acknowledge of our task and appreciation of the results. Besides the great suggestion,  we will explain all the concerns as follows:\n\n> larger scale human evaluation comparison with P2P on both synthetic and real images \n\nThanks so much for the suggestion. During this rebuttal, we conducted more human evaluation on 400 image pairs containing 200 real images and 200 synthetic images, with each having 5 voting. Here, as you suggested we share the P2P+dreambooth baseline comparision below, where Photoswap consistently outperforms P2P on all three aspects. \n\n| Metrics      | Photoswap    | P2P+DreamBooth   | Tie |\n| :----        |    :-----:   |          :-----:|   ----:|\n| Subject Identity Preservation      |**43.4%**| 30.0%   | 27.6% |\n| Background Preservation   |     **39.3%**   | 30.2%      |  30.5%|\n| Overall Swapping Quality   |   **37.3%**     | 27.1%| 35.6%   |\nTable 1.  Human evaluation on  400 images with 200 real image 200 synthetic images. Each image pair contains 5 ratings from Amazon Turk.  \n\n> Qualitative comparison with P2P on real image.\n\nAs suggested, we attached the qualitative comparison with P2P+DreamBooth in Figure 2 in the attached pdf. From the example, we can clearly see that Photoswap performs better on both subject swapping and background preservation, especially on preserving the pose of the source subject.  We will include more comparisons on real image in the revision.\n\n> Please provide clarification on which specific self-attention layer is being swapped\n\nThere are 16 Unet layers in the Stable Diffusion backbone. We tested the effect of different layers according to their position (up, middle, or lower) in the Unet and their latent size. Through around hundreds of experiments on the layer combination, we found that while the latent size of the layer play a minor role, the position of the layer to be swapped matters. More specifically, we find the essential part is to do swapping operation in all the decoder layers in the Unet. We will also release code to verify all findings to ensure all the results can be reproducible.\n\n\n"}, {"review_id": "XDqHEKgrBa", "reviewer": "Reviewer_TtiF", "summary": "The authors present a solution to the problem of personalized subject swapping, where the goal is to replace the subject in an image with another user-defined subject. Authors leverage pre-trained diffusion models to make local edits to an input image based on a collection of images of the subject to be inserted. This is done through concept inversion on images of a target subject and scheduling attention map swapping during the diffusion process, switching between the use of attention maps from the original diffusion process and the newer process conditioned on the concept modified text prompt.", "questions": "- How does this method perform when there is a large domain shift in the edited subject? For example, if I wanted to replace a car with a tree, does the method still perform well? \n- Does the structure of the subject in the reference image fix the structure of the inserted subject?", "limitations": "Authors adequately address the limitations and potential ethical risks in using image diffusion models trained on internet-scale data, especially discussions on the potential biases present in these models.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clarity: Work is well presented and method is logically sound. Motivation behind method is also well justified. For example, the attention map visualizations are helpful in motivating their scheduled attention swapping method.\n- Significance: The problem of personalized subject swapping is practically interesting and hasn't been explored with depth in prior works. The practical effectiveness of large image diffusion models have been limited by its controllability. This work takes a reasonable step towards making such models accessible and useful to users who don't have domain specific knowledge in image editing.\n- Quality of results: Experimental results are impressive, and achieving subject swapping with high fidelity.\n", "weaknesses": "- Missing baseline: Recent work such as Plug-and-Play diffusion has also explored the effectiveness of feature and attention map injection during the diffusion process for image editing. Using DreamBooth + plug-and-play [1] is a more reasonable comparison than using P2P. It is also an important comparison as it is not clear to me whether the author's proposed attention map swapping method is really more effective than the method proposed int plug-and-play.\n- Evaluation: Considering the lack of unified metrics for this task, human evaluation is adequate. However, it would be helpful to see other metrics such as subject fidelity [2] reported. It is unclear whether this method leads to lower subject preservation than prior work like DreamBooth.\n- Results: Results shown all include subject swapping that are very similar in nature (e.g. cars are swapped for a different looking car). Subjects being swapped may not necessarily be similar in nature (see questions sections for more detailed comment).\n- Contribution: both concept inversion and editing through attention map insertion have been explored in prior work, this work seems to combine both effectively, but contribution from their method is relatively small.\n\n\n[1] Tumanyan, N., Geyer, M., Bagon, S., & Dekel, T. (2022). Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation. ArXiv, abs/2211.12572.\n[2] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. ArXiv, abs/2208.12242.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks reviewer for the appreciation on our motivation, paper writing, technical contribution and results.   We will explain also the concerns as follows:\n\n> Comparison with PnP+Dreambooth. \n\nWe have attached a comprehensive human evaluation both here and in Table 1 from the pdf. We could see that Photoswap outperforms PnP by a large margin across all metrics.\nPnP was designed for changing the image style, where background details preservation is not the focus. PnP utilizes the text prompt to inject the style information while copying intermediate variables in the self-attention layer to keep the layout unchanged. However, in our task, the background is supposed to be the same while the source subject should be swapped into the target subject. To keep the background unchanged, the only difference between the source prompt and the target prompt in Photoswap is the subject name. Our training-free mechanism is also designed for keeping the background details while swapping the target.\n\n| Metrics      | Photoswap    | PnP+DreamBooth   | Tie |\n| :----        |    :-----:   |          :-----:|   ----:|\n| Subject Identity Preservation      |**52.7%**| 22.1%   | 25.2% |\n| Background Preservation   |     **49.1%**   | 20.7%      |  30.2%|\n| Overall Swapping Quality   |   **55.1%**     | 22.4%| 22.5%   |\n\n> Evaluation metric could be improved:\n\nWe shared the same opinion that human evaluation is important since this is a new task. We have conducted a larger-scale human evaluation and the results are attached in the pdf. We also believe subject preservation is important, and one metric used for human evaluation is subject identity recognition. Our Photoswap outperforms all baselines by a large margin. We also plan to add more metrics for evaluation given more time for the revision. \n\n> Could it swap between dissimilar subjects?\n\nYes. Photoswap does not need explicit mask of the source subject. The attention swapping in the Unet layer could be between any two shapes of subjects.  \n\n\n> Contributions\n\nAs you and Reviewer z7Bi mentioned, The task we undertook is inherently challenging, yet we have managed to achieve noteworthy results. Besides, we want to highlight that our simple training-free structure achieved amazing results without complicated structure. Our method could easily combined with other structure such as LoRA or ControlNet. As is well illustrated in the P2P comparison in the paper, and P2P, MasaCtrl, PnP comparison in this rebuttal, our model achieves a much better performance than all baselines across all metrics. We further analyzed the effectiveness of attention map and attention output, which provide insight for training-free image editing work.\n\n\n> How does the method perform when swapping has large domain gap?\n\nFundamentally, our method does not require similarity between source domain and target domain since we directly swap the attention variables. In Figure 8 in the paper, we show it works when swapping between portrait and painting.\n   \n> Does the structure of the subject in the reference image fix the structure of the inserted subject?\n\nNo, the structure of the subject is not rigidly defined. Once the identity of the subject is learnt, it adapts to the gesture of the source subject. As exemplified in the teaser image of our paper, the human faces or subjects in the resultant images align with the source image, instead of merely duplicating the reference image.\n\n"}, {"review_id": "sibmEvpRWi", "reviewer": "Reviewer_p4oH", "summary": "This paper introduces a new method for inserting a subject into a target image. The approach consists of 1) using dreambooth to extract the appearance information of the subject, and 2) copy the attention from the target image (obtained by regenerate the target image using DDIM inversion) to control the layout for the generated image. Empirical results show that the proposed method is preferred by raters ~50% of the time compared with P2P baseline. ", "questions": "Please refer to the weakness section. ", "limitations": "This paper does not discuss the limitation. While there's a section for ethical issues, the section doesn't provide useful discussion.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The proposed method is straightforward, and the empirical results look promising. ", "weaknesses": "* Technical contribution not clear. As mentioned in the paper, the two main components used in the paper, i.e. dreambooth and attention copying, have been proposed in prior works. It is unclear what the main technical contribution of this work is. In fact, it is even not clear, based on the limited description, what the main difference between the proposed method and the baseline is.\n\n\n* Unclear presentation. The approach section is hard to follow. In particular, Sec. 4.1 and 4.2 are distracting and it is unclear what information these two sections try to convey. The experiment section does not provide sufficient information for reproducing and understanding the implications of the experiments, e.g. what are the data, how were the baseline implemented, what the data and instructions are for the user study?\n\n\n* Limited experiments. While the authors claim that one of the main contributions of this work is extensive experiments, the actual evaluation is very limited. For a reasonable research paper, I will expect more solid and extensive evaluation regarding different aspects of the proposed approach, including but not limited to the success rate of the proposed approach, the effect of target image content and subject image, the effect of meta parameters, etc. From the experiments in the paper, it is even unclear how applicable the proposed method is in practice.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for your insightful suggestions. Here are our response to your concerns.\n\n> What is technical contribution. What is the main difference between the proposed method and the baseline. \n\nQuote Reviewer z7Bi, “The task of personalized subject swapping in images is fancy and interesting. This is the first work that could handle such challenging swapping task.” Such a task requires seamlessly integrating a new subjects into existing images in the exect position of the source subject. While previous methods are usually restricted to global editing, we achieved great results with logical framework, Photoswap. Note that Photoswap is a framework, which could incorporate more methods and components. For example, one could easily add LoRA weights into Photoswap or add controlnet layer to achieve fine-grained swapping. \nThe primary distinction between our approach and the baseline centers around the attention swapping process. In P2P, the authors predominantly use the cross-attention map for image editing, having identified a strong correlation between this map and the resultant image. In contrast, our technique employs attention swapping on the cross-attention map, self-attention map, and self-attention output, allowing for more detailed adjustments. As showcased in Figure 7 of our paper and the attached pdf file, PhotoSwap excels in both subject swapping and background conservation. The human evaluation results in Table 1 show that Photoswap outperforms the baseline by a large margin.\n\n> Unclear presentation in Section 4\n\nIn Section 4, we present the framework of Photoswap. Photoswap first learns a concept image into a token by a concept learning method, as in section 4.1. The training-free attention swapping process further keeps the background information unchanged during the target image generation process, and the learned subject is injected into the image through textual prompt as in section 4.2. Please refer to Algorithm 1 in the paper for a detailed process. We will also release the code for performance reproduction.\n\n> What are the data? How was the baseline implemented? What the data and instructions are for the user study?\n\nIn terms of concept source, for non-human, we utilized the images from the DreamBooth benchmark as our concept learning method is based on DreamBooth. For human, we collect celebrity images from the Internet. All the source images are also from the Internet. For the baseline, we use DreamBooth to transform a new concept into a textual token and then employ P2P as the attention-based process for subject swapping. The data used for the user study was swapping results on randomly sampled synthetic images and real images from the Internet. The user instructions and our user study interface are also attached in the pdf.\nWe will add those details to the next version. Please let us know if there are any more details missing in your mind.\n\n> Limited experiments \n\nIn the paper, we have tested Photoswap widely from synthetic images to real images, from human images to non-human subjects. We conducted experiments on complicated situations such as multiple subjects and occluded subjects. Since this is a new task, we built a baseline and did both human evaluation and qualitative comparison. We also conducted ablation study on the effect of different hyperparameters on human faces and common subjects. To validate the generalization of our framework, we also test that on other concept learning methods. We also did experiments to test the potential ethical concerns. Lastly, we also discussed the failure cases. In this rebuttal period, we further conducted a large-scale human evaluation on more baselines.\n\n> Success rate. Effect of target image content and subject image. Effect of meta parameters. From the How applicable the proposed method is in practice.\n\nIn our study, we assessed Photoswap's capability for subject swapping within images, a challenge that had not been effectively addressed before. Whether transitioning from real to synthetic images, everyday subjects to human faces, or artistic creations to film stills, we have illustrated Photoswap's proficiency in seamlessly substituting a subject from one image with a personalized subject from another. Both in the original paper and throughout this rebuttal, we've noted a commendably high success rate (exceeding 50%) in terms of users identifying the context from the source image and recognizing the identity from the reference image. The effects of the three hyperparameters, which influence the swapping steps, are detailed in the appendix. We will make our source code available for further examination of Photoswap's applicability. \nWe would be happy to address them if the reviewer has more specific suggestions or concerns during the discussion phase.\n\n"}, {"review_id": "tfzdGIyhrk", "reviewer": "Reviewer_Ep7B", "summary": "This paper presents a combination of DDIM inversion and Dreambooth to customize the appearance of existing images.\n\nThe authors also experimented with some attention layer hacking.", "questions": "Why we mainly conduct the comparison against prompt-to-prompt? Shouldn’t we mainly consider frameworks like imagic/masactrl to edit real image?", "limitations": "see Weaknesses", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tPaper is easy to read.\n2.\tFramework is reasonable. Dreambooth+DDIM inversion+Attention hacking will really lead to such results.\n3.\tEasy to reproduce. \n", "weaknesses": "1.\tMy main concern is the novelty: it seems that everything of in the combo of “Dreambooth+DDIM inversion+Attention hacking” are proposed and/or extensively discussed in previous works.  However, we are also aware of that many recent papers in this direction (like masactrl) are non-peer-reviewed works so that I am not leaning negative because of the novelty. Nevertheless, the combo of these components still looks a bit ad-hoc to me.\n2.\tThe experiments need some improvements: reference guided diffusion is sensitive to inputs. We should present random non-cherry-picked samples to study the method performance. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the suggestions, we will explain the concerns as follows:\n\n> About the paper novelty. \n\na.  Personalized subject swapping is an emerging vision task that has abundant user applications in practice.  The task we undertook is inherently challenging, due to the lack of training pairs. In the literature, there are few works that can address the problem in a unified way.    In contrast, as the results are shown in the framework, the proposed method is robust and can be generalized to different domains including human faces, animals, daily objects, vector art, and paintings.\n\nb.  We proposed a unified framework to address the problem.  The attention map manipulation process is non-trivial.  In section 4, all the steps in our method are based the observation of attention map analysis.  Without Photoswap,  simply applying existing attention map manipulation methods will not works well, as they could not preserve the non-swapped region well.   We use DreamBooth and DDIM inversion for implementation simplicity. Other concept learning (Lora, Suti) methods and inversion methods will also be compatible with our Photoswap framework.  \n\n\n> Why not compare to Masactrl?\n\nMasaCtrl is a non-peer-reviewed work, and it is released on arXiv just a month before the NeurIPS submission. But we still cited and discussed it when we noticed the paper before submission. Our paper is targeting a fundamentally different task, personalized subject swapping, while their method is for subject gesture changing.\n\nNonetheless, we have presented a comparison between Photoswap and MasaCtrl during this rebuttal stage, which is shown in Table 1 in the attached PDF and will be incorporated into our next version. We reproduced the results from MasaCtrl and acknowledged its commendable performance in altering subject gestures. However, it has notably inferior outcomes in the personalized subject swapping task. Through abundant human evaluation, Photoswap outperforms MasaCtrl across all metrics. \nUpon closer inspection, we observed that MasaCtrl primarily swaps the q vector in self-attention, having determined a strong correlation between the q vector and the subject shape. In our targeted task, the complete swapping of the source subject to the target subject necessitates a more comprehensive approach. Both the attention map—derived from the interaction of q and k—and the attention output—resulting from q, k, and v—is crucial.\n\n| Metrics      | Photoswap    | MasaCtrl+DreamBooth   | Tie |\n| :----        |    :-----:   |          :-----:|   ----:|\n| Subject Identity Preservation      |**79.1%**| 10.3%   | 10.6% |\n| Background Preservation   |     **72.8%**   | 10.2%      |  17.0%|\n| Overall Swapping Quality   |   **83.3%**     | 10.3%| 6.4%   |\n\n> A combo of methods\n\nPersonalized subject swapping is an interesting and challenging task, with our logical framework, Photoswap, we successfully addressed this complex challenge. It's important to emphasize that Photoswap is a versatile framework that can integrate various methodological elements. For instance, one can seamlessly incorporate LoRA weights into Photoswap or introduce a controlnet layer for more refined swapping. The task we embarked upon is fundamentally challenging, but we secured significant outcomes. It's essential to emphasize that a direct combination of DreamBooth with the earlier proposed attention map insertion is not optimal, as clearly demonstrated in the paper's P2P comparison and the rebuttal's P2P comparisons. Unlike prior approaches, we performed attention swapping on a broader set of attention variables. We also delved deeper into the efficacy of these variables, offering valuable insights for future investigations.\n\n\n> Improvements for experiments.\n\nAs is exhibited in the paper, Photoswap is a subject swapping model for general images, including real images and synthetic images, the target ranges from human faces to common house items, from daily life images, movie pictures, to artistic works. During the human evaluation, we did 2000 comparisons on both real images and synthetic images. All the swapping results used for evaluation are non-cherry pick results. We will also provide code for performance reproduction."}]

Author rebuttal:
Attached pdf

Meta-review: Reviewers and the AC read the rebuttal and took that into consideration for their final recommendation. The AC believes that the answers to the unresponsive reviewer in the rebuttal reasonably addressed their concerns. Many suggestions for improving the exposition, the baseline comparisons, and the evaluations were discussed with reviewers in the rebuttal. They should be included in the camera-ready paper.
Decision: Accept (poster)