SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the implementation of subject swapping on real images, we require an additional process that utilizes an image inversion method, specifically the DDIM inversion (Song et al., 2020), to transform the image into initial noise. This inversion method relies on a reversed sequence of sampling to achieve the desired inversion. However, there exist inherent challenges when this inversion process is applied in text-guided synthesis within a classifier-free guidance setting. Notably, the inversion can potentially amplify the accumulated error, which could ultimately lead to subpar reconstruction outcomes. To fortify the robustness of the DDIM inversion and to mitigate this issue, we further optimize the null text embedding, as detailed in Mokady et al. (2023). The incorporation of this optimization technique bolsters the effectiveness and reliability of the inversion process, consequently allowing for a more precise reconstruction. Without further notice, the DDIM inversion in this paper is enhanced by null text embedding optimization.
During inference, we utilize the DDIM sampling method with 50 denoising steps and classifier-free guidance of 7.5. The default step  $\lambda_{A}$  for cross-attention map replacement is 20. The default step  $\lambda_{M}$  for self-attention map replacement is 25, while the default step for self-attention feature  $\lambda_{\phi}$  replacement is 10. Please refer to Appendix A for analysis on attention swapping step, and Appendix E for detailed Unet layer swapping. Note that the replacement steps may change to some specific checkpoint. As mentioned in Sec. 4, the target prompt  $P_{t}$  is just source prompt  $P_{s}$  with the object token being replaced with the new concept token. For concept learning, we mainly utilize DreamBooth (Ruiz et al., 2023) to finetune a stable diffusion 2.1 to learn the new concept from 35 images. The learning rate is set to 1e-6. We use Adawm optimizer with 800 hundred training steps.
Figure 5: Photoswap results across various object and image domains, demonstrating its wide applicability. From everyday objects to cartoons, the diversity in subject swapping tasks has showcased the versatility and robustness of our framework across different contexts.
Photoswap
We finetune both the U-net and text encoder. The DreamBooth training takes around 10 minutes on a machine with 8 A100 GPU cards.
5.2 Personalized Subject Swapping Results
Figure 5 showcases the effectiveness of our Photoswap technique for subject swapping. Our approach excels at preserving crucial aspects such as spatial layout, geometry, and the pose of the original subject while seamlessly introducing a reference subject into the target image. Remarkably, even in cartoon images, our method ensures that the background remains intact during the subject change process. A notable example is the "cat" image, where our technique successfully retains all the intricate details from the source image, including the distinctive "Whiskers." This demonstrates our framework's ability to accurately capture and preserve fine-grained information during subject swapping. Please refer to Appendix C.
We further demonstrate the versatility of Photoswap by showcasing its effectiveness in multiple subject swap and occluded object swap scenarios. As depicted in Figure 6a, we present a source image featuring two sunglasses, which are successfully replaced with reference glass while preserving the original layout of the sunglasses. Similarly, in Figure 6b, we observe a source image with a dog partially occluded by a suit. The resulting swapped dog wears a suit that closely matches the occluded region. These examples serve to highlight the robustness of our proposed Photoswap method in handling various real-world cases, thereby enabling users to explore a broader range of editing possibilities. Refer Appendix F for identity control.
5.3 Comparison with Baseline Methods
Personalized object swap is a new task and there is no existing benchmark. However, we could modify the existing attention manipulation based methods. More specifically, we used the same concept learning method DreamBooth to finetune the same stable diffusion checkpoint to inject the new concept. To fairly compare with our results, we modified the existing prompt-based editing method P2P (Hertz et al., 2022), an editing method based on diffusion models. Note that origin P2P only works on a pair of synthetic images, in our setting we use the same concept learning DreamBooth and fix the seed to allow concept swapping. On the other hand, PnP (Tumanyan et al., 2023) could
(a) Multi-subject swap.
(b) Occluded subject swap.
Figure 6: Photoswap results on multi-subject and occluded subject scenarios. The results show that Photoswap can disentangle and replace multiple subjects at once. Also, Photoswap can identify the target object while avoiding influencing the non-subject pixels.
Figure 7: Qualitative comparison between P2P+DreamBooth and Photoswap. We can observe that P2P+DreamBooth is capable of achieving subject swapping. However, it faces challenges in preserving both the background and the reference subject accurately, while for Photoswap, it is robust to handle various cases.
also be implemented in a similar setting, however, we found PnP usually can not lead to satisfactory object swapping and may lead to a huge difference between the source image and the generated image. We suspect that it is because PnP is designed for image translation so it does not initiate the attention manipulation step from the beginning step. The qualitative comparison between Photoswap and  $\mathrm{P2P + DreamBooth}$  is shown in Figure 7. Check Appendix B for performance on other concept learning method. We observe that P2P with DreamBooth could achieve basic object swap, but it still suffers from background mismatching issues.
Human Evaluation. We conduct a human evaluation to study the editing quality by (1) Which result better swaps the subject as the reference and keeps its identity; (2) Which result better preserves the background; (3) Which result has better overall subject-driven swapping. We randomly sample examples and adopt Amazon MTurk² to compare between two results. Please refer to Appendix D for details. To avoid potential bias, we hired 3 Turkers for each sample. Table 1 demonstrates the comparison between our Photoswap and P2P. Firstly, more turkers  $(42.4\%)$  denote that our Photoswap better swaps the subject yet keeps its identity at the same time. Moreover, we can also preserve the background in the source image  $(39.3\%$  vs.  $30.2\%)$ , which is another crucial goal of this editing. In summary, Photoswap precisely performs subject swapping and preserves the remaining part from the input, leading to an overall superiority  $(37.3\%)$  to P2P  $(27.1\%)$ . Ku et al. (2023) shows Photoswap
2Amazon Mechanical Turk (MTurk): https://www.mturk.com.
Table 1: Qualitative comparison between Photoswap and P2P+DreamBooth, PnP+DreamBooth, MasaCtrl+DreamBooth separately. It includes 2000 comparisons on 400 image pairs, with 200 real images and 200 synthetic images. Each image pair contains 5 ratings from different human annotator in Amazon Turk. SS means Subject Swapping, BP means Background Preservation, and OQ means Overall Quality.
Photoswap	P2P	Tie	Photoswap	PnP	Tie	Photoswap	MasaCtrl	Tie
SS	42.4%	30.0%	27.6%	52.7%	22.1%	25.2%	79.1%	10.3%	10.6%
BP	39.3%	30.2%	30.5%	49.1%	20.7%	30.2%	72.8%	10.2%	17.0%
OQ	37.3%	27.1%	35.6%	55.1%	22.4%	22.5%	83.3%	10.3%	6.4%
achieve SOTA performance by comparing with DreamEdit (Li et al., 2023b) and BLIP-Diffusion (Li et al., 2023a).
5.4 Ethics Exploration
Figure 8: Results on real human face images across different races. Evidently, the skin colors are also successfully transferred when swapping a white person with a black person, and vice versa.
Like many AI technologies, text-to-image diffusion models can potentially exhibit biases reflective of those inherent in the training data (Sasha Luccioni et al., 2023; Perera and Patel, 2023). Given that these models are trained on vast text and image datasets, they might inadvertently learn and perpetuate biases, such as stereotypes and prejudices, found within this data. For instance, should the training data contain skewed representations or descriptions of specific demographic groups, the model may produce biased images in response to related prompts.
However, Photoswap has been designed to mitigate bias within the generation process of a text-to-image diffusion model. It achieves this by directly substituting the depicted subject with the intended target. In Figure 8, we present our evaluation of face swapping across various skin tones. It is crucial to note that when there is a significant disparity between the source and reference images, the swapping results tend to homogenize the skin color. As a result, we advocate for the use of Photoswap on subjects of similar racial backgrounds to achieve more satisfactory and authentic outcomes. Despite these potential disparities, the model ensures the preservation of most of the target subject's specific facial features, reinforcing the credibility and accuracy of the final image.
This paper introduces Photoswap, a novel framework designed for personalized subject swapping in images. To facilitate seamless subject photo swapping, we propose leveraging self-attention control by exchanging intermediate variables within the attention layer between the source image and reference images. Despite its simplicity, our extensive experimentation and evaluations provide compelling evidence for the effectiveness of Photoswap. Our framework offers a robust and intuitive solution for subject swapping, enabling users to effortlessly manipulate images according to their preferences. In the future, we plan to further advance the method to address those common failure issues to enhance the overall performance and versatility of personalized subject swapping.
Blattmann, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B. (2022). Retrieval-Augmented Diffusion Models. In NeurIPS.
Brock, A., Donahue, J., and Simonyan, K. (2018). Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations.
Zhang, L., Rao, A., and Agrawala, M. (2023). Adding conditional control to text-to-image diffusion models.
Zhang, P., Zhang, B., Chen, D., Yuan, L., and Wen, F. (2020). Cross-domain correspondence learning for exemplar-based image translation. In CVPR, pages 5143-5153.
Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., and Xu, C. (2022). Inversion-based creativity transfer with diffusion models. arXiv.
Zhou, X., Zhang, B., Zhang, T., Zhang, P., Bao, J., Chen, D., Zhang, Z., and Wen, F. (2021). Cocosnet v2: Full-resolution correspondence learning for image translation. In CVPR.
A Attention Swapping Step Analysis
In this section, we visualize the effect of the influence of swapping steps of different components. As discussed in the main paper, self-attention output  $\phi$ , and self-attention map  $M$ , derived from the self-attention layer, encompasses comprehensive content information from the source image, without relying on direct computation with textual features. Previous works such as Hertz et al. (2022) did not explore the usage of  $\phi$  and  $M$  in the object-level image editing process.
Attention map and attention output swap step
Changing  $\lambda_{\Phi}$ , steps of self-attention output swap
Changing  $\lambda_{\mathbb{M}}$  steps of self-attention map swap
Source Image
Changing  $\lambda_{\Lambda}$ , steps of cross-attention map swap
Figure 9: Results at different swapping steps. With consistent steps, swapping the self-attention output provides superior control over the layout, including the subject's gestures and the background details. However, excessive swapping could affect the subject's identity, as the new concept introduced through the text prompt might be overshadowed by the swapping of the attention output or attention map. This effect is more clear when swapping the self-attention output  $\lambda_{\phi}$ . Furthermore, we observed that replacing the attention map for an extensive number of steps can result in an image with significant noise, possibly due to a compatibility issue between the attention map and the  $v$  vector.
Figure 9 provides a visual representation of the effect of incrementally increasing the swapping step for one  $\lambda$  hyperparameter while maintaining the other two at zero. Although all of them can be utilized for subject swapping, they demonstrate varying levels of layout control. At the same swapping step, the self-attention output  $\phi$  offers more robust layout control, facilitating better alignment of gestures and preservation of background context. In contrast, the self-attention map  $M$  and cross-attention map  $A$  demonstrate similar capabilities in controlling the layout.
However, extensive swapping can affect the subject's identity, as the novel concept introduced via the text prompt might be eclipsed by the swapping of the attention output or attention map. This effect becomes particularly evident when swapping the self-attention output. This analysis further informs the determination of the default  $\lambda_{\phi}$ ,  $\lambda_{M}$ , and  $\lambda_{A}$  values. While the cross-attention map  $A$  facilitates more fine-grained generation control, given its incorporation of information from textual tokens, we discovered that  $\phi$  offers stronger holistic generation control, bolstering the overall output's quality and integrity.
B Results of Other Concept Learning Methods
We mainly use DreamBooth as the concept learning method in the experiments, primarily due to its superior capabilities in learning subject identities (Ruiz et al., 2023). However, our method is not strictly dependent on any specific concept learning method. In fact, other concept learning methods could be effectively employed to introduce the concept of the target subject.
To illustrate this, we present the results of Photoswap when applying Text Inversion (Gal et al., 2023a). We train the model using 8 A100 GPUs with a batch size of 4, a learning rate of 5e-4, and set the training steps to 1000. Results in Figure 10 indicate that Text Inversion also proves to be an effective concept learning method, as it successfully captures key features of the target object.
Figure 10: Results of Text Inversion (Gal et al., 2023a) as the concept learning module. It can successfully capture key subject features, but its performance drops when representing complex structures such as human faces.
Nevertheless, we observe that Text Inversion performance is notably underwhelming when applied to human faces. We postulate that this is because Text Inversion focuses on learning a new embedding for the novel concept, rather than finetuning the entire model. Consequently, the capacity to express the new concept becomes inherently limited, resulting in its less than optimal performance in certain areas.
C Failure Cases
Figure 11: Failure cases. The model sometimes struggles to accurately reconstruct hand details and complex background information such as formula on a whiteboard.
Here we highlight two common failure cases. First, the model struggles to accurately reproduce hands. When the subject includes hands and fingers, the swapping results often fail to precisely mirror the original hand gestures or the number of fingers. This issue could be an inherited challenge from Stable Diffusion. Moreover, Photoswap can encounter difficulties when the image comprises complex information. As illustrated in the lower row of Figure 11, Photoswap fails to reconstruct the complicated formula on a whiteboard. Therefore, while Photoswap exhibits strong performance across various scenarios, it's crucial to acknowledge these limitations when considering its application in real-world scenarios involving intricate hand gestures or complex abstract information.
For real images, we sourced all our images from internet searches. We employed the search prompt: 'a photo of <target>'. Here, the <target> variable could be a specific celebrity (e.g., 'Elon Musk') or a descriptive scene (e.g., 'a cute yellow cat running in the forest'). The celebrity names were identified through a Google search with the prompt "top celebrities 2023." For scene descriptions, we curated a list of 100 distinct search prompts to source images from the internet. In total, we aggregated 1,000 images using these prompts. All prompts, along with the collected image, will be made available in our next revision.
For synthetic images, we generated 1,000 images using text prompts with the text-to-image diffusion model version 2.1. These prompts spanned a range, including those centered on humans (e.g., "A photo of a woman looking left and smiling, Van Gogh style") and those focused on non-human subject (e.g., "An old car in the middle of the road, flanked by trees during autumn"). All prompts used in synthetic image generation will also be released too. For the human evaluation exhibited in this rebuttal and in the paper, we utilized the "random" package in Python to sample 200 images from both the real and synthetic datasets, respectively. Each image underwent evaluation by five distinct individuals on Amazon Turk. In total, this resulted in a comprehensive 6,000 ratings, as we compared our model against P2P, PnP, and MasaCtrl. Our findings unequivocally indicate that our model surpasses all other methods in performance.
Figure 12: Ablation results of  $M$  in Photoswap. With  $M$  value increase, the generated one is more similar to the style and identity of source image and dissimilar to the reference subject, and vice versa.
For source image processing, all we do is to resize the image into standard 512x512 pixels. It is worth noting that there is also no postprocessing needed for generated images.
E Unet Swapping Layer
There are 16 Unet layers in the Stable Diffusion backbone. We tested the effect of different layers according to their position (up, middle, or lower) in the Unet and their latent size. Through hundreds of experiments on the layer combination, we found that while the latent size of the layer plays a minor role, the position of the layer to be swapped matters. More specifically, we find the essential part is to do swapping operations in all the decoder layers in the Unet.
F Controlling Subject Identity
The effectiveness of the proposed mutual self-attention is demonstrated through both synthetic image synthesis and real image editing. Additionally, we perform an analysis of the control strategy with varying values of  $M$  during the denoising process. Figure 12 provides insights into this analysis. It is observed that when applying self-attention control with a large swapping step  $\lambda_{M}$  for  $M$ , the synthesized image closely resembles the source image in terms of both style and identity. In this scenario, all contents from the source image are preserved, while the subject style learned from the reference subject is disregarded. As the value of  $M$  decreases, the synthesized image maintains the subject from the reference image while retaining the layout and pose of the contents from the source image. This gradual transition in the control strategy allows for a balance between subject style transfer and preservation of the original image's contents.