SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jing Gu<sup>1</sup> Yilin Wang<sup>2</sup> Nanxuan Zhao<sup>2</sup> Tsu-Jui Fu<sup>3</sup> Wei Xiong<sup>2</sup> Qing Liu<sup>2</sup> Zhifei Zhang<sup>2</sup> He Zhang<sup>2</sup> Jianming Zhang<sup>2</sup> HyunJoon Jung<sup>2</sup> Xin Eric Wang<sup>1*</sup>
<sup>1</sup>University of California, Santa Cruz  
<sup>2</sup>Adobe <sup>3</sup>University of California, Santa Barbara
https://photoswap.github.io/
Figure 1: Photoswap can effortlessly replace the subject in a source image, which could be either synthetic (first two rows) or real (bottom row), with a personalized subject specified in reference images, while preserving the original subject pose and the composition of the source image.
In an era where images and visual content dominate our digital landscape, the ability to manipulate and personalize these images has become a necessity. Envision seamlessly substituting a tabby cat lounging on a sunlit window sill in a photograph with your own playful puppy, all while preserving the original charm and composition of the image. We present Photoswap, a novel approach that enables this immersive image editing experience through personalized subject swapping in existing images. Photoswap first learns the visual concept of the subject from reference images and then swaps it into the target image using pre-trained diffusion models in a training-free manner. We establish that a well-conceptualized visual subject can be seamlessly transferred to any image with appropriate self-attention and cross-attention manipulation, maintaining the pose of the swapped subject and the overall coherence of the image. Comprehensive experiments underscore the efficacy and controllability of Photoswap in personalized subject swapping. Furthermore, Photoswap significantly outperforms baseline methods in human ratings across subject swapping, background preservation, and overall quality, revealing its vast application potential, from entertainment to professional editing.
Imagine a digital world where the boundaries of reality and creativity blur, where a photograph of a tabby cat lounging on a sunlit window sill can effortlessly be transformed to feature your playful puppy in the same pose. Or envision yourself as a part of a famous movie scene, replaced seamlessly with the original character while preserving the very essence and composition of the scene. Can we achieve this level of personalized image editing, not just with expert-level photo manipulation skills, but in an automated, user-friendly manner? This question lies at the heart of personalized subject swapping, the challenging task of replacing the subject in an image with a user-specified subject, while maintaining the integrity of the original pose and composition. It opens up a plethora of applications in areas such as entertainment, advertising, and professional editing.
Personalized subject swapping is a complex undertaking that comes with its own set of challenges. The task requires a profound comprehension of the visual concept inherent to both the original subject and the replacement subject. Simultaneously, it demands the seamless integration of the new subject into the existing image. One of the critical objectives in subject swapping is to preserve the similar pose of the replacement subject. It is crucial that the swapped subject seamlessly fits into the original pose and scene, creating a natural and harmonious visual composition. This necessitates careful consideration of factors such as lighting conditions, perspective, and overall aesthetic coherence. By effectively blending the replacement subject with these elements, the final image maintains a sense of continuity and authenticity.
Existing image editing methods fall short in addressing these challenges. Many of these techniques are restricted to global editing and lack the finesse needed to seamlessly integrate new subjects into existing images. For example, for most text-to-image (T2I) models, a slightly prompt change could lead to a totally different image. Recent works (Nichol et al., 2022; Meng et al., 2022; Couairon et al., 2022; Cao et al., 2023; Zhang et al., 2023) allow user to control the generation with an additional input such as user brush, semantic layout, or sketches. However, it is still challenging to guide the generation process to follow users' intent on the generation of object shape, texture, and identity. Other approaches (Hertz et al., 2022; Tumanyan et al., 2023; Mokady et al., 2023) have explored the potential of using text prompts to edit image content in the context of synthetic image generation. Despite showing promise, these methods are not yet fully equipped to handle the intricate task of swapping subjects in existing images with user-specified subjects.
Therefore, we present Photoswap, a novel framework that leverages pre-trained diffusion models for personalized subject swapping in images. In our approach, the diffusion model learns to represent the concept of the subject  $(O_{t})$ . Then the representative attention map and attention output saved in the source image generation process will be transferred into the generation process of the target image to generate the new subject while keeping non-subject pixels unchanged. Our extensive experiments and evaluations demonstrate the effectiveness of Photoswap. Not only does our method enable the seamless swapping of subjects in images, but it also maintains the pose of the swapped subject and the overall coherence of the image. Remarkably, Photoswap outperforms baseline methods by a large margin in human evaluations of subject identity preservation, background preservation, and overall quality of the swapping (e.g.,  $37.3\%$  vs.  $27.1\%$  in terms of overall quality). The contributions of this work are as follows: 1) We present a new framework for personalized subject swapping in images. 2) We propose a training-free attention swapping method that governs the editing process. 3) The efficacy of our proposed framework is demonstrated through extensive experiments including human evaluation.
2.1 Text-to-Image Generation
In the early stages of text-based image generation, Generative Adversarial Networks (GANs) (Goodfellow et al., 2020; Brock et al., 2018; Karras et al., 2019) were widely used due to their exceptional ability to produce high-quality images. These models aimed to align textual descriptions with synthesized images through multi-modal vision-language learning, achieving impressive results on specific domains (e.g., bird, chair and human face). When combined with CLIP (Radford et al., 2021), a large pre-trained model that learns visual-textual representations from millions of caption-image pairs, GAN models (Crowson et al., 2022) have demonstrated promising outcomes in cross-domain text-to-image (T2I) generation. Recently, T2I generation has seen remarkable progress with autoregressive (OpenAI, 2021; Ding et al., 2021) and diffusion models (Nichol et al., 2022; Gu et al.,
2022; OpenAI, 2022; Sahara et al., 2022), offering diverse outcomes and can synthesize high-quality images closely aligned with textual descriptions in arbitrary domains.
Rather than focusing on T2I generation tasks without any constraints, subject-driven T2I generation (Nitzan et al., 2022; Casanova et al., 2021; Ruiz et al., 2023) requires the model to identify the specific object from a set of visual examples and synthesize novel scenes incorporating them based on the input text prompts. Building upon modern diffusion techniques, recent approaches such as Dream-Booth (Ruiz et al., 2023) and Textual Inversion (Gal et al., 2023a,b; Kumari et al., 2023; Mokady et al., 2023) learn to invert special tokens from a given set of images. By combining these tokens with text prompts, they generate personalized unseen images. To improve data efficiency, retrieval augmentation techniques (Sheynin et al., 2023; Blattmann et al., 2022; Chen et al., 2023) leverages external knowledge bases to overcome limitations posed by rare entities, resulting in visually relevant appearances and enhanced personalization. In our work, we aim to tackle personalized subject swapping, not only preserving the identity of subjects in reference images, but also maintaining the context of the source image.
2.2 Text-guided Image Editing
Text-guided image editing manipulates an existing image based on the input textual instructions, while preserving certain aspects or characteristics of the original image. Early works based on GAN models (Karras et al., 2019) only limits to a certain object domain. Diffusion-based methods (Zhang et al., 2023; Nichol et al., 2022; Feng et al., 2023) break this barrier and support text-guided image editing. Though these methods generate stunning results, many of them suffer from conducting local editing, and additional manual masks (Meng et al., 2022; Zeng et al., 2023; Meng et al., 2022) are required to constrain the editing regions, which is often tedious to draw. By employing cross-attention (Hertz et al., 2022) or spatial characteristics (Tumanyan et al., 2023), the local editing can be achieved but struggles with non-rigid transformations (e.g., changing pose) and retaining the original image layout structure. While Imagic (Kawar et al., 2023) addresses the need for non-rigid transformations by fine-tuning a pre-trained diffusion model to capture image-specific appearances, it requires test-time finetuning, which is not time-efficient for deployment. Moreover, relying solely on text as input lacks precise control. In contrast, we propose a novel training-free attention swapping scheme that enables precise personalization based on reference images, without the need for time-consuming finetuning.
2.3 Exemplar-guided Image Editing
Exemplar-guided image editing covers a broad range of applications, and most of the works (Wang et al., 2019; Huang et al., 2018; Zhou et al., 2021) can be categorized as exemplar-based image translation tasks, conditioning on various information, such as stylized images (Liu et al., 2021; Deng et al., 2022; Zhang et al., 2022), layouts (Yang et al., 2023b; Li et al., 2023c; Jahn et al., 2021), skeletons (Li et al., 2023c), sketches/edges (Seo et al., 2023). With the convenience of stylized images, image style transfer (Liao et al., 2017; Zhang et al., 2020) receives extensive attention, replying on methods to build a dense correspondence between input and reference images, but it cannot deal with local editing. To achieve local editing with non-rigid transformation, conditions like bounding boxes and skeletons are introduced, but require drawing efforts from users, which sometimes are hard to obtain. A recent work (Yang et al., 2023a) poses exemplar-guided image editing task as an inpainting task with the mask and transfers the semantic content from the reference image to the source one, with the context intact. Unlike these works, we propose a more user-friendly scenario by conducting personalized subject swapping with only reference images and obtain high-quality editing results. DreamEdit (Li et al., 2023b) uses iterative inpainting to achieve subject replacement. Nevertheless, the existing approach fails to establish a comprehensive correlation between the source and target subjects. Conversely, our technique ensures that pivotal attributes, such as body gestures and facial expressions, remain unaltered.
Diffusion models are a type of generative model that operates probabilistically. In this process, an image is created by gradually eliminating noise from the target that is characterized by Gaussian noise. In the context of text-to-image generation, a diffusion model typically involves a process where an initial random image is gradually refined step by step, with each step guided by a learned model, until it becomes a realistic image. The changes to the image spread out and affect many pixels over
Figure 2: The Photoswap framework. Given several images of a new concept, the diffusion model first learns the concept and converts it into a token. The upper part is the generation process of the source image, while the bottom part is the generation process of target image. The initial noise feature  $z_{T}^{t}$  is copied from  $z_{T}^{s}$  of the source. The attention output and attention map in the source image generation process would be transferred to the target image generation process. The final feature  $z_{0}^{t}$  is decoded to output the target image. Refer to Sec. 4 for more details.
time. Given an initial random noise  $\mathbf{z}_T \sim \mathcal{N}(0, \mathbf{I})$ , the diffusion model gradually denoise  $\mathbf{z}_t$ , which gives  $\mathbf{z}_{t-1}$ .
Diffusion models are probabilistic generative models that learn to generate images by simulating a random process called a diffusion process. In the image generation process, the diffusion model gradually predicts the noise at the current diffusion step and denoises to get the final image. In this study, we utilize a pre-trained text-to-image diffusion model, Stable Diffusion (Rombach et al., 2022), which encodes the image into latent space and gradually denoises the latent variable to generate a new image. Stable Diffusion is based on a U-Net architecture (Ronneberger et al., 2015), which generates latent variable  $z_{t - 1}$  conditioned on a given text prompt  $P$  and the latent variable  $z_{t}$  from the previous step  $t$ :
$$
\boldsymbol {z} _ {t - 1} = \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, P, t\right) \tag {1}
$$
The U-Net consists of layers that include repetition of self-attention and cross-attention blocks. This study focuses on manipulating self-attention and cross-attention to achieve the task of personalized subject swapping.
4 The Photoswap Method
Providing a few reference images of a personalized target subject  $O_{t}$ , Photoswap can seamlessly swap it with another subject  $O_{s}$  in a given source image  $I_{s}$ . The Photoswap pipeline is illustrated in Figure 2. To learn the visual concept of the target subject  $O_{t}$ , we fine-tune a diffusion model with reference images and do object inversion to represent  $O_{t}$  using special token *. Then, to substitute the subject in the source image, we first obtain the noise  $z_{T}^{1}$  that can be used to reconstruct the source image  $I_{s}$ . Next, through the U-Net, we obtain the needed feature map and attention output in the self-attention and cross-attention layers, including  $M$ ,  $A$ , and  $\phi$  (which we will introduce in Sec. 4.2). Finally, during the target image generation process that is conditioned on the noise  $z_{T}$  and the target text prompt  $P_{t}$ , in the first  $\lambda$  steps, those intermediate variables  $(M, A,$  and  $\phi)$  would be replaced with corresponding ones obtained during the source image generation process. In the last
For a synthetic image,  $z_{T}^{*}$  is the initial noise used to generate it. For a real image, we utilize an improved version of DDIM inversion (Song et al., 2020) to get the initial noise and re-generate the source image. See Sec. 5.1 for details.
Figure 3: SVD visualization of self-attention maps. Each image's attention map is resized to  $64 \times 64$  at every layer, and we calculate the average map across all layers for all diffusion time steps. Most significant components are extracted with SVD and visualized. Remarkably, the visualized results demonstrate a strong correlation with the layout of the generated image. The top two rows are visualization about synthetic images while the bottom two rows are about real images.
$(T - \lambda)$  steps, no attention swapping is needed and we can continue the denoising process as usual to obtain the final resulting image. Sec. 4.1 discusses the visual concept learning technique we used, and Sec. 4.2 details the training-free attention swapping method for controllable subject swapping.
4.1 Visual Concept Learning
Subject swapping requires a thorough understanding of the subject's identity and specific characteristics. This knowledge enables the creation of accurate representations that align with the source subject. The subject's identity influences the composition and perspective of the image, including its shape, proportions, and textures, which affect the overall arrangement of elements. However, existing diffusion models lack information about the target subject  $(O_{t})$  in their weights because the training data for text-to-image generation models does not include personalized subjects. To overcome this limitation and generate visually consistent variations of subjects from a given reference set, we need to personalize text-to-image diffusion models accurately. Recent advancements have introduced various methods, such as fine-tuning the diffusion model with distinct tokens associated with specific subjects, to achieve this "personalization" (Gal et al., 2023a; Ruiz et al., 2023; Kumari et al., 2023). In our experiments, we primarily utilize DreamBooth (Ruiz et al., 2023) as a visual concept learning method. It's worth noting that alternative concept learning methods can also be effectively employed with our framework.
4.2 Controllable Subject Swapping via Training-free Attention Swapping
Subject swapping poses intriguing challenges, requiring the maintenance of the source image's spatial layout and geometry while integrating a new subject concept within the same pose. This necessitates preserving the critical features in the source latent variable, which encapsulates the source image information, and leveraging the influence of the target image text prompt  $P_{t}$ , which carries the concept token, to inject the new subject into the image.
The central role of the attention layer in orchestrating the generated image's layout has been well-established in prior works (Hertz et al., 2022; Cao et al., 2023; Tumanyan et al., 2023). To keep non-subject pixels intact, we orchestrate the generation of the target image  $I_{t}$  by transferring vital
Figure 4: Self-attention map visualization across diffusion time steps. This representation reveals that the layout of the generated image is intrinsically embedded in the self-attention map from the initial steps. Consequently, to assert control over the layout, it is imperative to commence the attention swap at the earliest stages of the process.
variables to the target image generation process. Here, we explore how distinct intermediate variables within the attention layer can contribute to a controllable generation in the context of subject swapping.
Within the source image generation process, we denote the cross-attention map as  $A_{i}^{s}$ , the self-attention map as  $M_{i}^{s}$ , the cross-attention output as  $\psi_{i}^{s}$ , and the self-attention output as  $\phi_{i}^{s}$ . The corresponding variables in the target image generation process are denoted as  $A_{i}^{t}$ ,  $M_{i}^{t}$ ,  $\psi_{i}^{t}$ ,  $\phi_{i}^{t}$ , where  $i$  represents the current diffusion step.
In the self-attention block, the latent feature  $z_{i}$  is projected into queries  $q_{i}$ , keys  $k_{i}$ , and values  $v_{i}$ . We obtain the self-attention block's output  $\phi_{i}$  using the following equation:
$$
\phi_ {i} = M _ {i} \boldsymbol {v} _ {i} \quad \text {w h e r e} \quad M _ {i} = \operatorname {S o f t m a x} \left(\boldsymbol {q} _ {i} \boldsymbol {k} _ {i} ^ {T}\right) \tag {2}
$$
where  $M_{i}$  is the self-attention map, and  $\phi_{i}$  is the feature output from the self-attention layer. The cross-attention block's output  $\psi_{i}$  is:
$$
\boldsymbol {\psi} _ {i} = \boldsymbol {A} _ {i} \boldsymbol {v} _ {i} \quad \text {w h e r e} \quad \boldsymbol {A} _ {i} = \operatorname {S o f t m a x} \left(\boldsymbol {q} _ {i} \boldsymbol {k} _ {i} ^ {T}\right) \tag {3}
$$
where  $A_{i}$  is the cross-attention map. In both self-attention and cross-attention, the attention map  $M_{i}$  and  $A_{i}$  are correlated to the similarity between  $q_{i}$  and  $k_{i}$ , acting as weights that dictate the combination of information in  $v_{i}$ . In this work, the manipulation of the diffusion model focuses on self-attention and cross-attention within U-Net, specifically, swapping  $\phi$ ,  $M$ , and  $A$ , while keeping  $\psi$  unchanged.
Self-attention map  $M$ , as it calculates the similarity within spatial features after linear projection, plays a pivotal role in governing spatial content during the generation process. As visualized in Figure 3, we capture  $M$  during the image generation and highlight the leading components via Singular Value Decomposition (SVD). This visualization reveals a high correlation between  $M$  and the geometry and content of the generated image. Further, when visualizing the full steps of the diffusion process (Figure 4), we discern that the layout information is mirrored in the self-attention from the initial steps. This insight underscores the necessity of initiating the swap early on to prevent the emergence of a new, inherent layout.
Cross-attention map  $\mathbf{A}$  is determined by both latent variable and text prompt, as in Equation 3, and  $A_{i}^{s}v$  can be viewed as a weighted sum of the information from a text prompt. Copying  $A_{i}^{s}$  to  $A_{i}^{t}$  during the target image generation process improves the layout alignment between the source image and the target image.
Self-attention output  $\phi$ , derived from the self-attention layer, encapsulates rich content information from the source image, independent of direct computation with textual features. Hence, replacing  $\phi_i^t$  with  $\phi_i^s$  enhances the preservation of context and composition from the original image. Our observations indicate that  $\phi$  exerts a more profound impact on the image layout than the cross-attention map  $A$ .
Diffusion Steps
Algorithm 1 The Photoswap Algorithm
Inputs: source image  $I_{s}$  , reference images  $O_{t}$  , source image text prompt  $P_{s}$  , target image text prompt  $P_{t}$  , diffusion model  $\theta$ $\theta^{*}\gets \theta ,O_{t}\triangleright$  Finetune diffusion model to include the new concept   
 $z_{T}^{s}\gets DDIMInversion(ImageEncoder(I_{s}),P_{s})\triangleright$  Using DDIM to guarantee re-construction   
 $z_{T}^{t}\gets z_{T}^{s}\triangleright$  Using the same starting noise   
for  $i = T,T - 1,\dots,1$  do   
 $\epsilon^s,\phi_i^s,M_i^s,A_i^s\gets \epsilon_\theta^* (z_i^s,P_s,i)\triangleright$  Denoise to get the attention output and map for source image   
 $\phi_i^t,M_i^t,A_i^t\gets \epsilon_\theta^* (z_i^t,P_t,i)\triangleright$  Denoise to get the attention output and map for target image   
 $\phi_i^*,M_i^*,A_i^*\gets SWAP(\phi_i^s,M_i^s,A_i^s,\phi_i^t,M_i^t,A_i^t,i)$ $\epsilon^{*}\gets \epsilon_{\theta^{*}}(z_{i}^{t},P_{t},i,\phi_{i}^{*},M_{i}^{*},A_{i}^{*})\triangleright$  Denoise the updated attention map and output   
 $z_{i - 1}^{s}\gets DDIMSampler(z_{i}^{s},\epsilon^{s})\triangleright$  Sample next latent variable for source image   
 $z_{i - 1}^{t}\gets DDIMSampler(z_{i}^{t},\epsilon^{*})\triangleright$  Sample next latent variable for source image   
end for   
 $I_{t} = ImageDecoder(z_{0}^{t})$    
return  $I_{t}$    
function SWAP  $(\phi^s,M^s,A^s,\phi^t,M^t,A^t,i)$ $\phi^{*}\gets (i <   \lambda_{\phi})? \phi^{s}: \phi^{t}\triangleright$  Control self-attention feature swap   
 $M^{*}\gets (i <   \lambda_{M})?M^{s}:M^{t}\triangleright$  Control self-attention Map swap   
 $A^{*}\gets (i <   \lambda_{A})?A^{s}:A^{t}\triangleright$  Control cross-attention map swap   
return  $\phi^{*},M^{*},A^{*}$    
end function
Cross-attention output  $\psi$ , emanating from the cross-attention layer, embodies the visual concept of the target subject. It is vital to note that substituting cross-attention output  $\psi_i^s$  with  $\psi_i^t$  would obliterate all information from the target text prompt  $P_t$ , as illustrated in Equation 3. Given that  $k_i^t$  and  $v_i^t$  are projections of target prompt embeddings, we retain  $\psi_i^s$  unchanged to safeguard the target subject's identity.
Algorithm 1 provides the pseudo-code of our full Photoswap algorithm.