FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images
Authors: Zheng Yu
Yaohua Wang
Siying Cui
Aixi Zhang
Wei-Long Zheng
Senzhang Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: diffusion model
personalization
image generation
Primary area: diffusion_based_models
TL;DR: A novel diffusion-driven facial parts swapping methods with multiple reference images.

Abstract: Facial parts swapping aims to selectively transfer regions of interest from the source image onto the target image while maintaining the rest of the target image unchanged.
Most studies on face swapping designed specifically for full-face swapping, are either unable or significantly limited when it comes to swapping individual facial parts, which hinders fine-grained and customized character designs.
However, designing such an approach specifically for facial parts swapping is challenged by a reasonable multiple reference feature fusion, which needs to be both efficient and effective.
To overcome this challenge, FuseAnyPart is proposed to facilitate the seamless "fuse-any-part" customization of the face.
In FuseAnyPart, facial parts from different people are assembled into a complete face in latent space within the Mask-based Fusion Module.
Subsequently, the consolidated feature is dispatched to the Addition-based Injection Module for
fusion within the UNet of the diffusion model to create novel characters.
Extensive experiments qualitatively and quantitatively validate the superiority and robustness of FuseAnyPart.
Source codes are available at https://github.com/Thomas-wyh/FuseAnyPart.

FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images
Zheng Yu*t Shanghai Jiao Tong University & Alibaba Group cs us-yuzheng@sjtee tu. edu.
Yaohua Wang
Alibaba Group
xiachen, wwwhaayaabaaaaaaaaaaaeaaaaaaaaaa ierlbaa-nnnccoom inc. com
Stying Cui
Aixi Zhang
Peking University & Alibaba Group cuisiying. csySalibaba inc.iin.... com
Alibaba Group
aixi whwxaallbabaa inc. com
Wei-Long Zheng Shanghai Jiao Tong University weilongosjtu edu. cn
Senzhang Wang
Central South University szwang@csu. edu. cn
Face background
mouth
eyes
nose
Figure I: Results of facial parts swapping using the proposed FuseAnyPart at 512 x 512 resolution. The swapped face (central image) is generated by fusing the original face (top-left image) with three facial part reference images (bottom-left, top-right, bottom-right). Notably, FuseAnyPart can seamlessly blend facial parts from multiple reference images with significant differences in appearance, producing high-fidelity and natural-looking swapped faces.
Abstract
Facial parts swapping aims to selectively transfer regions of interest from the source image onto the target image while maintaining the rest of the target image unchanged. Most studies on face swapping designed specifically for full-face swapping, either unable significantly limited when it comes to swapping individual facial parts, which hinders fine- ine-grrined and customized character designs. However, designing such approach specifically for facial parts swapping is challenged by a reasonable multiple reference feature fusion, which needs to be both efficient and effective. To overcome this challenge, FuseAnyPart is proposed to facilitate the seamless "fuse-any-part" customization of the face. In FuseAnyPar Part,
* "Equal contribution.
I Work done during the internship at Alibaba Group. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

facial parts from different people are assembled into a complete face in latent space within the Mask-based Fusion Module. Subsequently, the consolidated feature is dispatched to the Addition-baan based Injection Module for fusion within the UNet of the diffusion model to create novel characters. Extensive experiments qualitatively and quantitatively validate the superiority and robustness of FuseAnyPart. Source codes are available at https: bestps://githll com wwwhhhmmswwh//uueeeeee wyyb/PusennyPart
1 Introduction
Imagine a person who possesses the face of Elon Musk, the eyes of Mark Zuckerberg, the nose of Sam Altman, and the mouth of Kobe Bryant. What would the visual composite of such a person look like? Currently, this dream can be realized through the facial parts swapping technology as shown in Fig. Different from traditional face swapping, which is coarse and typically replaces the entire face at once, facial parts swapping aims to transfer the individual facial components, e.g.. nose, mouth or eyes from varied sources onto the target image while maintaining the rest of the target image unchanged. A growing interest in facial parts swapping technology has emerged due to its broad applications such as innovative character creation, popular entertainment, privacy protection and beyond [29] [2)..
Most of the studies so far have primarily focused on full-face swapping, and can be roughly divided into GAN-based and diffusion-based approaches. The GAN-based methods 77 34| 27 [5] [2] usually perform face swapping by extracting the identity feature from the source images and then injecting them into generative adversarial networks [6]. Nevertheless, the GAN- techniques may not succeed in completely transferring the identity features, especially when there is a significant difference in shape between the source and the target. In addition, the GAN-based methods often involve an array of losses about image fidelity, identity, and facial attributes to guide the training, which increases the complexity of the training process. On the other hand, diffusion-based models [23] 24
26 33 have demonstrated a powerful capability in generating images with high resolution and complex scenes. Some efforts try to swap face (38 18 through diffusion models, achieving pleasant results.
However, the aforementioned methods, designed for full-face swapping, are either unable or signifi- cantly limited when it comes to swapping individual facial parts. If one needs to replace a facial part, it is only possible to swap the entire face, rather than independently swapping one or several facial parts individually, let alone the facial parts from different individuals. This high degree of coupling poses an inconvenience for users who seek more fine-graine -ha-igri and customized designs. Therefore, the focus on face-swapping shifts from an identity-centric to an attribute-level perspective.
The primary challenge in facial parts swapping lies in the fusion mechanism. Popular face-swapping techniques [15 38 36 [8] perform the fusion of source and target images in the latent space for harmonious generated images. Therefore, the feature fusion mechanism becomes critical in affecting the quality of the generated images. In the facial parts swapping task, the number of source images increases from one to multiple. further complicating the fusion process and making this issue more prominent. Previous methods [36 13 utilize adapters implemented by a cross- cooss-attention mechanism to fuse reference information into the UNet of diffusion models. However, as the gross-attention is initially designed for multi-moda. tasks 224), like text and image, it may be sub-optimal for facial parts swapping due to the difficulty in aligning fine-grained facial region features. What is more, the inclusion of multiple references increases computational needs, thus efficient fusion is essential.
To tackle these challenges, an innovative diffusion-driven approach dubbed Ans is proposed to facilitate the seamless "fuse-any-part" customization of faces. In FuseAnyPart, facial image is initially detected by an open-set detector to derive its various facial part masks. Then an image encoder extracts the facial part features based on the facial image and the aforementioned masks. Subsequently, these facial part features are assembled according to the masks within the Mask-based Fusion Module included in Fuse/nnPPart to generate a complete face in latent space. After this step, the cohesive feature is forwarded to the Addition- based Injection Module proposed by FuseAnyPart for fusion within the UNet of the diffusion model. The Addition-bal based Injection Module adds a minimal amount of parameters yet is highly effective in preserving the positional information and fine details of the image features, which demonstrates obvious superiority compared to the conventional cross- doss-attention mechanism. During the training stage, FuseAnyPart Part is trained by reconstructing
2

---

facial image conditioned on different facial parts, inspired by [8).8). In the inference stage, facial parts from images of various people can be fed into FuseAnyPart to create novel character.
Overall, the contributions can be summarized as follows: (1) To the best of our knowledge, Fuse- AnyPart is the first diffusion-baset work specifically designed for facial parts swapping. which is capable of simultaneous, multi-source and fine-grained facial parts swapping. (2) The proposed Masked-basee Fusion Module in PuseAnyPart allows dynamically aggregating specific parts from different faces in latent space. Then, the Addition-based Injection Module of FuseAnyPai injects this conditional information into UNet, which is more effective and efficient than the conventional coss-attention- -aaeee adapter methods. (3) Extensive experiments validate the superiority of Fuse- AnyPart. Ablation studies confirm the soundness of our design choices and the robustness of our proposed approach.
2 Related Work
Image Generation with Multiple References. InstantBooth [26]. InstanID [11, and IDAdapter use the average feature of all reference images, which contributes to improving generation quality. And photoMaker 13 generates an embedding by stacking embeddings from multiple ID reference images, which results in improved ID) representation. Moreover, it can create a mixed ID) embedding by controlling the proportion of identity images within the input reference image collection. Although the aforementioned methods introduce multiple reference image mechanism, the role of these reference images is similar to that of providing a single reference image, affecting only the generation of a specific subject. FastComposer [3], on the other hand, achieves the generation of multiple subjects by injecting different reference image features into distinct word embeddings. Currently, image generation using multiple reference images remains area ripe for exploration, such as generating human faces with multiple inputs.
Facial Parts Swapping. In recent years, region- controllaaee face swapping has emerged as fascinat- ing subfield within the broader domain of facial manipulation and generation. This technology enables precise control over specific regions of a face in an image, allowing the exchange or modification of features such as the eyes, nose, or mouth, while maintaining the integrity the original image's 's context. The E4S model [I5] achieves precise editing results by manipulating masks of specific regions, such as the eyes or lips, using a reference image as guide. Meanwhile, Diffswap [38] is a technique that selectively determines the regions swap by constructing masks that cover varying facial areas. Although these methods are capable of transferring specific facial regions from a source image target image, the results often exhibit unnatural boundaries. Furthermore, to achieve the replacement of facial features from multiple reference images onto a single face, multiple iterations typically required, complicating the process. This iterative approach can be time-consuming and may not consistently produce seamless, natural-looking results, indicating that there is still room for improvement in the field of cegionlcontrollable face swapping technology.
3 Method
3.1 Preliminary
StableDiffusion. Our model is based on the StableDiffusion 224 model, which progresses the diffusion process in low-dimensional latent space with a pre-trained autoencoder. Using a latent representation, StableDiffusion can maintain the essential features and structure of the data while requiring fewer steps and less time to generate high-quality samples. First, the variational autoencoder compresses the input image with a fixed Markov chain of T steps. Let ", = ar + are be the noised data at the t-th timestep, where ar a are predefined functions of and (o I). UNet eg is responsible for the denoising process by predicting the noise e. The denoising process can be conditioned by the additional condition C. The training objective is to minimize the ELBO of the denoising process, which is defined as:
latent representation zo, which is gradually added Gaussian noise
C= E=
co 1, ()
During inference time, UNet gradually predicts the noise ( () and recovers the initial latent representation 2g from random noise 27 (V l). Finally, the image is generated by mapping 2o back to pixel space with the variational autoencoder.
3

---

Flemert-sttoe Product
Trainable Modulea
Diflusion model
Addiion-bossed Irjection Module
Modules
map
T
P
T
VAE
Ita
Pt-1
Linear
T
Addilon-beeed Injection Mocule
Interpolation
Sai Delectar
Mayar
Moee
Naruth
4
Mask-based Module
M.)
Image Encoder
I
F
UNet
Cond Image Features
Figure 2: Illustration of FuseAnyPart. The process begins with open-set detector identifying facial image to obtain various facial part masks. Following this, an image encoder uses these masks and the facial image derive the corresponding facial part feature. These facial part features and masks are then fed into the Mask-based Fusion Module to piece together a complete face in latent space. Subsequently, the consolidated feature is dispatched the Addition-based Injection Module for fusion within the UNet of the diffusion model.
Image Prompt Adapter. Image prompt adapter (36] is innovative approach incorporate image features into the generation process without model fine-tuning for each new concept. This approach addresses the challenge faced by previous methods, which struggled effectively extract and utilize detailed image features from image prompts. Similarly to text prompts, image prompts can also condition the generative process through the cross-attention mechanism. Specifically, decoupled cross- cross-attention strategy is employed in which an additional cross-attentis layer is added to every original rros-aatenttinn layer to inject image features. The output of this new cross-attention layer can be articulated as
Znew Attentionn(Q (qaiiiionQ, (QK,V) Attention K", V*),
where À is weight factor. Attention(Q K', V*) is cross-attention of the new added cross-attention layer, Kt c, Wi and V W. are key and values matrices of the corresponding operation. e; the image features, and Wi and W are the relevant weight matrices.
3.2 Overview
The goal of facial parts swapping to selectively transfer regions of interest such as the eyebrows, eyes, nose, mouth from the source image onto the target image while maintaining the rest of the image unchanged. Some methods [15 swap region feature obtained by mask pooling to facilitate facial parts swapping and use masks as structure guidance to maintain detail and coherence in the generated results. However, the applicability this approach limited advanced StableDiffusion models utilizing tross-attention mechanisms, due to the difficulty in aligning fine- fine-grained facial region features with the latent features in UNet. Consequently, it is necessary to fine-tune the entire SD model with a significant volume of data. Additionally, there is notable paucity of methodologies that permit the incorporation multiple source images to selectively transfer features from sources onto target image in step.
To address these problems, we map face image into multiple non-overlapping region image features and perform bask- fusion at the feature map level between the source and target image features. This fusion results new facial image features, which are then integrated into the generation process through an Addition blasti Injection Module.

---

3.3 Facial Feature Decomposition and Aggregation
For simplicity, we consider three regions for swapping including the eyes (including the eye- brows), nose, and mouth. We use an open-set detection model to get region masks Meyes- M Mnose. Mouthh and the remaining region mask is Mremain = M @ Mnose @ M mooth- Let M {Meyes, Mrose mouth, M J be the region masks, a given face image I can be repre-
sented as the union of multiple regional images: I = {4)) where I; I O M;
Following most of the previous methods 2 37 32 we utilize a pre-trained CLIP image encoder Φ to extract image representations from regional images. Contrary to the preceding work that harnesses the more abstract, global, and high-level features from the last layer, we use the uncompressed feature map from the penultimate layer, which retains greater spatial information and finer details. For a face image 1, its feature representation can be decomposed into multiple where f. o(1,). Replacement is conducted at the feature map level,
) {fi) where a target image's 's features F/ = {fe, Jizo with 0, 2, are replaceable, corresponding respectively to eyes, nose, and mouth. Mathematically, the feature replacement between target image features F. = {fr, 13 o and source image features F, = {fa. 1 is realized in the Mask-based Fusion Module, described as follows:
components F {ff
Sto
R; = True
f/. = Un O + G(fa, @ Mr,
(3)
R; + True
where E 1,, 3). R₁ indicates whether the region feature fg. is replaced by fa.. G m1, m2) is an interpolation function that resizes the region covered by mask my in image I to fit the region of mask m2. The resulting features Fi 1 are aggregated and then fed into Multi-Layer Perceptron (MLP) to generate the final condition feature map, which is input into the UNet as C' = MLP 00 Fl. 3 Mr,) providing the conditional information to guide the generative network.
3.4 Addition- based Injection
It has been discussed for long time how inject image features into the UNet using cross- attention mechanism. and there are two principal methodologies. One is direct method that feeds the concatenation of image features and text features into the layers of cross-attention. However, this method can be ineffective when image features are misaligned with textual features in the concatenation process. The other one, which has been widely adopted in numerous works, employs adapter modules with decoupled cross-attention [36). Nevertheless, the cross mechanism still faces the challenge of inaccurate feature fusion because attention maps may fail focus appropriately on the correct regions.
Thus, we propose the Addition-based Injection Module for integrating image features into the UNet which directly adds fine-grained image features to latent features within the UNet. Specifically, the output of the injected layer is described as follows:
7' = Z + X Inter((innerr(C)))
where Z is the latent feature within the UNet, C is the swapped face image feature map that servers as the condition information, Linear(-) is linear layer, Inter(-) is function which is capable of resizing to match the dimensions of Z and À is weight factor. It is feasible because the latent space features within the UNet also comprise feature map that contains positional information, which has a corresponding spatial relationship with the condition feature map C. By adding fine-grained image features at their respective locations, we ensure alignment of the newly introduced features with the original latent space features in terms of position. Furthermore, the injection of such image features is not confined exclusively to the cross-attention layers; it can be integrated at any level within the UNet architecture. Compared to the cross-attention mechanism, this method reduces the number of added parameters and computational load while increasing the flexibility and controllability of feature injection. It enables the model to be fine-tuned with less training data, enhancing efficiency without compromising on the richness of the generated details.
3.5 Training and Inference
To preserve the regions of the face image that are not subject to replacement, FuseAnyPart follows the practice of [[] Specifically, We concatenate the latent vector 2m. derived from the masked
5

---

image Its, the associated mask Me., and the noised latent vector 2, to form a new latent vector 2 Co Mag. 21,, which fed into convolution layer for dimension adjustment. The feature vector 2, (onv(=z) is subsequently introduced into the UNet, serving as the query. Our training objective is similar to the original StableDiffusion model, formulated as:
L E B... M.
t, Em, M C)
(5)
During the inference phase, our model possesses the capability transfer facial regions from multiple source images onto a target image. By deconstructing and reassembling facial image features, we can construct mixed facial features, facilitating flexible and controllable facial parts swapping.
4 Experiment
Dataset. We train our model on the CelebA-HQ 11 dataset. The CelebA-HQ dataset contains 30,000 high-resolution face images of celebrities widely used for face generation and face swapping tasks. This dataset has been pre-processed and aligned, and is available in three different resolutions. In our experiments,
use the 1024 x 1024 resolution. Our evaluation set is sampled from the FaceForensics+* [25) dataset, which contains I, 1,000 videos. We randomly sample 10 frames from each video and obtain 10,000 images. Then we use GPEN 35) for portrait enhancement and crop and align these images by landmarks to the resolution of 512 x 512. Additionally, we collected some high-quality face images from the internet intended for qualitative visual results.
Implementation Details. Our implementation is based on HuggingFace diffusers 30 library and we use StableDiffusson v1-5 [24] and OpenAl's diip-vitlaarge
We train our model on 16 NVIDIA A100 GPUs (80GB) with batch size of 16 per GPU using the AdamW optimizer [16] with constant learning rate of le-4 and weight decay of 0.01. During training. facial part reference images are randomly sampled from images with the same ID, and the target image is consistent with the face reference image. During the inference stage, we use the DDIM (28) sampler with 50 steps and set À = 1.0. Since we do not use a text prompt, we set the text prompt to empty.
14 vison model 22
4.1 Qualitative Comparisons
We collected a series of high-quality celebrity images from the Internet to conduct qualitative experiments. To demonstrate the effectiveness of our approach, we have structured the qualitative experiments into three sets: fuse any part, multiple parts replacement, and multiple parts replacement with reference images in different styles.
4.1.1 Fuse Any Part
With grounding-dino [14, an open-set object detection model, our method is capable of extracting region-specific features of faces based on text, such as "eyes", "nose". and "mouth". Limited by the performance grounding-ding paper, "reference image" is used to describe both the source and target images. The face reference image is the target, while the facial part reference image is the source.
the "eyes" include the "eyebrows" in the subsequent chapters. In this
We select eyes, nose, and mouth for attribute-level facial parts swapping. For single part replacement, we compare our method with StableDiffusion (4), IP-Adapter (36), FacePartsSwap [. E4S [55] and Diffswap [38], and the results for eyes, nose and mouth presented respectively in Fig. 3
Fig. and Fig. 9 Since SD and IP-Adapter aren't 't designed for facial parts replacement, we cut out the desired attributes from the source image and overlaid them onto the target image, resulting in a pixel blend image, the inputs are source-target image pairs. By utilizing SD's image-to-image generation function with the denoising strength set 0.5, we can reconstruct the spliced image. For the -ddAdtpeer, the spliced image serves as an image prompt, acting as additional information to condition the generation process. FacePartsSwap specifically focuses on exchanging facial parts and E4S and Diffswap are face swapping methods that can utilize different masks during the inference process achieve partial facial region replacement.
4.1.2 Multiple Parts Replacement
Multi-attribune replacement differs from swapping the entire facial information onto target image, as it only involves replacing certain attributes within their corresponding regions, and the number of

---

face
IAddeter
FacePatsSwap
BAS
DiftSwap
OuuS
ITRO
contatte
ODieL
PPOCIOL
ROCOCIeL
RUC
Figure 3: Qualitative comparison of eyes swapping. Our method produces ugh-fidelity results that maintain the consistency of facial features while ensuring a natural appearance.
facial parts
faoe
SD
IP-Aclrter
FaopPatsSwap
E4S
DillSwap
Ouss
Figure 4: Qualitative comparison of multiple facial parts swapping with a single reference face.
Our method can naturally replace multiple facial parts of one face with those of another and better preserve both the characteristics and the facial part shape. More results are presented in Fig. TO
replaced attributes can be arbitrary. We demonstrate the simultancous replacement of eyes, nose and mouth. Fig. 4 showcases the results multi-attribute replacement using source-target pairs.
Both the SD and IP-Adapter struggle to maintain the non-replacement areas unchanged, and the simi- larity the replaced attributes not high, highlighting the limitations of pixel space manipulations. While FaceParts wap and E4S can retain a higher degree of attribute similarity, the replacement results often appear visually inconstant and forced, particularly when there a significant difference between the source image and the target image, such as in skin tone or facial angle. In contrast, the replacement effect Diffswap is not pronounced.
Our method outperforms these approaches by offering better consistency across both the replaced attributes and the unaffected areas, leading more seamless and natural integration of the replaced features regardless of discrepancies in the reference images. Moreover, Fig. (igg5s]mmoossratee the results when each replaced attribute originates from different source images. Our method still significantly surpasses other approaches, as it can combine distinct attribute features and generate natural- -atural-looking facial photos. This superior performance indicates that our method effectively extracts and integrates the characteristics of individual attributes, even when dealing with varied sources. It reinforces our method's flexibility and robustness in handling complex face manipulation scenarios where each facial feature may require different treatment based on its unique reference image.
7

---

mouth
face
SD
1P-Adipber
Sar
F45
DiftSsap
Gues
Figure 5: Qualitative comparison of multi swapping with multiple reference faces. Our method remains robust to different appearances of various reference facial parts.
Figure 6: Facial parts swapping on images with different styles. As an extended application. Any can use the facial parts of reference images with different styles to generate harmonious faces without changing the style the target face. We show that the features of the facial parts and the image style can be well decoupled.
4.1.3 Fusion Across Different Styles
Fig. shows the results of our method performing multi-atrriben replacements images spanning various styles. This a limitation often encountered in many face swapping methods, as they typically rely facial segmentation models which are bound by the constraints of their training data and tend to have poor generalization on unseen data. Benefiting from the open-set detection model, our method extract any regional feature from the reference images, which helps our method to generalize well on the data with different distributions.
To get images in different styles, apply style modification model [1 to real photographs to generate a series of images in different styles, including cartoon, 3D, sketch, and more. We then use these stylized images as references to perform multi-attribute replacements following the experimental setup described earlier. Despite the style discrepancies between reference images, our method is still able to accurately extract and fuse the targeted features. The generated results maintain the characteristics of the reference images while preserving the style of the original face reference. This demonstrates our method's robust capability to handle diverse styles and perform complex attribute fusion tasks effectively.
4.2 Quantitative Comparisons
Evaluation Metric. Following common practice, we adopt Fréchet Inception Distance (FID) 10 21 to evaluate the quality of the generated images. Our method is capable of generating faces with multiple reference images, including a face reference image and three facial part reference images (an eyes image, nose image and mouth image). To evaluate the effect of facial part reference images, we propose a metric named FPSim (Facial Part Similarity) to measure the similarity between the corresponding facial parts of the generated face and those of reference images, and FPSim-E, FPSim-N and FPSim-M are distinct metrics that respectively measure the similarity of the eyes, nose, and mouth. FPSim is defined as fa.fs where fa and fo are the attribute-level features of the fTT generated face and the corresponding reference images. To extract attribute-level features, we train three facial attribute-level feature extractor models with ResNet50 (9] and ArcFace loss [ on the CelebA-HQ dataset for eyes, noses and mouths respectively. To measure the ability to reconstruct

---

Table Quantitative Comparisons on FE++. We report Fréchet inception distance, eye similarity, nose similarity, mouth similarity and Mean Square Error and show that our method achieves SoTA or competitive results compared with existing methods. FacePartsSwap is essentially a cut paste method, rather than generative one, and thus has higher FPSim. Therefore, we only present its results here and do not include the quantitative comparisons.
Methods
FID
FPSim-Ef
FPSim-Nt
FPSim-M
MSE
StableDiffusion 24
18.57
0.3080
0.2215
0.2127
1.66
IP-Adapter 36
69.35
0.2865
0.2066
0.1886
13.72
FacePartsSwap
44.23
0.3269
0.2190
0.2220
24.40
E4S [1]
30.61
0.2764
0.4047
0,1903
3.03
Diftswap
38
12.07
0.2461
0.1967
0.1731
0.15
Ours
10.54
0.3186
0.2234
0.2196
0.77
Table 2: Quantitative comparison of feature fusion under different ablative configurations. Both the generation quality and facial part similarity are measured.
Method
FID
FPSim-Ef
FPSim-Nt
FPSim-M f
MSEJ
Cross-attention
15.81
0.2542
0.1763
0.1771
1.02
Multiple Addition-in-Com
15.32
0,2407
0.1834
0,2063
1.94
Cross-attention +
16.66
0.2706
0.1897
0.1797
0.66
Cross-attention + Addition-in-CA
10.51
0.3108
0.2128
0.2158
0.71
Cross antsstttttntion- Addition-in-CA, Hierarchy
28.96
0.2808
0.2034
0.2077
1.33
Addition-in CA (Ours)
10.54
0.3186
0.2234
0.2196
0.77
Cross-atiention: Using cross-attention inject conditional features.
Add-in-Conv: Addition within convolutional layers.
Add-in-CA: Addition within cross- attentinn layers.
Hierarchy: Use of hierarchical features.
faces with reference images from the same ID, we compute the Mean Square Error (MSE) between generated images and reference images.
Quantitative Comparison. As indicated in Tab. D we compare our method with previous methods on the FaceForensics** (25] dataset. The results show that our method outperforms previous methods in FID significantly. indicating that we can generate high-fidelity swapped faces and can better preserve naturalness and harmony. Mcanwhile, we also achieve comparable results attribute-level metrics, demonstrating that our method can also keep the characteristics observed limitation in DiftS wap (38), with its tendency to yield results more resemble the source face rather than the intended target face, as illustrated Fig. Therefore, when both the source and target faces come from the same ID, DiffSwap achieves lower MSE (0.15) compared to ours (0.77). Nonetheless, it is more common for the IDs of the source and target faces to differ; in these cases, our method consistently shows superior performance.
the swapped facial parts. Notably,
4.3 Ablation Study
Qualitative comparisons are in Fig. and the quantitative comparison shown in Tab. where both the generation quality and facial part similarity are measured.
Cross-attention vs. Addition. To inject conditional features of reference facial parts, we directly add the swapped face image feature to the UNet latent feature instead of using cross-attention. As shown in the first and last rows Tab, 2 direct addition significantly enhances the swapping performance. We also try to use different cross-attention modules for different facial parts and add their results together to form the latent features of the UNet (denoted by "Multiple Cross-attention")) This method provides only limited improvement to the model's 's performance (the 2nd row of Tab. a| the 4th column Fig. 7 Moreover, we try to combine the two fusion methods, adding the swapped face image feature to the output of cross-attention layers (the 2nd row of Tab. (2). Although there is a slight improvement in image fidelity (FID and MSE), the facial part similarity all decreased (FPSim-E, FPSim-N and FPSim-M).

---

Tocial pars
Crossmmttion
Mukiple Chos- alleniin
Addiamn-nsCom
Cmoosttetion
Cowaettorm AdllismiinsAA
Crnoottattonn AdliiinniCC Hierardy
Additinnis:AA (Ouns
Figure 7: Qualitative comparison of different ablative settings.
Feature Injection Across Layers. We inject the swapped face image feature between the two convolutional layers in each ResNet block of the UNet, rather than in the cross-attention layers. The results in Tab. 2 suggest that the Addition-baat -baeed Injection Module should be positioned in the cross-attention layers (Row 3 and 4). From the 5th and 6th columns of Fig. 7 fusion the cross-attention layer preserves more details and achieves higher similarity than that in the convolution layers.
observe that
Features from the 4th block of the CLIP image encoder and one after every four blocks extracted and concatenated form the output of the CLIP image encoder as the hierarchical feature, which contains abundant facial visual information. According to the Row and 5 of Tab. 2] the method is unable to generate high-fidelit and realistic faces with significant decrease image quality metrics, which is confirmed by the 7th column in Fig. 7
Hierarchical Feature.
Societal Impacts, Limitations and Conclusion
Societal Impacts. The proposed FuseAnyPart is fundamentally harmless. Nevertheless, misuse of it, e.g., applications with copyright issues and racial issues, could have negative effects on society. As result, we call for conscientious and ethical implementation of FuseAnyPart.
Limitations. While FuseAnyPart demonstrates strong performance in facial parts swapping, it still has some limitations at the current stage. First, while FuseAnyPart performs well on range of images, there may be challenges with faces that have extreme poses, occlusions, or expressions. Additionally, method is primarily designed for facial parts swapping and does not directly tackle the challenge of preserving or transforming facial expressions during the process of swapping. Fuse AnyParr Part is based diffusion models, which typically exhibit high computational complexity due recursive iterations. Algorithms like Latent Consistency Models (LCM) 17 can accelerate inference by reducing the number of iterations, while techniques such int8 model quantization can significantly lower computational load. Together, these strategies enhance the speed of FuseAnyPart Like most generative models, FuseAnyPar relies on high-quality training datasets. The quality of the images can be enhanced using super-resolution methods.
Conclusion.
This paper proposes FuseAnyPart, novel diffusion-driven method for facial parts swapping. FuseAnyPart first extracts multiple decomposed features from face images with masks obtained from an open-set detection model. Then parts from different faces aggregated in latent space with the Mask-based Fusion Module. An injection module injects this conditional information into UNet for fusing effectively. Extensive experiments validate the superiority of PuseAnyPart.
10

---

References
[1] Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, Yaohui Li, Changhua Meng. Huijia Zhu, Weiqiang Wang, al. Diffute: Universal text editing diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
[2] Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding. Yangyang Song, Shizun Wang. Wang, Hao Yang. Jing Liu, Kang Du, et al. Photoverse: Tuning- models. arXiv preprint w7Xiv:2309005990 2023.
fure free image customization with text-to-image diffusion
[3] Siying Cui, Jiankang Deng, Jia Guo, Xiang An, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning-frce frreegffree personalization of text-to-image models. arXiv preprint arXiv:2403. 13535, 2024.
[4] Jiankang Deng, Jia Guo, Niannan Xuc, and Stefanos Zafciriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4690-4699, 2019.
[5] Claudio Ferrari, Matteo Serpentoni, Stefano Berretti, and Alberto Del Bimbo. What makes you. you? lyzing recognition by swapping face parts. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 945-951. IEFF, 2022.
[6] lan Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, Advances in neural information processing systems, 27, 2014.
[7] Jingtao Guo and Liu. Facial parts swapping with generative adversarial networks. Journal of Visual Communication and Image Representation, 78: 103152, 2021.
[8] Yue Han, Jiangning Zhang. Junwei Zhu, Xiangtai Li, Yanhao Ge. Wei Li, Chengjie Wang. Yong Liu, Xiaoming Liu, and Ying Tai. A generalist facex via learning unified facial representation. arXiv preprint arXiv:2401.000 00555 2023.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[10] Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[11] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710. 10196, 2017.
[12] Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. Advancing high fidelity identity swapping for forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5074-5083, 2020.
[13] Zhen Li, Mingdeng Cao, Xintao Wang. Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked embedding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[14] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:23033.05499 2023.
[15] Zhian Liu, Maomao Li, Yong Zhang, Cairong Wang, Qi Zhang, Jue Wang, and Yongwei Nic. Fine-grained face swapping via regional gan inversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8578-8587, 2023.
[16] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[17] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[18] Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint arXhv:2303.09319, 2023.
[19] Yifang Men, Yuan Yao, Miaomiao Cui, Zhouhui Lian, and Xuansong Xie, Det-net: Domain-caliin calibrated translation for portrait stylization. volume 41, pages 1-9, 2022.
11

---

[20] Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision. pages 7184-7193, 2019.
[21] Gaurav Parmar, Richard Zhang, and Van Zhu, On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11410-1144 14200, 2022.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[23] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 1(2)3, 2022.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[25] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics*#:: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 1-11, 2019.
[26] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXi::2304003411, 2023.
[27] Sridhar Sola and Darshan Gera. Unmasking your expression: Expression-conitioned gan for masked face inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5907-5915, 2023.
[28] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[29] Qianru Sun, Ayush Tewari, Weipeng Xu, Mario Fritz, Christian Thcobalt, and Bernt Schicle. A hybrid model for identity obfuscation by face replacement. In Proceedings of the European conference on computer vision (ECCV), pages 553-569, 2018
[30] Patrick von Platen. Suraj Patil, Anton Lozhkov. Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusioo models.
com www.ggggaggggeiaanagggeeeeeeeeee
2022.
[31] Qixun Wang. Xu Bai, Haofan Wang. Zekui Qin. and Authony Chen. Instantid: Zero-shot Serentty-preeevingg generation in seconds. arXiv preprint arXiv:2401.07522 2024.
[32] Yuxiang Wei, Yabo Zhang. Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943-15953, 2023.
[33] Guangxuan Xiao, Tianwei Yin. William T. Freeman, Frédo Durand, and Song Han. Fastcomposer: Tureng-free multi-subject image generation with localized attention. arXiv, 2023.
[34] Zhiliang Xu, Hang Zhou, Zhibin Hong. Ziwei Liu, Jiaming Liu, Zhizhi Guo, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Styleswap: Style-based generator empowers robust face swapping. In European Conference on Computer Vision, pages 661-677, Springer, 2022.
[35] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pages 672-681, 2021.
[36] Hu Yc, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
[37] Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang, Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang, Yao Hu, Han Pan, and Zhongliang Jing. Ssr-encoder: Encoding selective subject representation for subject-driven generation, 2024.
[38] Wenliang Zhao, Yongming Rao, Weikang Shi, Zuyan Liu, Jie Zhou, and Jiwen Lu. Diffswapp: High- fidelity and controllable face swapping via 3d-aware masked diffiusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8568-8577, 2023.
12

---

A
More qualitative results
As discussed in Sec. 4.1.1
the qualitative comparison of the nose and mouth swaps is presented in
Fig. 8 and 9
face
SD
RaodPartswwap
EAS
DiffSwap
Ours
enIe
peite
combnte
Amm
1080
aCiet
ROC
EPCIR
rUcaciet
Figure 8: Qualitative comparison of nose swapping.
face
SD
IP IP-Adapter
FacePartsSwap
E4S
DiffSwap
Ours
H
ND
S
lth
ntiss
Ith
lth
lth
Figure 9: Qualitative comparison of mouth swapping.
facia parts
SD
IP-Adapter
FacePartsSwap
DiftSwap
Ouss
Figure 10: More qualitative comparisons of multiple facial parts swapping. This provides additional examples related to Fig.
As shown Fig. FuseAnyPar Part performs well when swapping facial parts from individuals of significantly different racial or age groups
13

---

facial parts
face
swapped face
facial parts
face
swapped face
Y Young Asaa to Old Black
Middle-ages black to Y Young Asian
Figure 11: Illustrations facial parts from significantly different racial and age groups. Facial part
swapping between source and target images that significantly differ in age and race.
icon
MAcOCom
mnGOcom UIGGO
enter cenente
cenente
cenenter
cenente
cen
LIVES
LI LIVES
LIVI LIVES
LIV LIVES
LIV1 LIVES LIVI
im
ilm
ilm
ilm
im
ilm
ociet socociet
socociet
socociet
socociet
6OC
ncole incncnie enchcole
incesolz
Ancheoly
inc
thres: *
thres; = 0.2 thres; = 0.4 T thres; = 0.6 * T thres; = 0.8
The skin color change issue can be effectively resolved by replacing the generated skin regions with the inverted latent representations denoising process. The threshold indicates the number of steps performed above the replacement operation the denoising process.
Figure 12:
original skin color using DDIM inversion the
FuseAnyPar Part may encounter issues with color changes in generated images, but this problem can be addressed by replacing the generated skin regions with inverted latent representations of the original skin color using DDIM. The results are presented in Fig. 12
FuseAnyPart was also qualitatively compared with DiffFace, and the results are shown Fig. More diverse results of multi swapping with multiple reference faces presented in Fig. 44
14

---

Source
Target
DiffFace
Ours
eyes
mouth
svappted
Comparison with DiffFace. DiffFacc generates images with local distortions in the eyes and mouth, whereas our method produces cleaner results that are more similar to the source image regarding the facial parts.
Figure 13:
Figure 14: Qualitative results of swapping face parts from different sources to a target face.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in abstract and introduction in Sec. accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
16

---

The approach presented in this paper utilizes deep learning and adopts data-driven experimental methodology.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The quantitative findings utilize datasets that are publicly accessible.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: The code and model files will be made available to the public after the author has finished refining them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https SttSSSabmisiioooiiiioi.oooii for more details.
//nips.cc/ppp wwwppcclccgguiecuceccccceooooope wttpggudddee/eeebeininnneeee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: The authors will personally evaluate the risks and decide the publication of the model files.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: The code and data mentioned, but were unable to find the license for the dataset we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Currently, the data and code remain private, but there may be consideration for public release in the future.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: No Crowdsourcing used in this work.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We not sure if we have obtained IRB approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
21


Reviews:
[{"review_id": "5XMiejGWBB", "reviewer": "Reviewer_b8uw", "summary": "\"FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images\" introduces a novel framework for swapping individual facial parts using a diffusion model that effectively utilizes multiple reference images. The paper outlines the methodological innovation and superiority of FuseAnyPart over traditional GAN-based and diffusion-based methods, which primarily focus on full-face swapping. This approach enables high-fidelity and cohesive blending of facial parts from disparate sources, enhancing fine-grained character customization capability.", "questions": "1.  What is the performance impact when using lower-quality or varied lighting conditions in input images?\n2.  Could this method be adapted for real-time applications, and if so, what optimizations would be necessary?\n3.  How does the method perform when facial parts from significantly different racial or age groups are swapped?", "limitations": "The authors adequately discuss the limitations, including the high computational requirements and the dependency on high-quality reference images. They also mention potential challenges in diverse application scenarios, which is vital for setting realistic expectations for the method's deployment.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "*   **Originality**: This paper introduces a unique application of diffusion models to the problem of facial parts swapping, diverging from the traditional focus on full-face swaps.\n*   **Quality**: Demonstrates improved quality and robustness in facial parts swapping through qualitative and quantitative results.\n*   **Clarity**: The paper is articulate and well-organized, with thorough explanations and clear visual aids that enhance the understanding of the proposed method.\n*   **Significance**: Offers significant practical applications in various fields, including digital media creation and personalized entertainment.", "weaknesses": "*   **Complexity**: The computational complexity might limit its application in real-time or on lower-end devices.\n*   **Scope of Data**: More diverse testing on datasets from various demographics could enhance the robustness and generalization claims.\n*   **Dependency on High-Quality Inputs**: The method's effectiveness relies heavily on the quality of input images, which could limit its applicability in less-controlled environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W. The weakness of FuseAnyPart.]** \n1) Diffusion models typically have high computational complexity due to the need for recursive iterations, which limits their applications to real-time or on lower-end devices.\n2) More diverse testing results are illustrated in Figure 1 in the newly submitted PDF to validate the robustness and generalization of FuseAnyPart.\n3) Most models rely on high-quality training datasets. The quality of images can be improved using super-resolution methods.\nFor example, the dataset used in this work is CelebA-HQ, which is reconstructed with super-resolution through GANs.\n\n**[Q1. Performance of low quality inputs.]** \nThe performance of FuseAnyPart may be affected by lower-quality or varying lighting conditions input images.\nHowever, some preprocessing techniques, such as facial alignment and **super-resolution**, can be adopted to mitigate these effects.\n\n\n**[Q2. Optimizations for real-time applications.]** \nAlgorithms like **Latent Consistency Models (LCM)** can overcome the slow iterative sampling process of Diffusion Models, enabling fast inference with minimal steps instead of the usual dozens or hundreds.\nIn engineering, techniques like int8 **model quantization** can significantly reduce computational load.\nTogether, these strategies can speed up FuseAnyPart.\n\n**[Q3. Results of significantly different racial or age groups.]** \nThese results are shown in Figure 1 in the newly submitted PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hA2wOANfGE", "reviewer": "Reviewer_Aivf", "summary": "This paper explores the partial face swapping problem. Rather than swapping the whole face from A to B, partial face swapping aims to swap some specific area (or organ) of A to B. In this paper, a diffusion based partial face framework is proposed. Besides, two modules are designed to better fuse the extracted feature into the diffusion unet. Facial Feature Decomposition effectively extracts facial semantics and Addition based Injection module integrates the semantics into the diffusion model. Further experiments demonstrate the effectiveness of the propsoed framework.", "questions": "See the weaknesses.", "limitations": "The author have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The task (partial face swapping) is interesting and more challenging compared to whole face swapping. It needs fine-grained controls and thus worth exploring.\n\n2. The storyline of this paper is simple and clear.", "weaknesses": "1. The paper says \"the primary challenge in facial parts swapping lies in the fusion mechanism\". Could you please detail it? In my point of view, this challenge also exists in conventional face swapping task. Face swapping manipulates the face area of an image while partial face swapping manipulates a smaller area. Previous mask-based face swapping methods first generate faces with the same expression as the target face and then pastes it to the target face according to the mask. \n\n2. Some details are missing in the method:\n* What is the $z_T$ (the initial point of the denoising process) used in the training and inference? is it a Gaussian noise or the target image?\n* Why the training objective is to reconstrct the target (Eq 5)?\n* What is the dimension of The extracted feature $f$? If it is a H*W feature map, how to fed it into MLP? \n\n3. There are two few qualitative results in the article (the main paper and supp). The experiments adopt CelebA datasets which do not have IDs, how to sample from the same ID (Line 207)? I guess it is sampled from the same image?\n\n4. More comparisons with conventional face swapping method (fsgan, simswap, diffFace) should be given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1. The primary challenge lies in the fusion mechanism.]** \nTraditional face-swapping methods first perform face reenactment and then paste it onto the target image in pixel space, as illustrated in FSGAN and DiffFace.\nHowever, operations in pixel space often result in unnatural images with visible seams, leading to a low success rate in practical applications and producing many poor outcomes as shown in Figure 3 of the newly submitted PDF.\nThe current mainstream face-swapping techniques now perform the fusion of source and target images in the latent space, resulting in more harmonious generated images.\nTherefore, the feature fusion mechanism becomes critical in affecting the quality of the generated images.\nIn the facial parts swapping task, the number of source images increases from one to multiple, further complicating the fusion process and making this issue more prominent.\nAs shown in Table 2 and Figure 1 of this paper, the authors conducted extensive experiments to validate the superiority of the proposed fusion mechanism quantitatively and qualitatively.\nWe thank the reviewer for highlighting this issue and suggesting that the task description be more precise.\nHowever, we hope the unique aspects and challenges of the task are adequately appreciated.\n\n**[W2. The details.]** \n1) **The Initial Point of the Denoising process.** During the training phase, the initial point consists of concatenating the target image with noise in latent space, the masked image in latent space, and the mask itself, as illustrated by z\\_t' in Figure 2.\nIn the inference phase, the target image with noise in latent space is replaced with Gaussian noise at the initial point.\nThe above is discussed in Lines 186-189 of the paper.\n2) **The training Objective.** FuseAnyPart uses reconstruction as a proxy task and does not require swapped paired faces.\nFor instance, the target face image and the source face images, which provide different facial parts, are sampled from different images of the same identity.\nTherefore, reconstructing the target image allows the FuseAnyPart to gain the ability to generate natural images by merging different organs.\nOur paper details the training and inference pipeline in Lines 60-62.\n3) **The dimension of the extracted feature.**\nThe feature $f$ from the CLIP image encoder has a shape of $(h\\times w) \\times c$, which corresponds to $h\\times w$ visual tokens.\nThis feature $f$ can be input into an MLP to align with the dimensions of the latent features in UNet.\n\n**[W3. Questions on the CelebA dataset.]** \nThe CelebA dataset now includes identity annotations in the “Anno/Identity\\_CelebA.txt file,” which can be downloaded from the official CelebA website. During training with FuseAnyPart, source and target images are sampled from different images of the same identity.\nMore qualitative results are added in Figure 4 in the newly submitted PDF, and they will be added in the final edition of the paper.\n\n**[W4. More comparisons.]**\nThe state-of-the-art face-swapping methods supporting face parts swapping are all included in Table 1 of our paper.\nAccording to their papers, FuseAnyPart beats Diffswap in FID and OSim, and Diffswap beats SimSwap(2020) and FSGAN(2019).\nA qualitative comparison of FuseAnyPart with DiffFace is shown in Figure 3 of the newly submitted PDF, demonstrating a significant advantage of FuseAnyPart.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ukOfQwNrM", "reviewer": "Reviewer_jimd", "summary": "This paper delves into the strategy of facial parts swapping which are not studied before, their method aims at fusing facial parts from different sources into the overall background/face images. It involves masking facial landmark areas(i.e. eyes, mouth, nose) and fusing with mask-based operation. Finally, the conditions are injected into diffusion models with an Interpolation layer.", "questions": "1. lines 163-166, do you have literature or visualization results to prove 'method can be ineffective when image features are misaligned with textual features'?\n\n2. What is the target/labelled image for training, considering there is no correct swapped reference image? Is it classifier-free guidance?\n\n3. Is the code available?\n\n4. Which version of the VAE model is used in this work? Is there any lightweight but effective pre-trained VAE?", "limitations": "this work is interesting for part-based facial editing. But some details are missing such as the evaluation protocol OSim should be explained in detail, as well as the landmark detection network.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper addresses the problem of part-level facial at the feature level which is interesting.\n2. The proposed injection method improves the final generation than the traditional cross-attention method.\n3. Qualitative and Quantitative experiments have shown the effectiveness of the proposed method.", "weaknesses": "1. the evaluation metric(OSim) about whether the face is correctly swapped is not explained in detail, maybe providing more text description is easier for the reader to understand. For example, does the OSim-E input only include the eye image, or is it the entire image with only the eyes unmasked? Additionally, what is the label used for training OSim, is it based on identity?\n\n2. the Addition-based Injection Module is simple, does advanced modules ( for example convolution/ self-attention) lead to inferior performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1. Details about OSim.]**\nOsim measures the similarity between the swapped facial parts (eyes, nose, mouth) in the generated image and those in the reference images.\nFor example, we utilized the CelebA-HQ dataset and employed the grounding-dino detection model to identify and extract bounding boxes for the eyes.\nEach cropped eye region (the eye image) was then labeled with the identity (ID) of the corresponding individual.\nThen, we train the ResNet50 with arcface loss, which maximizes the intra-class similarity (among the same IDs) and minimizes the inter-class similarity (across different IDs). \nAfter training, we use the ResNet50 model to extract feature vectors from the eye regions. Osim is computed using cosine similarity between these feature vectors: $Osim = \\frac{f(a)\\cdot f(b)}{\\Vert f(a) \\Vert \\Vert f(b) \\Vert}$, where f represents the feature extraction function implemented by our trained ResNet50 model and a and b are the input images of the specific organ.\n\n**[W2. Alternatives to the Addition-based Injection Module.]**\nThe Addition-based Injection Module beats multiple attention-based baselines in Table 2 and Figure 7.\nAlthough simple, the Addition-based Injection Module is the most suitable for the facial parts swapping task and validated by extensive experiments.\n\n**[Q1. The  visualization results.]** \nThis point refers to the baseline method, IP-Adapter.\nIP-Adapter depends on both text and image prompts.\nIf the text prompt is inaccurate or even conflicts with the image prompt, it may have little effect on the results, leading to outcomes that do not align with the text prompt.\nWe will clarify this point to ensure it is clear in the final edition.\nPlease check the Figure 5 in the newly submitted PDF.\n\n**[Q2. The target image for training.]**\nFuseAnyPart uses reconstruction as a proxy task for training and does not require swapped paired faces, which draws inspiration from E4S and DiffSwap.\nFor instance, the target face image and the source face images, which provide different facial parts, are sampled from different images of the same identity.\nOur paper details the training and inference pipeline in Lines 60-62.\n\n**[Q3. Code available.]**\nOnce the paper is accepted, the training and inference code, model weights, and essential documentation will be released to the public.\nThe authors aim to make a valuable contribution to the NeurIPS community.\n\n**[Q4. The VAE version.]**\nFuseAnyPart is based on the widely used SD-v1.5, and the VAE model utilized in this work is the same as that in SD-v1.5.\nThe authors acknowledge that a lightweight yet effective pre-trained VAE may exist, but exploring this is beyond the scope of FuseAnyPart.\n\n**[L. The landmark detection network.]**\nA visual grounding model is used to detect bounding boxes around faces, rather than relying on a landmark detection network, as described in Line 216.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YF2HhkCGmn", "reviewer": "Reviewer_mdEG", "summary": "This paper introduces a facial parts swapping framework based on diffusion models, named FuseAnyPart. Unlike traditional methods that swap entire faces, FuseAnyPart allows for the swapping of individual facial features. The framework fine-tuned a pre-trained CLIP model to extract features from different facial regions and a stable diffusion model to generate the swapped face. To merge these features, an Addition-based Injection Module is utilized, incorporating the facial region features into the UNet feature map. Extensive experiments validate the effectiveness of the proposed method.", "questions": "Fusing multiple facial parts to generate a new face is an interesting idea. However, it is unclear how to evaluate the performances quantitatively. In addition, such generated faces are not visually plausible to me. The FID scores also indicate that the proposed method is not state-of-the-art.", "limitations": "The proposed method can only perform facial parts swapping on aligned faces, which limits its applications in practical situations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is easy to follow.\n2. Utilizing a diffusion model to achieve region-based face swapping is interesting.\n3. Extensive experiments verify the effectiveness of the proposed methods.", "weaknesses": "1. The results in Figure 3 show that the skin color also changes when only the eyes are swapped, which the authors did not discuss.\n2. The method of feature injection into the SD through addition has already been used in IPAdapter and ControlNet.\n3. Since there is no swapped face used for training, is the source face directly used as the target face during training? If so, how is the feature decoupling between different regions achieved during inference?\n4. The term `one-step' mentioned in line 65 contradicts the 50 steps of the ddim sampler described in the Implementation Details. The term ‘one-step’ should be used with caution in methods based on diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1. The problem of skin color change.]**\nFuseAnyPart may encounter skin color changes, particularly when there is a significant difference between the skin colors of the source images and the target image.\nThis issue arises because the source and target images are fused in the latent space, and the global attention mechanism diffuses the color features.\nHowever, this of skin color changes issue can be effectively resolved by replacing the generated skin regions with the inverted latent representations of the original skin color using DDIM inversion. This ensures the skin color region remain unchanged.\nThe improved results are illustrated in Figure 2 of the newly submitted PDF.\nThe pseudocode is as follows.\n```plaintext\nAlgorithm: Merging with Ground Truth Latent\nInput: latents_ref_img, init_noise, pixel_mask\nOutput: latents\n\n1. Initialize thres_i\n2. If i < thres_i\n   a. Retrieve noise_timestep from scheduler:\n      - noise_timestep <- scheduler.timesteps[i + 1]\n   b. Add noise to the latent of the reference image to obtain ground truth noised latent:\n      - gt_latent <- scheduler.add_noise(latents_ref_img, init_noise, noise_timestep)\n   c. Update latents using pixel mask:\n      - latents <- (1 - pixel_mask) * gt_latent + pixel_mask * latents\n```\n\n**[W2. Feature injection difference to IPAdapter and ControlNet.]**\nIPAdapter is not an addition-based injection method. It employs a decoupled cross-attention strategy, where an additional cross-attention layer is added to each original cross-attention layer to inject image features. And our method could perform better in facial parts swapping tasks.\nIn Section 4.3 of our paper, titled \"Cross-attention vs. Addition,\" we compared our feature injection method with cross-attention. The results indicate that our injection method is superior.\nControlNet is an addition-based injection method. We have not yet compared it in our experiments. However, for face parts swapping tasks, the majority of the target image remains unchanged, and the features needing injection are only those corresponding to the parts requiring replacement, thus eliminating the need for a complex network structure. Our method involves the addition of only linear layers, whereas ControlNet requires replicating the parameters of UNet. Therefore, our approach has a clear advantage regarding the increase in parameter count. For this task, our method is simple but effective.\n\n\n**[W3. The training strategy.]**\nFuseAnyPart uses reconstruction as a proxy task and does not require swapped paired faces, which draws inspiration from E4S and DiffSwap.\nFor instance, the target face image and the source face images, which provide different facial parts, are sampled from different images of the same identity.\nOur paper details the training and inference pipeline in Lines 60-62.\nA visual grounding model is employed to detect bounding boxes for feature decoupling across different regions.\n\n\n**[W4. The term 'one-step'.]**\nThe 'one-step' mentioned in Line 65 means that the eyes, mouth, and nose can be swapped in a single inference, unlike traditional inpainting methods, where they are swapped sequentially.\nTherefore, it does not contradict the 50 steps of the DDIM sampler.\nWe appreciate the reviewer's careful reading and will revise the paper in the final edition to clarify this point.\nThe term 'one-step' in Line 65 will be replaced with 'simultaneous'.\n\n\n**[Q. Evaluate the performances quantitatively.]**\nAs described in Section 4.2, three metrics, including Fréchet Inception Distance (FID), Organ Similarity (Osim), and Reconstruction Mean Squared Error (MSE), are adopted to evaluate the performance quantitatively.\nFID is used to evaluate the generated faces' overall image quality and visual realism. \nOsim is used to evaluate how well individual features such as the eyes, nose, or mouth match between the generated image and the sources, focusing on the fidelity of the swap within localized regions.\nThe MSE metric measures the pixel-wise accuracy of the reconstructed image against the original. It helps quantify the preservation of the non-swapped parts of the face and the seamless integration of swapped parts.\nRegarding the visual plausibility, we provided some qualitative results in the paper, and additionally, we have included more qualitative results in the newly submitted PDF.\nThe results showcase the efficacy of our method in generating plausible faces even when there are significant differences between the source and target images, including differences in age and race.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and meta reviewers,

We appreciate all reviewers for their valuable comments and suggestions. 
We have carefully addressed the comments and added details and comparisons as follows:


- We have provided solutions to accelerate inference and improve low data quality.
- We have added a wider range of testing results, particularly for significantly different race and age groups.
- We have addressed the issue of skin color variation and provided visual demonstrations of the outcomes.
- We have detailed why the fusion mechanism poses the primary challenge in facial part swapping.
- We have clarified the training process and specific variables for better understanding.
- We have outlined the training objective of FuseAnyPart, which uses reconstruction as a proxy task.
- We have included new baseline results from DiffFace for qualitative comparison.
- We have resolved the issue concerning the identity files in the CelebA dataset.
- We have added details regarding the evaluation metric (OSim).
- We have included information on the Addition-based Injection Module.
- We have revised the term "one-step" to "simultaneously" for clarity.

We will release our code and checkpoints in the camera-ready version, and please see below our responses to each reviewer.
If you have any questions or suggestions, please feel free to leave your comments on OpenReview.

Authors of FuseAnyPart

Meta-review: The paper received all accept recommendations after rebuttal. Reviewers found the proposed method interesting and easy to read, and their concerns have been addressed after rebuttal. AC agrees with reviewers and is happy to accept this paper for publication.
Decision: Accept (spotlight)