Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traceability
Authors: Yunshu Dai
Jianwei Fei
Fangjun Huang
Chip Hong Chang
Venue: ICML 2025 poster (2025)
Keywords: Face swap
generative adversarial networks
responsible and ethical AI
provenance tracing
Primary area: applications->computer_vision

Abstract: As AI generative models evolve, face swap technology has become increasingly accessible, raising concerns over potential misuse. Celebrities may be manipulated without consent, and ordinary individuals may fall victim to identity fraud. To address these threats, we propose Secure Swap, a method that protects persons of interest (POI) from face-swapping abuse and embeds a unique, invisible watermark into nonPOI swapped images for traceability. By introducing an ID Passport layer, Secure Swap redacts POI faces and generates watermarked outputs for nonPOI. A detachable watermark encoder and decoder are trained with the model to ensure provenance tracing. 
Experimental results demonstrate that Secure Swap not only preserves face swap functionality but also effectively prevents unauthorized swaps of POI and detects different embedded model's watermarks with high accuracy. 
Specifically, our method achieves a 100% success rate in protecting POI and over 99% watermark extraction accuracy for nonPOI. Besides fidelity and effectiveness, the robustness of protected models against image-level and model-level attacks in both online and offline application scenarios is also experimentally demonstrated.

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traceability
Yunshu Dai Jianwei Fei?
Fangjun Huang 13
Chip Hong Chang*
Abstract
Fugitives List
Record
As Al generative models evolve, face swap tech- nology has become increasingly accessible, rais- ing concerns over potential misuse. Celebrities may be manipulated without consent, and or- dinary individuals may fall victim to identity fraud. To address these threats, we propose Se- cure Swap, method that protects persons in- terest (POI) from face-swapping abuse and em- beds unique, invisible watermark into nonPOI swapped images for traceability. By introduc- ing an ID Passport layer, Secure Swap redacts POI faces and generates watermarked outputs for nonPOL. A detachable watermark encoder and decoder are trained with the model to ensure provenance tracing. Experimental results demon- strate that Secure Swap not only preserves face swap functionality but also effectively prevents unauthorized swaps of POI and detects different embedded model's watermarks with high accu- racy. Our method achieves a 100% success rate in protecting POI and over 99% watermark ex- traction accuracy for nonPOI. Besides fidelity and effectiveness, the robustness of protected mod- els against image-level and model-level attacks in both online and offline application scenarios is also experimentally demonstrated. Code can be found at Secure-Swap.
Airport Security Checkpoint
POI List
00010101
Face Swap model Security Checkpoint
Figure I. An illustration of our motivation behind Secure Swap. If the input image involves POI, the swap will rejected; otherwise, the model will complete the swap and apply invisible watermarks in the generated images.
1. Introduction
In the self-media era, the general public has easier access to face swap technology. However, while this technology offers new possibilities, it has also raised significant con- cerns due to its potential for misuse. Users can provide the source and target faces in pairs into the face swap model The source image provides the identity (i.e., the face be transferred), and the target image is the image where the person's 's identity is replaced by that of the source. Simply put, it uses the source person's face to replace the target person's 's face. Face swap operations targeting persons of interest (POI) can easily cause social unrest due to the sen- sitivity of their identities, while ordinary people (nonPOI) are more vulnerable face swap fraud due limited de- fense awareness and forensic capabilities (Van der Sloot & Wagensveld, 2022). To address these risks, different re- gions have proposed distinct governance frameworks for face swap technologies !
"Shooo of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University. 'State Key Laboratory Internet of Things for Smart City. Department of Computer and Information Science, University of Macau. "Guangdong Province Key Labo- ratory of Information Security Technology. "School of Electrical and Electronic Engineering, Nanyang Technological University.. Correspondence to: Fangjun Hung <huangfj @
Currently, the growing application of generative technolo- gies among the public has led to a more intricate and con- voluted crisis. Open-source tools are becoming more ac- cessible and easier to deploy for local inference, while API
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
htttpa:/ www. http:///wwwwwwwo.ooooooooo///// www.:/ewwrr...rrggggn/////...... generat www.aaaiiooeeeeeeeeeeeeeeeeeee regulattieerree

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
services are expanding at an unprecedented rate. Traditional defensive strategies, such as incorporating front-end mod- ule within API or model, are no longer sufficient. As result, the demand for robust and generalizable governance of face swap models have grown increasingly urgent.
occurrence of forgery, allowing for early response to po- tential forgery threats. At present, the most active defense measures are POI protection and watermark provenance.
POI Protection. celebrities and national leaders typically possess extensive social influence and receive high levels of attention. Their statements, actions, and images have a significant impact on the public, which is further amplified by the prevalence of so- cial media. Therefore, forgers have particular penchant for forging celebrities to disseminate false information, engage in scams, or perpetrate malicious activities. At present, a series of protection works for POI has emerged. Agarwal et al. (Agarwal et al., 2019) focused on the videos of talking in a formal setting like news interviews and public speeches. This work tracks face and head movements and then extracts the presence and strength of specific action units for POI DeepFake detection. Chu el al. (Chu et al., 2022) hypothe- sized that certain words or sentences spoken by POI tend to be accompanied by a series of regular face movements and lip motions. They adopt a dual stream structure for speaking pattern modeling, which combines the Action Unit Module that depicts face muscle movements and the Lip Motion Module that extracts sequential lip motions for detection. Likewise, Cozzolino et al. (Cozzolino et al., 2023) hypothe- sized that each person has specific characteristics that are unlikely to be reproduced by a synthetic generator. They extracted audio-visual features that characterize the identity and used them to create a POI DeepFake detector.
Persons of Interest (POI) such as social
Several studies have been conducted to counteract the abuse of face swap models, primarily through forensic detec- tion (Dai et al., 2022; Fei et al., 2022a; Wang et al., 2023; 2022; Wu et al., 2023), which is applied to forged images after the abuse has already occurred. However, proactive defense measures during the model development stage, such as function purification (Dai et al., 2024a;b) and forensic collaboration, have been largely overlooked. Under a re- sponsible governance framework, is essential for model developers to take on the obligation of ensuring the legiti- mate use of their models, incorporating generative content security and forensic accountability.
To address these challenges, we propose Secure Swap, a method specifically designed for GAN-based face swap models. Secure Swap prohibits face swapping on POI and marks the synthesized products of nonPOI by embedding invisible watermarks for provenance tracking. Inspired by the concept of security checkpoints, our method integrates an identity checkpoint within the face swap model: if the input image involves a POI, the model redacts the face: otherwise, it generates a swapped image with an embedded invisible watermark. This approach ensures the ethical use of face swap technologies and reduces the risk of face fraud. The contributions of our work are summarized as follows:
Existing methods to address unauthorized face swaps fall into two categories: proactive protection and post-hoc detec- tion. Post-hoc detection suffers from inherent latency and cannot effectively prevent image misuse. Current proactive methods primarily rely on adversarial attacks by adding perturbations to protected images, such as POI photos, to prevent forgery (Huang et al., 2022; 2021; Ruiz et al., 2020). These approaches offer image-level protection, re- quire costly preprocessing, and cannot scale to large vol- umes.
We propose a secure face swap model enhancement method called Secure Swap to enable any GAN-based face swap models to operate responsibly and with account- ability. It redacts POI faces and ensures every swapped image of nonPOI carries an invisible watermark, simpli- fying real/fake detection and provenance tracking.
Extensive evaluations on four face swap models and three human face datasets demonstrate that Secure Swap achieves a 100% success rate in protecting POI and greater than 99.9% watermark extraction accuracy, without com- promising the visual quality of original outputs.
Generative model watermarking, Generative model water- marking has been explored to embed traceable information into outputs of GANs, ensuring copyright protection and supporting proactive forensies (Fei et al., 2022b; Yu et al., 2021: 2022b). For instance, Yu et al. (Yu et al., 2021) in- troduced method to embed watermarks into training data, while others (Fei et al., 2022b; Wu et al., 2020) enhanced robustness against manipulations like compression and crop- ping. However, these methods are limited to post-forgery de- tection, whereas our approach proactively prevents identity misuse by instructing generative models to refuse forgery.
Secure Swap is robust against various attacks, including image-level attacks, model-level attacks, and adversarial attacks. incurs minimal training overhead and allows multiple instances of the same model to be traced with unique watermarks without additional training.
2. Related Work
Unlike existing watermarking methods (training data based embedding (Yu et al., 2021), watermark decoder based supervised embedding (Fei et al., 2024; Fernandez et al.,
Compared to passive defense (Fei et al., 2022c; Yang et al., 2023; Yu et al., 2022c; Zhao et al., 2023), proactive de- fense entails implementing preventive measures before the
2

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
Pretrained G
Teacher model
nonPOI output www.eeeee.......................
nonPOT input
Knowledge distilation
Weight sharing
Lxn
Student model
LwM
G
Watermark decoder
PD Rasspont
F
Fr.
NonPOI imput
nonPOl output
Lys
Lpos
PÔT input
Watermark
POI output
Figure 2. An overview of the training pipelines of the proposed Secure Swap method.
2023)), our method directly encodes the watermark into model parameters. Although the watermark is also extracted from images, it becomes part of the model parameters. Once initialized by watermark, the model no longer requires external watermark input during inference. This supports efficient, scalable creation of uniquely watermarked model instances. Our design allows rapid deployment of distinct watermarked model instances scale.
use the Learned Perceptual Image Patch Similarity (LIPIPS) (Zhang al., 2018) as a vehicle distill the knowledge from G to Ges- We employ Knowledge Distillation (KD) to guide the teacher-student model framework, enabling the student model Gss to learn the non-POI outputs generated by the teacher model G. It ensures that the student model maintains visual quality and functionality comparable to the teacher model while incorporating the security features. The knowledge distillation loss is calculated based on the nonPOI outputs of the teacher model G and the student model Gss to ensure that the swapped images generated by Gss is visually close to those of G. The knowledge distillation loss LKD is given by:
3. Methodology
We propose Secure Swap, a method enabling GAN-based face swap models to redact POI and embed invisible water- marks for provenance tracing, promoting the governance of responsible face-swapping technology. An illustration of this scenario is provided in the Appendix. Fig. 2 provides an overview of Secure Swap, which depicts the training pipelines involved in transforming developer's 's face swap model G into released version Gss that incorporates curity functions into the original face swap functionality. We first create a replicate model Gss with the same archi- tecture and parameters as the developer's 'ss originally trained face swap model A small subnetwork with learnable pa- rameters, called the ID Passport layer, then inserted into Gss- The ID passport layer designed with the aim black out the faces of POI outputs and embedding watermark fea- tures into the nonPOI outputs. Then, Gss is retrained with a dataset that contains the original dataset of nonPOI im- ages and an additional set POI images, all with labeled identities. Four loss functions are used simultaneously for the training of Ggs.
H1 W ΣΣ
LKD Σ
Wi
(GG(, x)))
H.W. #=1==1
*i(Gss 2,) 2.))n,
where G(v,,, x2) and Gss vy) are the images generated by the teacher and student models, respectively with the same source image v, and target image z₁. (i( represents the activation of the 1-th layer of pre-trained network for the image J. wy is the learned weight of the 1-th layer. Hi and W, are the height and width, respectively of the (-th layer's activation map. || ||2 denotes the Euclidean L2 norm.
In addition, we use the identity distance between the source and swapped images as a vehicle to regulate the training of Gss. Let x., Ha and ys,t = G(x,, 2x)) denote the source, target, and generated images, respectively. Following the previous research (Shiohara et al., 2023), (Deng et al., 2019), popular tool used in the field of face recognition and verification, to extract identity information for the identity distance calculation. The identity loss be- tween a source image V, and swapped image ye,t is defined as follows:
adopt ArcFace
3.1. Face swap functionality preservation
In order to preserve the face swap functionality and visual quality of the originally trained model G when instilling new security features into the secure swap model Gss, we use G guide the training of Gss. More specifically, we
Lia 1 cos(FEa(ss); Paalys,)))
(2)
3

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
where Egg denotes ArcFace encoder and cos(u, v) is the
Final IDConv
cosine similarity between vectors u and
footune)
kernel
To ensure the student model Gss focuses only on changing the identity of the output to that of the source image without altering the background, expressions, garment, headdress and other contents of the target image, when the two dif- ferent source and target images are sampled from the same identity, the swapped image should look perceptually similar to the target image. This case (2; and x, belong to the same identity) of face swapping is analogous to reconstruction. The reconstruction loss free is calculated by the 11 distance between the target image 24 and the generated image 2s,t as follows:
wmm
IDCony,
IDConv;
kemel
Watermark encoder
IDCony
1
Kerool
Coofficient Generator
Wottemmrk
Gemerator
(Ujstrosm featurd)
Figure 3. Architecture of ID passport layer with the input upstream feature F and the output final feature F".
I Ua,ell if ID(a)) = ID().).
each training batch. The watermark encoder is trained to generate different watermark feature vectors WM from w and watermark decoder (see Fig. 2) is simultaneously trained to extract w embedded through F' into the nonPOI generated face swap outputs. The combined output of all IDConv blocks is convoluted with to produce the final feature map F" to the next layer of Gss. When the train- ing is completed, the model developer will input a unique w into the trained watermark encoder to generate a fixed feature vector WM program the IDConv in the passport layer of each trained Ggs instance. The entire trained wa- termark encoder will be detached from the passport layer of the trained Gss instance before it is released to the cus- tomer or licensee. The trained watermark encoder is used to attach to the passport layer of any trained Gss instance to program a different w into it. Details of the kernel generator. coefficient generator and watermark encoder involved in the construction of IDConv are provided in Fig. 4.
fr0
(3)
0
otherwise.
Aside from enforcing the forward consistency of swapping images with the same identity, we also take into account the backward consistency of converting the swapped image back to the original target image. This is accomplished by applying cycle consistency loss Leye in (4) to 20% (p = 0.2) of nonPOI source and target images with the same identity when training the Ggs. The cycle consistency loss Leye is
Laye ||-re G( 11
(4)
The loss functions, (2), (3) and (4), are combined into face swap loss for the nonPOI training of the Gss. The face
swap loss LFs is obtained as
Lis Lio + Lree + Leye.
(5)
For POI outputs, to enforce Gss to output black redacted image, the L2 distance between the POI output image and a black image (denoted as B) is minimized by the following POI loss function:
3.2. POI redaction and provenance traccability
As mentioned earlier, the only difference between the ar- chitectures of G and Gss is the ID Passport layer, which is added into the middle layer nearer to the output of Gss- The input image propagates through multiple layers of Gss in the form of feature maps before reaching the ID passport layer. The ID) passport layer converts the output feature map F of its preceding layer to the feature map F' that carries the identity and watermark features. F' is output to the succeeding layer in Gss. as shown in Fig. 2.
Cro
B (Es,
(6)
X. P. POI
where X, is a source image set.
For nonPOI outputs, a watermark decoder D', is trained with the Binary Cross-Entropy (BCE) loss to accurately extract the watermark from the nonPOI outputs. BCE loss is commonly used in binary classification tasks. It measures the difference between the predicted probabilities and the actual binary labels, and is calculated by:
The general architecture of the ID passport layer is shown in Fig. 3. consists of several parallel ID convolution layers, IDConv, to IDConv, Each kernel Kernel, of IDCon,, = 1,2, .n, derived from an upstream feature map F through kernel generator. Each IDConv; is also weighted by coefficient generated by a coef- ficient generator from F. and a watermark feature vector WM- = [wm], wm2, wmy] generated by a detachable watermark encoder from a binary watermark string w. Dur- ing training, a different randomly generated w used for
LwM
E .a MaanPPO w~ (0,1)
w; log (Dic (yo,t),
+ (1 w:) log (1 (1-o (Dw (vot)))))
where yo.1 = Gss( 21), Dw denotes the watermark de- coder, a(-) denotes the sigmoid activation function, n.p
4

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
Coefficient generator
Watermark encoder
C2
wm
Cooy 64XG4
GAP
Reshape
Relu
6x64x
0
32xx
Rola
128x64
Wm
Watermark
wmy
W
Weight
Weighe
Kernel generator
Cony, 64866
kernel, Initialize
IDConv;
h x
w
Comv 64x
Rela
Conn 64xx64
Comn 64x66
ReLa
Conv, 64x66
Kernel
Initialize
IDConv
Final IDConv
W
Cony, 64x66
kernel Initialize
IDConv
Figure Kernel generator (bottom). coefficient generator (top left) and watermark encoder (top right) for IDConv construction. The specific network structure parameters shown in the figure use BlendFace an example. Different face swap models may have variations parameters but the overall generation process remains basically the same.
the length of the watermark w, and w, is the i-th bit of the watermark. Note that during training, D. learns to decode different watermarks using randomly generated w as input to the passport layer of the model. Once the model has been trained, a fixed w is selected by the model developer to pro- gram the corresponding WM into the IDconv kernels of the passport layer through the watermark encoder. Thereafter, the watermark encoder is detached from II) passport layer.
The coefficients are derived from F through convolution (Conv) layer, an average pooling (Pool) layer and fully connected (FC) layer, followed by ReLu and FC layer after reshaping.
Watermark Encoder: It uses randomly generated binary bits during training and the hashed code of a developer's selected message after training to generate watermark vector WM through cascade of a FC-ReLu-FC layers. The elements, wm, um2.. wm f. are converted into weights in the f channel dimensions of each IDConv.
The overall loss function used for training the Secure Swap model is given by:
L= Crs + X2 + AzCror + A4
5. Experimental Results and Discussions
where dr to aa the hyper parameters used to regulate the respective loss functions.
5.1. Datasets and models
We utilize CelebA (Liu al., 2015), which contains over 10599 identities, serves nonPOI dataset to fine-tune the face swap model. To prevent the model from learning dataset-specifiic features instead of POI identity, two distinct datasets VGGFace2 (Cao et al., 2018) and PubFig83 (Kumar et al., 2009) are mixed to serve as POI dataset.
4. ID Passport Layer
Details of the kernel generator, coefficient generator and wa- termark encoder involved in the construction of IDConv are shown in Fig. 4. Their purposes are delineated as follows:
Different input images yield differ- ent weights, reflecting diverse -deltity-related information. This is leveraged to extract the identity-related latent fea- tures from F through a series of convolution layers and ReLU activation layers to produce multiple kernels. These kernels are initialized and used in subsequent IDConv lay- ers.
Kernel Generator:
We conduct experiments on various face swap models, in- cluding SimSwap (Chen al., 2020). Faceshifter (Li et al., 2020), BlendFace (Shiohara al., 2023), and MobileFS- GAN (Yu et al., 2022a). When training these models, we primarily adopt the default parameters explicitly specified in their official implementation. The weights of different loss terms in the total training loss Y1 A2. by and A are all set 1.0. Other training details are provided in Appendix.
Coefficient generator: When each IDConv inherits iden- tical kernel parameters from F, consecutive convolution layers could end up performing the same operation, which could hinder convergence. To prevent this, it is essential to introduce variations among the IDCony blocks by weighing the kernels with different coefficients, C1,02, ..", Ca-
5.2. Fidelity analysis
Fidelity refers to the faithful reproduction of the original face swap functionality from the face swap model to
5

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
between G and Ggs are generally very small, often within 0.05 for most models. This indicates that the success rate of face swapping for nonPOI images is preserved.
Table 1. Comparison of the absolute and relative identity and at- tribute fidelities. The bracketed figure for each metric refers to their difference.
Model
Type
Idemiity Similarit//Atrrin Similarity
To evaluate the second criterion, we evaluate the perceptual differences between images generated by G and Gss with Peak Signal-to-Noise Ratio (PSNR) (Hore Ziou, 2010), Structural Similarity Index Measure (SSIM) (Hore & Ziou, 2010) and LPIPS (Zhang al., 2018). Higher PSNR values indicate better similarity between two images. SSIM ranges from O to 1, with values closer to 1 representing higher similarity. LPIPS quantifies perceptual differences, where lower values indicate smaller differences. As shown in Table 2, we observed that with POI numbers varying from 128 to 1024, Gss consistently upholds the visual quality of swapped face images, with PSNR above 32, SSIM exceeds 0.95, and LPIPS averages below 0.05. This implies that the visual quality of nonPOI is preserved.
Are/Are-R
Blend/Bllend R
Att/Attr R
0.148/0.386
0.024/0.15 159
0.880D0.572
SimSSap
Gas
0. 0.148/0.899
0.030/0.229
0.866/0.565
(-0.000/+0.100
100))
(+0.005/+0.070)
(-0.014F-0.007)
FaceShifter
G
0.767/0.847
0.466/0.851 851
0.831/0.543
Gas
0.823/0.897
0.496/0.901
0.841/0.538
(+0.055/+0.050)
(+0.029/+0.0499
(+0.010X-0.005)
0.217/0.322
0.106/0.409
0,954/0.560
BleadFaace
Gas
08/88/3
0.056/0.435
0.950V0.558
(-0.029/+0.002)
(+0.050/+0.226)
(+0.004--00022)
Mobile FSGAN
G
0,488/0,537
0,367/0.611
0.866/0.557
Gas
0.523/0.591
0,404/0.683
0,857/0.553
(+0.034/40.054)
(+0.037/+0.071)
(-0.0091-0.044)
the secure model Ggs. Here, fidelity can be quantitatively assessed on two criteria: (1) the success rate of face swap for nonPOI, i.e., whether the identity of target image has been correctly altered to that of the source image; (2) the visual quality of the swapped images for nonPOI.
5.3. Effectiveness analysis
Effectiveness is verified by the success rate of POI redaction, and the watermark performance on nonPOI outputs. We adopt SRmmn from (Huang et al., 2022) for the measurement of the success rate of POI redaction. We also utilize MSE, ACCwM and TPR to evaluate watermark performance. The calculation details are shown in the Appendix. A higher SRmask ACCwM and TPR score indicates better effectiveness while lower MSE indicates better visual quality. Table 3 shows the SRmask: AcCWM, and TPR of various Secure Swap models with different number of POI IDs. In most cases, the average watermark extraction accuracy exceeds 99%. The exception is MobileFSGAN, which has a slightly lower Acowm of around 97%. Considering the high TPR, 97% accuracy is not an obstacle to effective provenance tracking.
Table Perceptual quality comparison of nonPOI face swapped images between Gss and G across varying numbers of POI IDs.
Number of POI
Models
PSNR
SSIM f
LPIPS
128
34.342
0.982
0.018
BlendFace
512
34.359
0
0.016
1024
34.157
0.976
0.010
128
33.965
0.965
0.034
SimSwap
512
34.854
0,964
0.062
1024
34.604
0.951
11.044
128
34.602
0.968
0.041
FaceShifter
512
34.168
0.968
0.031
1024
33.915
0.965
0.033
128
34.015
0.952
0.024
Mobile
512
33.390
0.957
0,023
FSGAN
1024
33.413
0.959
0.021
Table Effectiveness of POI redaction and watermark.
To evaluate the first criterion, we analyze the face swap outputs of G and Gss for the same pair of input images by comparing their absolute and relative similarity in terms of identity and attributes. To increase the confidence of our eval- uation, we use two different encoders, ArcFace (Arc) (Deng et al., 2019) and BlendFace (Blend) (Shiohara et al., 2023), as EN for the identity embedding. The calculation meth- ods for absolute/relative identity similarity and attribute similarity are provided in the Appendix. A high attribute similarity indicates that only the identity has been swapped while everything else remains unchanged. high identity similarity means that the swapped face's identity is similar to the expected identity. What we need to focus on is the difference in score between G and Gss on these metrics. The smaller the difference, the better the fidelity of Gss- As in Table 1, the changes in the identity distance (Arc, Are-R, Blend, Blend-R) R) and attribute distance (Attr, Attr-R) metrics
Models
POI IDs
SR 1
MSEJ
Acewu
TPR
128
1.000
0.000
0.987
0,9999
BlendFace
512
1.000
0.000
0.999
0,9999
1024
1.000
0.000
0.995
0.9999
128
1.000
0.000
0,998
0,9999
SimSwap
512
1.000
0.000
1.000
0,9999
1024
1.000
0.000
0.999
0,9999
128
1.000
0.014
0,990
0,9999
FaceShifter
512
1.000
0.051
0.999
0,9999
1024
1.000
0.021
1.000
1.0000
128
1.000
0.000
0.985
0.9998
Mobile ESGAN
512
1.000
0.184
0.976
0,9990
1024
1.000
0.000
0.982
0.9997
To further assess how our watermark embedding influences the model's generative performance, we conducted a com-
6

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
parison with the latest watermarking approaches such as arrtributing (Nie et al., 2023), Stable Signature (Fernandez et al., 2023), PluggableWM (Bao et al., 2024), WFWM (Fei et al., 2024). Higher PSNR and SSIM values indicate better watermark imperceptibility. As shown in Table 4 on Page 7 left column, our method shows excellent performance in imperceptibility.
maintains very high accuracies of 99.66% and 96.96% re- spectively. JPEG (quality factor 50) compression reduces the accuracy to 76.85%.
Fig. 5 shows strong robustness to Brightness and Contrast adjustments, with accuracy consistently above 98%, while more challenging attacks like Flip and Crop (crop rate = 0.4) result in lower accuracies of 79.38% and 81.43%, re- spectively.
Table 4. Performance comparison of different watermark methods. The model used for comparison is BlendFace.
To enhance the model's robustness, we retrain Gss by ran- domly applying one of the seven image processing attacks to the nonPOI results for each batch of training. The attacked output images were passed through the watermark decoder, which learns to decode the predefined watermark from these additional attacked samples with the watermark loss. We called the retrained Secure Swap model with enhanced wa- termark decoding capability the augmented model. The watermark extraction accuracy and PSNR values of the orig- inal and augmented Secure Swap BlendFace models are plotted against attack strengths for eight image processing attacks in Fig. 6.
Watermark works
PSNR f
SSIM f
Arrtributing (ICML2023)
28.149
0,903
Stable Signature (ICCV2023)
29.250
0.911
PluggableWM (TDSC2024)
31.284
0.939
WFWM(TIFS2124)
34.165
0.949
Ours
34.321
0.981
5.4. Robustness analysis
We consider two scenarios of model application: the first involves interactive online use via an API service, which allows applications to communicate with the model over the internet, and the second involves local inference by downloading the model with Secure Swap, as released by the developers, which has the flexibility to run the model in an offline environment. We perform robustness testing for both scenarios to evaluate the model's performance under different deployment conditions.
Figure 6 compares the watermark accuracy and PSNR of the original and augmented Secure Swap BlendFace models under varying attack strengths for eight image processing methods. It shows that the augmented model Gss (SS with aug) consistently outperforms the oon-aueeeteed model G (SS w/o aug) across various image attacks. For instance, in the JPEG Compression and Brightness Enhancement plots, the Accww of each augmented model remains high as attack intensity increases, while the non-augmented model's 's ac- curacy drops sharply. Under Gaussian Noise and Cropping attack, the augmented model demonstrates a more gradual decline Accww compared to the steeper drop of the non- augmented model. These results indicate that the augmented Secure Swap method can effectively resist strong image pro- cessing attacks that degrade image quality significantly.
Oniginal image
Nouse (0.2)
Blur (23) Acc: 96.96
JPEG (50) Aoc: 76.85
Brightness (0.5) Aot: 98,04
Aoc: 99.66
Model-level Attacks. When Gss is downloaded by users for local inference, it becomes potentially vulnerable to white-box model modification attacks. this scenario, attackers have full access to the model's architecture and parameters, allowing them to thoroughly analyze and manip- ulate it. Once they become aware of the secure swap mech- anism, they may specifically target the ID) passport layer. Possible attacks could involve removing the ID passport layer entirely or introducing perturbations to compromise its integrity. Such modifications could undermine the credi- bility of the model's 's watermark, leading to unauthorized or malicious use of the model. We evaluate the robustness of Gas against three types of white-hox model attacks. They are model quantization, model pruning, and noise addition to model weights, and mixture of these three attacks. The evaluation results with varying attack strength are shown in
Brightness (1.5) 99.87
Contrast 100.00
Contrast (1.5)
Flip Aoc: 79.38
(0.4) Acc: 61.43
Aoc; 98,70
Figure 5. NonPOI results under various image-level attacks.
Image-level Attacks. In the scenario where users interact with the system via an API service, their only means of com- munication with Ggs is through the input of images, which may expose Gss to various image processing attacks. These include unintentional image degradation during transmis- sion, such as compression artifacts introduced by formats like JPEG, as well as deliberate manipulations, such blur- ring, noise addition, or brightness and contrast adjustment. We provide nonPOI results under various image process- ing attacks on Secure Swap BlendFace model in Fig. 5 as exemplification. Despite significant perturbations such as Noise (anoive = 0.2) and Blur (kernal size 23), the model
7

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
JPEG Compression
Brightness Enhancement.
Gaussien Noise
Contrast Enhancement
369
300
106
P5/
PSWR
PSWR
13
50-
13
52
65
75
20
90
0.3
02
04
06
08
J4
15
504
0.6
6:0
01
632
634
C36
2.1
0.2
035
05
0.7
13 Enhirccment
03
03
Quality
Entannement
Cropping Artack
Gaussian Blurring
Horizontal Flip
Mixed Attack
103
110
100
33
33
50
103
50
02
02-
PSM
PSMR
mg
SNR
05 1.65
0.55
Flip
Stangtt
Figure 6. Robustness of original and augmented Secure Swap BlendFace model against different strengths of eight different image processing attacks. The grey line indicates average PSNR of output images after the attack.
Protectdd
Prstected Mdunity (abs)
Taeart
Swapped
widely used fast gradient sign method (FSGM) (Dong et al., 2018) and projected gradient descent (PGD) (Madry et al., 2018) adversarial attacks. We directly used the identity extractors embedded within each face swap model, such as ArcFace or BlendFace, to perform these adversarial at- tacks. Table 5 shows the SRmast of Gss obtained upon the attacks. For both FGSM and PGD, even when the perturba- tion amplification factor has increased to 0.1, Gss can still maintain a SRmac of 100% for POI redation. Fig. 7 displays the outputs of four POI after the FGSM adversarial attack. The first column shows the clean, protected images, which are the photos of the POI. The second column displays the POI images with the adversarial perturbations added. The third column displays the target photos, and the fourth col- umn shows the face swap results. Each row displays the results with different perturbation strengths of FGSM. The adversarial attacks may result in outputs that are not totally black, but the faces are still heavily redacted, and the POI cannot identified even when the adversarial perturbations are very high and visually perceivable.
Iduallt (cean)
Image
Image
0-005
Figure 7. POT results of Gas against white-box FGSM.
Fig. 9 of Appendix. As attack intensity increases, the PSNR of images from all four attacked models drops significantly, often below 20, and even below 10 under mixed attacks. In contrast, watermark extraction accuracy decreases much more slowly, with Acewsr remaining above 75%- rea- when image quality is severely degraded. This suggests that even with white- box access, attackers cannot remove the watermark without compromising the model's 's utility.
5.5. Ablation study
sonable verification threshold
Capacity of POI. To evaluate this, we increased the num- ber of POI ID to 1024, 2048 and 4096, and measure the protection performance of POI by SRmask and the quality of generated face swapped images for nonPOIs by LPIPS. The results are shown in Table 6, where the values of SRmast and I.PIS are given before and after the delimiter Y/' respec- tively. It is observed that the LPIPS scores for all Secure Swap models are very low and remain almost unchanged across all three numbers of POI IDs, indicating that their generated images are close to real images in both quality and diversity, and they can accommodate a large number of POI IDs without compromising the generated image qual-
Adversarial Attacks. Adversarial attacks, implemented by adding adversarial noise to images, is different type of image-level attack (Akhtar& & Mian, 2018 8). It presents severe threats to the identity of protected individuals if an attacker can use an identity extractor to perform adversar- ial attacks that successfully prevent SS from "recognizing" " the POI. To evaluate this feasibility, we consider the most
8

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
ity. Moreover, the success rates of POI redation are 100% throughout all evaluations. Combining with the LPIPS re- sults, it can be concluded that Gss can effectively protect at least 4K+ individuals as POI without compromising the model performance on nonPOI images.
that compared G, Gss requiring slightly more time per iteration but significantly less time for full training.
Table 8. Comparison of Runtime for Each Iteration and Complete Training of Models G and Gas
Running Time (G/ Ggs)
Model
Iteration
Training
Table 5, SRmnt of different Gse upon FGSM and PGD attacks
BlendFacee
1.15 / 1.41s
95h 50m / 7h 52m
with different perturbation strengths (c = 0.01 to 0.1).
SimSwap
0.67 /0.91s
93h 3m / 5h 4m
Models
Attack
0.01
0.02
0.05
0.1
FaceShifter
0.96 1.19s
132h 58m 6h 40m
MobileFSGAN
0.72/
48h 16m / 5h 47m
FGSM
1.00
1.00
1.00
1.00
BlendFace
PGD
1.00
1.00
1.00
1.00
Limitation and exploration: quantitative boundaries for newly added POI. In our experimental setup, POI number is approximately 1,000. When new POI is added, the model requires retraining to accommodate the updated dataset. However, as shown in Table 8, our method features a low training overhead, making the retraining process signifi- cantly less quantitative boundaries for newly added POI while ensuring watermark performance, we conduct experiments by adding POI numbers from IK to 10K. As shown in Table 9, the gradual increase of POI number does not significantly im- pact the model's ability to skip irrelevant POIs or maintain watermarking effectiveness.
FGSM
1.00
1.00
1,00
1.00
SimSwap
PGD
1.00
1.00
1.00
1.00
FGSM
1.00
1.00
1.00
1.00
FaceShifter
PGD
1.00
1.00
1.00
1.00
FGSM
1.00
1.00
1.00
1.00
MobilcFSGAN
Furthermore, to explore the
PGD
1.00
1.00
1.00
1.00
Table 6. SRaak/ / LPIPS of different face swap models with dif- ferent number of protected identities.
Models/ Identities
1024
BlendFace
1.000/0.01
/.000/U0 0.018
1.000/0.018
0.018
SinSwap
1.000/ 0.060
1.000/ 0.061
1.000 0.0063
Table 9. Performance of POI redaction and watermark embedding when POI number arising
FaceShifter
1.000 0.043
1.000/ 0,043
1.000 0.000
0.044
MobileFSGAN
0.000/0.021
1.000 0.002
1.000 0.024
POI number
1k
3k
5k
"10k
Training Dataset Requirement. For some POI, the avail-
ability of high-quality images may be limited. To study the impact of training dataset size on success rate of POI reda- tion, we trained the SS model with different sizes of POI training sets, each training set contains different numbers of images per POI, and tested the model performance on unseen POI images. The results are shown in Table 7, it shows that the Secure Swap models perform well in POI redation even when they are trained with relatively small datasets. For any POI, SKmast of more than 95% can be achieved with just 4 images, and increasing the dataset size to 16 images yields 100% success rate of POI redation.
R 1
1.000
1.000
0.995
0.989
FID 4
2.39
2.33
2.26
2.18
PSNR
34.321
34.402
34.026
33,897
6. Conclusion
Our work proposed robust Secure Swap mechanism to protect POI from unauthorized face swaps and embed wa- termarks in the generated images of nonPOI to facilitate provenance tracking. We designed an ID Passport layer to generate unique identity-specific feature maps from diverse upstream latent features of each pair of source and target images. Watermark is also embedded into nonPOI output images through the ID Passport layer by training a detach- able watermark encoder to transform randomly generated watermarks into weights of the ID passport layer, and a corresponding decoder to extract these watermarks from the generated face swapped outputs. The trained encoder can be attached any instances of the trained Secure Swap model to embed customer-specific watermark for provenance trac- ing. Notably, our method exhibits outstanding robustness against image-level, model-level, and adversarial attacks. In the future, we aim to adapt Secure Swap to emerging generative models to enhance security and promote ethical, accountabl applications of generative Al in legitimate face swap technologies.
Table 7. SRuast of different face swap models with different num- bers of training photos of protected identity.
Models Photos
2
4
8
16
BlendFace
0.965
0.982
0.990
0.994
1.000
SimSwap
0.942
0.962
0.987
0.989
1.000
FaceShifter
0,907
0.954
0.973
0.987
1.000
MobileFSGAN
0.915
0.931
0.970
0.994
1.000
Training Overhead. Table 8 presents the runtime required for training G and Gss on a workstation equipped with one NVIDIA A100 GPU (80 GB), an Intel Xeon Platinum 8358 2. GHz, and 1024 GB of memory. It shows

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
Acknowledgment
In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 943-952, Vancouver, BC, Canada, Jun. 2023.
This work was supported in part by the National Natural Science Foundation of China under Grants 62472454 and U2336208. This rescarch was also supported by the Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Award No. MOET2EP50200-0003. The authors would like to express their sincere gratitude for the generous support provided by these funding agencies.
Dai, Y., Fei, J., Wang, H., and Xia, Z. Attentional local contrastive learning for face forgery detection. In Inter- national Conference on Artificial Neural Networks, pp. 709-721. Springer, 2022.
Dai, Y., Fei, J., and Huang. F. Idguard: Robust, general, identity-centric poi proactive defense against face editing abuse. In 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 11934-11943. IEEE, 2024a.
Impact Statement
This work contributes the development of secure and ac- countable generative AI systems by proposing Secure Swap, robust face swapping framework that integrates identity protection and provenance tracking. The introduction of the ID Passport layer and detachable watermarking mechanism enables scalable deployment and individualized traceabil- ity. Our method enhances the reliability of face synthesis technologies under real-world attack scenarios. Looking forward, adapting this framework to diffusion-basee models aligns with the industry's 's move toward more secure, eth- ical, and controllable generative applications, supporting responsible innovation in Al-driven media generation.
Dai, Y., Fei, J., Huang, F., and Xia, Z. Face omron ring: Proactive defense against face forgery with identity aware- ness. Neural Networks, 180: 180:106639, 2024b.
Deng, J., Guo, J., Xue, N., and Zafeiriou, S. ArcFace: Additive angular margin loss for deep face recognition. Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 4690-4699, Long Beach, CA, USA, Jun. 2019.
Dong, et Boosting adversarial attacks with momentum. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 9185-9193, Salt Lake City, UT, USA, Jun. 2018
References
Agarwal, S. et al. Protecting world leaders against deep fakes. In Proc. IEEE Conf. Comput. Vis. Puttern Recog, Workshop, volume pp. 38-45, Long Beach, CA, USA, Jun. 2019.
Fei, J., Dai, Y., Yu, P., Shen, T., Xia, Z., and Weng, Learn- ing second order local anomaly for general face forgery detection. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 20270-20280, New Orleans, LA, USA, Jun. 2022a.
Akhtar, N. and Mian, A. Threat of adversarial attacks on deep learning computer vision: A survey. IEFE Access, 6:1 14410-1443, Feb. 2018
Fei, J., Xia, Z.,, Tondi, B.,, and Barni, M. Supervised gan watermarking for intellectual property protection. In Proc. IEEE Int. Workshop Inf. Forensics Secur. pp. 1-6, Shang- hai, China, December. 2022b.
Bao, H., Zhang, X., Wang, Q., Liang, K., Wang, Z., Ji, S., and Wenzhi, C. Pluggable watermarking of deep- fake models for deepfake detection. for Crttical Infrastructure Workshop@ IJCAI-24, 2024.
Fei, J., Xia, Z., Tondi, B., and Barni, M. Wide flat minimum watermarking for robust ownership verification of gans. IEEE Transactions on Information Forensics and Security, 2024.
Cao, Q. et al. VGGFace2: A dataset for recognising faces across pose and age. In Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., pp. 67-74, Xi'an, an, China, May 2018.
Fei, al. Learning second order local anomaly for general face forgery detection. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. USA, Jun. 2022c.
Chen, R., Chen, X., Ni, B., and Ge, Y. SimSwap: An efficient framework for high fidelity face swapping. In Proc. ACM Int. Conf. Multimedia, pp. 2003-2011, Seattle, WA, USA, Oct. 2020.
New Orleans, LA,
Fernandez, et al. The stable signature: Rooting water- marks in latent diffusion models, In Proc. IEEE Int. Conf. Comput. Vis., pp. 22466 22477, Paris, France, Oct. 2023.
Chu, B. et al. Protecting world leader using facial speaking pattern against deepfakes. IEEE Signal Process. Lett., 29: 2078-2082, 2022.
Hore, A. and Ziou, D. Image quality metrics: PSNR vs. SSIM. In Proc. IEEE Int. Conf. Pattern Recognit., pp. 2366-2369, Istanbul, Turkey, Jun. 2010.
Cozzolino, D., Pianese, A., Nießner. M., and Verdoliva, L. Audio-visual if deepfake detection.
10

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
Huang, H., Wang, Y., Chen, Z., Zhang, Y., Li, Y., Tang. Z., Chu, W., Chen, J., Lin, W., and Ma, K.-K. Cmua- watermark: A cross-model universal adversarial water- mark for combating deepfakes. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp- 989-997, 2022.
fingerprint domain. In 2023 IEEE International Confer- ence Multimedia and Expo (ICME), pp. 1187-1192. IEEE, 2023.
Wang. R. et al. Anti-forgery: Towards stealthy and ro- bust deepfake disruption attack via adversarial perceptual- aware perturbations. Int. Joint Conf. Artif. Intell., pp. 761-767, Jul. 2022.
Huang, Q. Zhang, J., Zhou, W., Zhang, W. and Yu, N. Initiative defense against facial manipulation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1619-1627, 2021.
Wu, H., Liu, G., Yao, Y., and Zhang, X. Watermarking neu- ral networks with watermarked images. IEEE Trans. Cir- cuits Syst. Video Technol., 31(7):2591-2600 Oct. 2020.
Kim, J., Lee, J., and Zhang, B.-T. Smooth-Swap: A simple enhancement for face-swapping with smoothness. In Proc. IEEE Conf. Comput. Vis. Pattern Recog.. pp. 10779- 10788, New Orleans, LA, USA, Jun. 2022.
Wu, X., Liao, X., and Ou, B. SepMark: Deep separable watermarking for unified source tracing and deepfake detection. In Proc. ACM Int. Conf. Multimedia, pp. 1190- 1201, Ottawa, ON, Canada, Oct. 2023.
Kumar, N., Berg, A. C.,, Belhumeur, P. N., and Nayar, S. K. Attribute and simile classifiers for face verification. In Proc. IEEE Int. Conf. Comput. Vis., pp. 365-372, Kyoto, Japan, Sep.2009. 2009.
Yang, W. et al. AVoiD-DF: Audio-visual joint learning for detecting deepfake. IEEE Trans. Inf. Forensics Secur, 18: 2015-2029, Mar. 2023.
Yu, H.. Zhu, H., Lu, X., and Liu, J. Migrating face swap to mobile devices: a lightweight framework and super- vised training solution. Proc. IEEE Int. Conf. Multime- dia Expo, pp. 1 1-6, Taipei, Taiwan, China, Jul. 2022a.
Li, L. et al. Advancing high fidelity identity swapping for forgery detection. In Proc. IEEE Conf. Comput. Vis. Pattern Recog.. pp. 5074-5083, Seattle, WA, USA, Jun. 2020.
Yu, N., Skripniuk, V., Abdelnabi, S., and Fritz, M. Artifi- cial fingerprinting for generative models: Rooting deep- fake attribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 14448-14457, 2021.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. Proc. IEEE Int. Conf. Comput. Vis., pp. 1730-3738, Dec. 2015.
Madry, A. al. Towards deep learning models resistant to adversarial attacks. In Proc. Int. Conf. Learn. Represen- tations, pp. 1658-1670, Vancouver, BC, Canada, April. 2018.
Yu, N. et al. Responsible disclosure of generative models using scalable fingerprinting. In Proc. Int. Conf. Learn. Representations, pp. 1-10, Virtual, Apr. 2022b.
Nie, G., Kim, C., Yang, Y., and Ren, Y. Attributing im- age generative models using latent fingerprints. In Inter- national Conference on Machine Learning, pp. 26150- 26165. PMLR, 2023.
Yu, P. et al. Improving generalization by commonality learn- ing in face forgery detection. IEEE Trans. Inf. Forensics Secur, 17:547-558 Feb. 2022c
Zhang, R. et al. The unreasonable effectiveness of deep features as perceptual metric. In Proc. IEFE Conf. Comput. Vis. Pattern Recog., pp. 586-595, Salt Lake City, UT, USA, Jun. 2018
Ruiz, N., Bargal, S. A., and Sclaroff, Disrupting deep- fakes: Adversarial attacks against conditional image trans- lation networks and facial manipulation systems. In Proc. Eur. Conf. Comput. Vis., pp. 236-251, Virtual, Aug, 2020. Springer.
Zhao, Ct al. ISTVT: Interpretable spatial-temporal video transformer for deepfake detection. IEEE Trans. Inf. Forensics Secur. 11.33--148,,, Jan. 2023.
Shiohara, K., Yang, X., and Taketomi, T. BlendFace: Re-designing identity encoders for face-swapping. In Proc. IEEE Int. Conf. Comput. Vis., pp. 7634-7644, Paris, France, Oct. 2023.
Van der Sloot, B. and Wagensveld, Y. Deepfakes: regulatory challenges for the synthetic society. Computer Law & Security Review, 46:105716-1059) Sep. 2022.
Wang, H., Fei, J., Dai, Y., Leng, L., and Xia, Z. General gan-generated image detection by data augmentation in

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
A. Responsible Face Swapping: Governance and Security
process facilitate identification of generative images from real images and traceability of downstream distribution to allow the authority to enforce ethical and responsible use of generative model at copyright or crime tribunal.
In this section, we first outline the specific scenario and problems to which our method applies under the foundation for responsible AI governance. To set the stage for the two security problems addressed by our solution, we then introduce the major stakeholders in this framework and their roles, incentives, privileges and constraints.
00010101
POI
naked eye analysis
nonPOr
Generative model
W M Extractor
A.1. Responsible face swapping governance
00010101
allecated
WM set
Sturistial Test
Our work aims to provide a feasible and efficient solution for model developers to responsibly release their face swap models under the responsible generative Al govemance framework. Three main parties are involved in this ideation. Developers: This role possesses full control of the model architecture. training strategy, and dataset (including the selection of POI). Developers release online API or open- source models to provide face editing services. Note that the training method used for the traceable watermarking component is proprictary to the developers and will not be publicly released for security reasons. Authority: This role possesses the right to require developers to fully disclose watermarking information and keep records of watermark- ing traceability when publishing models. They also have the right to investigate and collect evidence on the use of non-compliant models, and then penalize the subjects of the offense. Users: (a) regular users, this role only utilizes the provided service to swap ordinary faces that do not in- volve any POI. Their swapped images are used solely for legitimate purposes. (b) malicious users, this role has two intents. One intent is to directly forge POI images through the provided face swapping service. Upon finding that the POI swap failed, they may try to undermine the POT pro- tective functionality of the model when downloaded for local inference. The other intent is to directly forge nonPOI images for fraud. Upon finding that the swapped images carrying watermarks, they may try to remove the watermark or undermine the watermark generation mechanism. Their presence is the very reason for the problems tackled in this work.
Detection
Provenance tracking
Yt Al generated?"
Who genernteed it?'
X
0001010L
Figure 8. The process of image watermark (WM) detection and provenance tracking after deploying Secure Swap.
Fig. 8 epitomizes the essence of our proposition towards achieving celebrity protection and provenance tracing. Given the many-to-one relationship between faces and iden- tity, model modification centered on identity offers a more effective approach to ensure the model responds with dif- ferentiated behaviors towards different identities. A model developer can forbid POI swap attempt by preemptively forcing its model to redact POI face from the output in way that can be easily detected by naked eye. For the swapped output of nonPOI inputs, two questions are to be addressed in this work. 1. Detection: 'Is this image gener- ated by AI?", and 2. Provenance tracking: 'Who generated this image?'.
In the era of generative AI, criminal justice system is in- creasingly challenged by the indistinguishability between "original" content produced by generative models and real content. Watermarking is a well-accepted technology to proactively mark content to convey authenticity. Tech com- panies like Google, Microsoft, Meta, and Amazon have all pledged develop watermarking systems for identifying Al-generated content. Typically, each developer can have a unique predefined watermark registered with the authority. The developer's ' model can learn to embed this watermark invisibly into the generated images of nonPOI inputs. The watermark embedded in the swapped outputs can be de- coded by a watermark extractor for statistical analysis. If a watermark is detected in a suspect image, it can be deter- mined that it is forged image generated by face swap model. If the detected watermark matches the watermark of specific model, it can be inferred that the image was generated by that particular face swap model.
A.2. Celebrity protection and provenance tracing
We tackle the security of responsible face swapping on two fronts. Firstly, due to the incredible spotlight on celebrities, we aim to provide security to celebritics (POI) in face swap models to prevent their facial image from being abused. Sec- ondly, we aim to make each developer's model identifiable from its generative outputs for forensic analysis to track the authenticity and source of generative images of ordinary individuals (nonPOI). Under the responsible generative AI governance framework, developers are advocated to insti- tute transferable forensic features into their model building
Unlike the methods discussed in Sec. 2, we aim to empower model developers to embed proactive security features di-
12

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
rectly within face swap models. This will allow the mod- els to preemptively block any malicious attempts targeting specific identities while also watermarking the generated content. By addressing potential threats at the source, it prevents the creation of fake images without requiring any preprocessing of input images. Furthermore, we would like to safeguard all images containing any protected identities, rather than selectively targeting certain images. This to ensure that the model consistently applies security measures across all relevant outputs. For practical implementation and cost efficiency, it is also important to reduce the devel- opment expenses while maintaining high levels of security and reliability by training the model once but the outputs of multiple instances of the same model can still be embed- ded with their respective unique instance watermarks for provenance tracking.
The True Positive Rate (TPR) can be evaluated follows:
11 (7²²) Pie (1 Pw)
TPR(T)
(10)
where Pw represents the experimental watermark detection accuracy. Using (9) and (10), we can balance the FPR and TPR to achieve a more accurate watermark detection.
Watermark for provenance tracking. Each watermark w hashed from the developer's s selected copyright message can be assumed to be a i.i.d binary string drawn from the set {0, 13"" for embedding into a model distributed to a downstream commercial customer. Let wi[] denotes the watermark embedded into the model G, distributed to the customer D;, where i = 1,2, .N and N is the total number of customers. If a model is abused for generating face swapped image x for fraud, the specific downstream commercial customer D; can be reliably traced by extracting the watermark w' from the face swap image x to comparing against {wo(1) w/ w=[2).... [N]]
Watermark detection. Fach instance (copy) of a trained secure swap model Gss is watermarked before it is released. To watermark an instance, unique copyright or license mes- sage selected by the developer is first hashed into a binary watermark w of length Ray- Then the trained watermark en- coder attached to the ID passport layer of the instance to program w into its !Dcony kernels. The watermark encoder is then detached before the watermarked instance is released. To detect if the watermark w E {0, 1 1 programmed into a distributed instance G is present in a nonPOI image x, we can pass x to the trained watermark decoder D.,, to ex- tract the watermark w'. The detection test (Fernandez et al., 2023; Yu et al., 2021) checks the number of matching bits, Match(w; w'), between and wo against non-negative threshold S nw- If Match(w, w, w') 2 т, the image x is highly probable to be generated from G..
For provenance tracking, we have N detection hypotheses. If all N hypotheses are disproved, we infer that x is not gen- erated by any of the models distributed to the downstream commercial customers. Otherwise, we attribute the image x to a specific model G3 of the downstream commercial customer Dj. where
j = arg max { (Match(w', w(()) 11,,2,.., NN
(11)
Given that N tests are being conducted, the probability of encountering false positives in this detection process is elevated. The global FPR at a given threshold T is:
To increase the confidence of watermark detection, we per- form a hypothesis test on T. We test the statistics of the alternative hypothesis H₁: "x was generated by a model GA instance" " against the null hypothesis Ho: "r was not generated by model GA". Under Ho, assume that all bits w Vi E Nw] are independent and identically dis- tributed (i.i.d.) Bernoulli random variables with parameter 0.5. Then Match(w,u w') follows a binomial distribution with parameters (nw; 0.5). This assumption has been exper- imentally verified by (Fernandez et al., 2023). Noted that if a wotwaatrmereed image yields a watermark matching score greater than the threshold r. the image is considered false positive sample. The False Positive Rate (FPR) is the probability that Match( w. w²) takes value bigger than the threshold r. is obtained from the Cumulative Distri- bution Function (CDF) of the binomial distribution, and a closed-forn expression of FPR can be expressed with the regularized incomplete beta function I_ (a. b) as follows:
FPR(T, = - (1 - FPR(T)) = N FPR(T). (12)
Through careful management of detection thresholds and multiple hypothesis testings, reliable provenance tracking can be achieved.
B. Training details.
When training these models, we primarily adopt the default parameters explicitly specified by the authors in their official implementation. For SimSwap (Chen et al., 2020), we train the model with batch size 32 for 500,000 steps using the Adam optimizer with a learning rate of 0.0004, beta is 0.0, and beta 2 is 0.999. For Faceshifter (Li et al., 2020), we train the model with batch size 16 for 500,000 steps using the Adam optimizer with a learning rate of 0.0004, beta 1 is 0.0, and beta 2 is 0.999. For BlendFace (Shiohara et al., 2023), we train the model with batch size 16 for 500,000 steps using the Adam optimizer with a learning rate of 0.0001, beta 1 is 0.5, and beta 2 is 0.999. The discriminator updated five
FPR(T) = PMMatch(w, w/) ) > r(Ho) = (T -T).
(9)
13

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
times for each generator update. For MobileFSGAN (Yu et al., 2022a), we train the model with batch size 64 for 500,000 steps using the Adam optimizer with learning rate of 0.0002, beta is 0.5, and beta 2 is 0.999. The discriminator is updated twice for each generator update. The learning rate for models are linearly decayed to zero from 250,000 steps.
follows:
Σ,En Mas) G(z)(1))) Gss (z) (z)
Lz(G, Gss)
E.E,
(15)
where x (Iz, (e) are the input faces, and (i. is the pixel
coordinate of x. The binary mask M(s.j) is determined by the pixel discrepancy between the edited image G(x) and the original image x at (i, in (16).
C. Fidelity analysis
The identity distance is calculated by the cosine similarity between the normalized face embeddings of the source and swapped images (Shiohara et al., 2023). A higher distance implies a successful face swap. To increase the confidence of our evaluation, we use two different encoders, ArcFace (Arc) (Deng et al., 2019) and BlendFace (Blend) (Shiohara et al., 2023), as Eu for the identity embedding. Apart from identity distance, we also utilize Attribute (Aur), an at- tribute extractor, to calculate the attribution distances be- tween the source and swapped images. A smaller attribute distance indicates a better face swap performance. To clim- inate the influence of irrelevant parameters such as vector size and vector dimension, we calculate the cosine similar- ity of extracted feature vectors for ArcFace BlendFace and Attribute. Besides similarity in identity between the source and swapped images, it is crucial that the swapped images are dissimilar to target images visually. For a more compre- hensive evaluation, we also calculate the relative identity and attribute similarity that account for both the source and target images, which are annotated by "-R". as follows (Kim et al., 2022):
- 11,
if (G((,,)0)0..) - (#a) 0.5 0, otherwise
M(s
(16)
E. Model-level attack details
We evaluate the robustness of Ggs against three types of white-box model attacks. They are model quantization. model pruning, and noise addition model weights, and mixture of these three allacks. We also vary the attack strength of each type of attack. For noise addition, we added Gaussian noise with varying standard deviations (Std Dev or noise). For pruning, we set a certain ratio of weights to zero based on their absolute values. For quantization, we set the weight precision to decimal numbers with varying numbers of fractional digits. The evaluation results with varying attack strength are shown in Fig. 9.
F. Analysis of loss functions.
We balance multiple losses by assigning equal weights to each term. This is intentional and based on the following reasons: The watermark encoder, kernel generator, and co- efficient generator have their Icarnable parameters, which enable our losses to achieve stable convergence without re- quiring careful weighting. Besides, all components of the model are optimized jointly and simultaneously. The over- all loss function consists of face-swapping losses (FS), a knowledge distillation loss (KD), a POI-specific loss, and a WM loss. FS and KD are applied to nonPOI samples, while the POI and WM losses act ID protection and watermark embedding, respectively, which are non-conflicting. Equal weights are assigned to all components based on three ob- servations: (I) Numerical scales remain comparable across losses; (II) FS and KD address similar objectives, and all losses operate on disjoint data partitions (nonPOI vs. POI); (III) The WM loss modifies only imperceptible features, pre- serving image quality and avoiding interference with other losses.
S(I,, Gss(I,, 21))
S-R S(x,, Gss(ut,, 21)) S(x,, Gss (x..
(13)
where S here refers to the similarity of identity or attribute between two face images with range [-1,
D. Effectiveness analysis
SRmask can be calculated by:
1 THD
SRmakk
(LL Gss) > 0.05), (14)
where Typ denotes the number of POI IDs. 1(-) is an in- dicator function, which produces 1 if the condition in the argument is true and 0 otherwise. L22 G, Gss) denotes the weighted Le norm between the outputs of G and Gss. Fol- lowing (Huang et al., 2022), we consider the POI redation to be successful if Lz( G, Gss) > 0.05. The weighted L2 norm in (14) can be determined by the pixel-level discrep- ancies between G and Gss when fed with POI images as
We also acknowledge the importance of each loss compo- nent introduced to address a specific task: (I) FS loss and KD loss: enhances visual fidelity in face-swapped images. (II) POI loss: enforces protection of POI identities. (III)
14

---

Robust Secure Swap: Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
Model Naise (ace)
Model Pruning (BlendFace)
Model Quantization (BlendFacel,
Model Mixed Attack (Blend Race
BlendFace attucks
11
Ozz PSWn
PSHA
0.30
3.008
93
8-1
Decimal
Attack Strangth
(SimSwap)
Model Pruning (S (Smpp
Quantizati on (SimSwap)...
Mixed Attack (SimSwap
25 Anta
SEAr PSW
SinSwap attacks
30
00
-25
PSWA 3
PSWA
Acgee
Dam ma
8004
Std Dev
Pruning Fate
Dedmal Digits
Attack Strength
Model Noi so (FaceShifter)
Model Pruning (FaceShiftorl
Model Quantization [FaceShiftrr]
Madel Mixed Attack (FaceShiftor)
FaceShifter attacks
Artre
ACCIN
30
50
53
1re4
0.30
1017
0.208
007
8-1
Sto Day
Pruning Rata
Digits
Strength
Model Naise (MabileFSGAN)
Model Pruning (MobileFSGAN)
Model Quantization ti (MobileFSGAN)
.Model Mixed Attack (MobileFSGAN)
MobilePSGAN attacks
193
(72
A
FSM
-25
PSWR
Acger
8004
3.008
C3
Std
Pruning Rate
Decimal Digits
Attack Strength
Figure 9. Watermark accuracy (Accwa)) of Secure Swap (a) BlendFace (b) SimSwap (c) FaceShifter and (d) MobileFSGAN models after attacks model noising (Ist column), pruning (2nd column), quantization (3rd column) and mixed attack (4th column).
WM loss: embeds watermark into the output of nonPOI images.
G. Stability testing of our watermark performance
To evaluate their contributions, we took BlendFace as an example and conducted ablation studies by removing each loss independently in the table below. The results show significant performance degradation in the corresponding tasks on the removal of any loss term.
Our watermarking method demonstrates promising perfor- mance in both effectiveness and usability. In terms of ef- ficiency, by binding the watermark to model weights, our method enables scalable watermarking for efficient deploy- ment in distribution scenarios. To address the potential influence of randomness, we report the average accuracy and standard deviation over 1,000 different embedded wa- termarks, as shown in the Table 11. The low standard de- viations indicate consistent performance across watermark instances. These results suggest that our method remains robust and is not affected by randomness.
Table 10. Performance comparison of different loss configurations.
Loss
PSNR
WM_Acc SRmask
w/o FS loss
32.3
0.93
0.98
w/o KD loss
27.7
1.00
0.99
w/o POI loss
34.5
0.00
0.99
w/o WM loss
35.2
1.00
0.51
15

---

Robust Secure Swap:
Responsible Face Swap With Persons of Interest Redaction and Provenance Traccability
Table 1I. Performance metrics of different models.
Model
Bace
SimSwap
FaceShifter
MobileESGAN
Acc
99.48
99.93
100.0 0.00
98.24
std
0.33
0.29
0.30
H. Training Overhead.
Table presents the runtime required for training G and Gss on a workstation equipped with one NVIDIA A100 GPU (80 GB), Intel Xeon Platinum 8358 CPU@2.60 GHz, and 1024 GB of memory. It shows that compared to G, Gss requiring slightly more time per iteration but significantly less time for full training. Gss takes slightly longer iteration time (i.e, the runtime for each batch of training data) than due to the added ID passport layer, but its convergence time significantly shorter than that of G. This because Gss inherits the pretrained weights of G before it is trained. Take BlendFace as an example, G requires 300,000 iterations to converge, while Gss needs only 20,000 iterations to achieve convergence. Therefore, even though Gss has a slightly higher iteration time than G. its overall training time around or more than an order of magnitude lower than that of G.
16


Reviews:
[{"review_id": "qa7yrCNNsV", "reviewer": "Reviewer_JXWv", "summary": "The paper presents a method that incorporates a trainable adapter into an existing GAN-based face-swapping pipeline to safeguard the privacy of Persons of Interest (POIs) by redacting their appearance in the output. Additionally, it embeds a watermark for traceability while preserving identity transferability for non-POI swaps. Experimental results demonstrate the effectiveness of the method in achieving POI redaction, watermark embedding, and robustness against potential attacks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **(A) Method & Evaluation Criteria**\n- **A-Q1: Performance on FFHQ.**\nWe did not consider FFHQ as it lacks ID labels to support direct POI reduction evaluation. Still, we performed evaluation via data augmentation. To evaluate FFHQ as nonPOI, we trained models with FFHQ and calculated the quality difference between Gss and G outputs, as shown in columns 2 to 5 of the table below. We also present the results of directly inferring on FFHQ without training, as shown in columns 6 to 9. To evaluate FFHQ as POI, we randomly selected 128/512/1024 images from FFHQ, treating each image as a POI. For each POI, only one image was used for training, and 100 augmented samples were generated for test. Results (columns 10 to 12) show that even with one image per POI for training, we achieve over 85% protection success.\n\n|Model |PSNR |SSIM|LPIPS |FID|PSNR'|SSIM'| LPIPS'|FID'|POI=128|POI=512|POI=1024| \n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| \n|SimSwap|34.60|0.97|0.01|15.67|34.11|0.97|0.01|15.21|0.98|0.92|0.87 | \n|FaceShifter|34.23|0.97|0.03|17.42|33.98|0.97|0.03|17.05|0.98|0.90|0.89|\n|BlendFace|34.24|0.96|0.02|12.81|34.08|0.96|0.03|12.59|0.97|0.91|0.88|\n| MobileFSGAN|34.27|0.95|0.01|16.32|33.89|0.96|0.02|15.80|0.94|0.89|0.86|\n- **A-Q2: ArcFace with threshold.**\nFor time cost of new POI, see D-Q3.\nArcFace-based defense is vulnerable to model-level attacks, as it didn't modify the model.\nOnce the matching is bypassed, protection fails. \nIn contrast, our method is robust and any model-level attacks would destroy generation capability (Fig. 9 in Appendix). \nMoreover, threshold-based methods suffer from a fixed threshold, which may result in high FPR or FNR if chosen improperly. \n- **A-Q3: Loss analysis.** See Review tYBZ A-Q3\n- **A-Q4: ID passport layer.** See Review tYBZ A-Q1&2\n***\n### **(B) Experimental Design and Analyses**\n- **B-Q1: Diverse POI images.**\nVGGFace2 and PubFig83 (evaluated in our paper) vary in illumination, hairstyles, poses, age, etc. Based on the results, we consistently achieve protection effectiveness across these diverse conditions.\n- **B-Q2: Watermark performance.**\nWe reported 3 FID scores on CelebA in the table below: FID from baseline model G, FID from our method, and FID from our method with watermark only (no POI defense). Our method caused a small FID increase of around 1, which remains visually acceptable. When POI defense is removed and only watermark is applied, the FID increase is marginal and stays within a 1-point range compared to baseline, indicating that our method has a small impact on the image in both POI reduction and watermark.\n\n|Model|G (no WM)|Gss (WM + POI)|Gss (only WM)|\n|:-:|:-:|:-:|:-:|\n|SimSwap|12.95|14.52|13.12|\n|FaceShifter|14.67|15.98|15.34|\n|BlendFace|10.58|11.20|11.07|\n|MobileFSGAN|13.96|14.65|14.35|\n\n- **B-Q3: FPR.**\nWe evaluated the FPR of POI redaction. Our evaluation consistently yields a FPR of 0: throughout our experiments, we did not observe any failure cases in which a nonPOI sample was mistakenly treated as a POI and subjected to redaction. We determine success using SRmask (by Eq. 15 in Appendix, with the threshold=0.05). \n\n***\n\n### **(C) Essential References**\nSee Review HFDi A-Q1\n### **(D) Questions for Authors**\n- **D-Q1: Adaptation to DM.**\nOur method can be applied to diffusion based faceswap models. We can insert the ID passport layer into the decoder (rather than the UNet) and finetune the decoder to prevent POI generation. We evaluated this design on DiffSwap [1]. Table below reports the protection performance and fidelity for protecting 128 POI with 16/32 training images, which shows unaffected image quality and near-perfect protection with only 32 training images per POI.\n\n|Metric|128(16)|128(32)|\n|-|-|-|\n|SRmask|0.975|0.996|\n|LPIPS|0.019|0.019|\n\n[1] DiffSwap: High-Fidelity and Controllable Face Swapping via 3D-Aware Masked Diffusion\n- **D-Q2: Diverse POI images.**\nSee (B) B-Q1\n- **D-Q3: New POI.**\nSince the model has already converged on old POI(1024) and watermark, only a little finetuning is needed. Table below shows the computation time for 10 or 100 new POI, when old POI is or not involved. \n\n|Model|1024+10 (old POI involved)|1024+100 (old POI involved)|1024+10 (old POI not involved)|1024+100 (old POI not involved)|\n|-|-|-|-|-|\n|SimSwap|~1h 20m|~2h40m|~40m|~1h10m|\n|FaceShifter|~1h10m|~2h20m|~1h|~1h15m|\n|BlendFace|~1h10m|~2h20m|~50m|~1h5m|\n|MobileFSGAN|~1h10m|~2h|~35m|~1h10m\n\nThe table below shows SRmask score (old/new) on old and new POI, with or without old POI included during finetuning. Results show that once old POI are involved in finetuning, their protection remains effective.\n|Model|1024+10 (old POI involved)|1024+100 (old POI involved)|1024+10 (old POI not involved) | 1024+100 (old POI not involved)|\n|-|-|-|-|-|\n|SimSwap|1.0/1.0|1.0/1.0|0.76/1.0|0.62/1.0|\n|FaceShifter|1.0/1.0|1.0/1.0|0.68/1.0|0.60/1.0|\n|BlendFace|1.0/1.0|1.0/1.0|0.75/1.0|0.69/1.0|\n|MobileFSGAN|1.0/1.0|1.0/1.0|0.72/1.0|0.65/1.0 \n- **D-Q4: FFHQ.**\nSee (A) A-Q1", "claims_and_evidence": "The two major claims—POI redaction and watermark embedding without compromising output quality—are well-supported, except for some ambiguity in POI redaction when different POI images are used as model inputs during inference.", "methods_and_evaluation_criteria": "1.\tThe FFHQ dataset includes face images with extreme poses and varying ages. It would be beneficial to show results on this dataset as well to evaluate whether the method performs consistently (Both with training and without training).\n2.\tHow long does it take to train the model to redact a new POI? Additionally, why not simply compare the incoming source image with stored POI images using ID similarity (e.g., ArcFace) and remove matches based on a threshold? While this approach requires storing images, wouldn’t it be more efficient than lengthy model training?\n3.\tThe method incorporates numerous loss components, but their individual significance is not thoroughly analyzed.\n4.\tThe motivation behind the design choices and the positioning of the ID passport layer within the face-swapping model remain unclear.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "1.\tDid you test POI redaction with diverse images, such as a celebrity’s photo under dark lighting, with a different hairstyle, or at a slightly different age? Can the model still effectively redact these variations?\n2.\tA common face-swapping evaluation metric is FID (Fréchet Inception Distance), which measures output image fidelity/quality. How does adding the watermark impact FID?\n3.\tDid you evaluate the False Positive Rate (FPR) of the redaction? Specifically, what percentage of redaction occurs when a non-POI is used as the source image?", "relation_to_broader_scientific_literature": "The idea is valuable for addressing privacy concerns (POI redaction) directly during face-swapping generation, unlike most existing methods that apply redaction after the forged image has already been created.", "essential_references_not_discussed": "The essential references are appropriately discussed to provide a clear understanding of the contributions. However, further discussion is needed on the difference between previous generative model watermarking techniques and the approach presented in this paper. (Lines 73-76 [Right] seem to refer to POI redaction, not watermarking, I believe.)", "other_strengths_and_weaknesses": "Strengths:\t\n        1.\tThe task is valuable to the research community.\n\t2.\tThe experiments are extensive.\n\nOther Weaknesses:\n\t1.\tFigure 1 is not referenced in the text.\n\t2.\tThe works in the tables should be properly cited.\n\t3.\tIn Table 3, are the POI IDs 128, 512, and 1024 for SimSwap?", "other_comments_or_suggestions": "1.\tCorrect the spellings (E.g Tab 4, Ln262 [right]  Arrtributing-> Attributing)\n2.\tTable 4 can be summarized in one sentence as everything are 1.\n3.\ty_{s,t} -> x_{s,t} in eq.3,4.", "questions_for_authors": "1.\tWhy isn’t this method suited for diffusion-based face swapping? Diffusion models are a powerful alternative to GAN-based methods, offering excellent image quality and are currently a leading generative tool. Could this method be integrated into diffusion U-Nets?\n2.\tDid you test POI redaction with various images during inference, such as a celebrity’s photo under dark lighting, with a different age, or hairstyle? Can the model still redact these variations effectively? (This was previously asked)\n3.\tHow long does it take to train the model to add a few new POIs? What impact does this have on the redaction of existing POIs? For instance, if A, B, and C are the previous POIs and now you need to add X, Y, and Z, will redaction for A, B, and C still be valid? A simpler experiment could involve adding PubFig83 followed by VGGFace2 as POIs and validating on both datasets.\n4.\tWhat are the results on an untrained dataset, such as directly inferring on FFHQ? I want to understand the impact of the Robust Secure Swap method on the generalizability of the original face-swapping model.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2yg1zi7TJ", "reviewer": "Reviewer_HFDi", "summary": "This paper introduces a method to prevent unauthorized face swaps involving persons of interest (POIs), while embedding an invisible watermark in non-POI results. Experiments demonstrate that the method maintains the performance of the original face swap model, effectively prevents unauthorized swapping, and ensures watermark-based provenance. The authors also conducted robustness tests under various attack scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **(A) Other Weaknesses**\n- **A-Q1(I): Comparisons with other methods addressing unauthorized face swaps and watermark.**  \n\n**Compare with anti-faceswap methods.** Existing methods to address unauthorized face swaps fall into two categories: proactive protection and post-hoc detection. Post-hoc detection suffers from inherent latency and cannot effectively prevent image misuse. Current proactive methods primarily rely on adversarial attacks by adding perturbations to protected images, such as POI photos, to prevent forgery [1][2][3]. These approaches offer image-level protection, require costly preprocessing, and cannot scale to large volumes. In contrast, our work targets identity-level protection. Given the one-to-many nature between identity and images, our method supports scalable protection without preprocessing and achieves higher robustness. As suggested, the following table compares our defense success rates (SRmask) against existing methods when protecting 128 images with different identities.\n\n| Model| Disrupting Deepfakes[1]| Initiative Defense[2] | CMUA[3] | Ours |\n|:-:|:-:|:-:|:-:|:-:|\n| SimSwap | 0.93 | 0.94  | 0.99 | 1.00 |\n| FaceShifter | 0.91 | 0.90  | 0.94 | 1.00 |\n| BlendFace  | 0.85 | 0.84 | 0.89 | 1.00 |\n| MobileFSGAN   | 0.90  | 0.92 | 0.95 | 1.00 |\n\n**Compare with watermark methods.** Unlike existing watermarking methods (training data based embedding [4], watermark decoder based supervised embedding [5,6]), our method directly encodes the watermark into model parameters. Although the watermark is also extracted from images, it becomes part of the model parameters. Once initialized by a watermark, the model no longer requires external watermark input during inference. This supports efficient, scalable creation of uniquely watermarked model instances. Our design allows rapid deployment of distinct watermarked model instances at scale.\n\n[1] Disrupting Deepfakes: Adversarial Attacks Against Conditional Image Translation Networks and Facial Manipulation Systems  \n[2] Initiative Defense against Facial Manipulation  \n[3] Cmua-watermark: A cross-model universal adversarial watermark for combating deepfakes  \n[4] Artificial fingerprinting for generative models: Rooting deepfake attribution in training date  \n[5] Wide flat minimum watermarking for robust ownership verification of gans  \n[6] The stable signature: Rooting watermarks in latent diffusion models\n\n- **A-Q1(II): Watermark performance improvement and stability.**\n\nOur watermarking method demonstrates promising performance in both effectiveness and usability, as shown in Figure 4 in our main paper and Figure 9 in the Appendix. \nIn terms of efficiency, by binding the watermark to model weights, our method enables scalable watermarking for efficient deployment in distribution scenarios.\nTo address the potential influence of randomness, we report the average accuracy and standard deviation over 1,000 different embedded watermarks, as shown in the table below. The low standard deviations indicate consistent performance across watermark instances. These results suggest that our method remains robust and is not affected by randomness.\n\n|Model|BlendFace|SimSwap|FaceShifter|MobileFSGAN|\n|-|-|-|-|-|\n|Acc|99.48|99.93|100.0|98.24|\n|std|0.33|0.29|0.00|0.30|\n\n- **A-Q1(III): Flip accuracy improvement by suggested data augmentation strategies.**  \n\nAccording to your suggestion, we used a stronger data augmentation to improve the watermark robustness against horizontal flip attack. \nSpecifically, we increased the probability of applying horizontal flipping to 30% in each training step, and evaluated the average watermark accuracy on 1k CelebA images, as shown in the table below. It demonstrates that flip accuracy has improved after applying data augmentation. Additionally, to explore the impact of image quality, we calculated the PSNR between the watermarked and non-watermarked images. Results in the table below show that augmentation did not introduce any noticeable degradation in image quality. Overall, thanks to the reviewers' suggestion, we adopt data augmentation. This strategy improves flip accuracy.\n\n| Method| SimSwap | FaceShifter | BlendFace |MobileFSGAN |\n|-|-|-|-|-|\n| Horizontal Flip|93.7|93.9|94.1|93.7\n| PSNR| 33.7| 34.7 | 34.3 | 34.1\n\n***\n\n### **(B) Questions For Authors**\n\n- **B-Q1: Clarification of horizontal flipping appears as a straight line.**\n\nIn our experiments, unlike Gaussian noise or adversarial perturbations, which can be applied at varying levels, flipping is a binary transformation. Thus each image has only two possible states: flipped or not flipped, without any gradual increasing attack intensity.  Consequently, the result is shown by a straight line in our evaluation, rather than a curve that reflects increasing attack strength. We will clarify this in the paper to avoid potential misunderstandings.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. They are good.", "experimental_designs_or_analyses": "Yes. Tables 1 and 2 demonstrate that the model maintains the performance of the original face swap model. Tables 3 and 4 show the effectiveness of preventing unauthorized face swaps and successfully embedding watermarks. Figures 4, 5, and 6, along with Table 5, present the results under various attack scenarios.", "relation_to_broader_scientific_literature": "This paper primarily focuses on the security aspect, while incorporating the face swap task, which is a form of image editing.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The topic is both interesting and important in the field of security. This paper demonstrates that the proposed method can prevent unauthorized face swaps involving persons of interest (POIs) and embed an invisible watermark in non-POI results. \n\nHowever, the paper could be improved in several areas. For instance, it does not include comparisons with other methods that address unauthorized face swaps involving POIs. Additionally, the performance improvement attributed to the watermarking technique appears marginal and may be influenced by randomness. Moreover, the flip accuracy for non-POI results under various image-level attacks is relatively low, which could potentially be improved through data augmentation strategies.", "other_comments_or_suggestions": "This is an interesting topic with potential applications across various domains—not only in face swapping, but also in talking head generation, digital humans, and related fields.", "questions_for_authors": "Figure 5 shows the results of horizontal flipping on the augmented SecureSwap BlendFace model. However, it appears as a straight line, which differs significantly from the other curves—could the authors clarify the reason for this behavior?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDAT4w9bqC", "reviewer": "Reviewer_tYBZ", "summary": "The work proposes a novel method to transfer a general face swap method to a secure face swap method, where POI is rejected and non-POI is passed to generate swapped face image with a tracable, unique, invisible watermark. Specifically, an ID Passport layer is proposed to recognize if the input face image is POI and a detachable watermark encoder and decoder  is trained to insert a tracable and invisible watermark into a swapped face of non-POI face image. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\nAfter reading rebuttal, my final rating is weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **(A) Other Weaknesses**\n\n- **A-Q1: ID passport layer.**\n\n**(I) Motivation**. Our design of the ID passport layer is driven by the goal of enabling ID-sensitive processing within the faceswap model. It is motivated by the need to differentiate outputs between POI and nonPOI based on ID information contained in the intermediate feature F. To this end, the feature F not only serves as input but also partially defines the convolutional parameters. This design enables the model to dynamically adapt the convolution operation according to ID.\nThis design also encodes the watermark into the generated weights, enabling watermarking on non-POI without extra modules.\n\n**(II) Position**. The ID passport layer is placed at a later stage of the faceswap model, close to the output. In BlendFace, which contains 8 blocks, we inject it at the 7th block. This position has two advantages: (1) proximity to the output allows more direct manipulation in RGB space for watermark embedding; (2) features at this block have already captured rich ID-related information, which makes POI protection more effective. \nIn contrast, shallow layers are less effective, as the distance from the output increases the difficulty of watermark embedding in RGB space, and the features are semantically weak in terms of identity, which compromises the ability for POI protection.\n\n**(III) Work pipeline**.\nThe ID passport layer receives two inputs: feature F from the upstream network and the watermark, and outputs the final feature F'. As shown in **Fig. 3 in paper**, the layer is composed of multiple parallel ID-aware convolutions, denoted as IDConv1 to IDConvn. Details of IDConv are provided in A-Q2.\n\n- **A-Q2: The structure of each IDConv.**\n\nThe structure of each IDConv is a layer of 3*3 convolution. The construction of the IDConv is presented in **Fig. 8 in the Appendix**.\n\n**(I)** Watermark encoder. The watermark encoder feeds input watermarks into a series of FC and ReLU layers to obtain $f$ watermark features $[wm_1,wm_2,…,wm_f]$. \n\n**(II)** Kernel generator. We feed the upstream feature $F\\in \\mathbb{R}^{c\\times h\\times w}$ into several convolution and ReLU layers to obtain a new feature $F1\\in \\mathbb{R}^{c’\\times h’\\times w’}$. Then $F1\\in \\mathbb{R}^{c’\\times h’\\times w’}$ goes through $n$ different convolution layers and generates $n$ different kernels of size $(f\\times c’’\\times h’’\\times w’’)$. We utilize $[wm_1,wm_2,…,wm_f]$ to weight the $n$ kernerls for watermark embedding. Finally, we use $n$ different weighted kernels to initialize the $n$ IDConv. \n\n**(III)** Coefficient generator. It takes feature $F\\in \\mathbb{R}^{c\\times h\\times w}$ as input and passes it through the convolution and pooling layers. This is followed by the reshape operations to obtain n tensors of size $(c’\\times h\\times w)$, which are then transformed into n coefficients $[c_1, c_2,…,c_n]$ by the ReLU and fully connected layer. Finally, we use the $n$ coefficient $[c_1, c_2,…,c_n]$ to weight the n IDConv.\n\n\n- **A-Q3: Analysis of loss functions.**\n\nWe balance multiple losses by assigning equal weights to each term. This is intentional and based on the following reasons: \nThe watermark encoder, kernel generator, and coefficient generator have their learnable parameters, which enable our losses to achieve stable convergence without careful weighting. Besides, all components of the model are optimized jointly and simultaneously.\nThe overall loss function consists of face-swapping losses (FS), a knowledge distillation loss (KD), a POI-specific loss, and a WM loss. FS and KD are applied to nonPOI samples, while the POI and WM losses act on ID protection and watermark embedding, respectively, which are non-conflicting. Equal weights are assigned to all components based on three observations:\n\n**(I)** Numerical scales remain comparable across losses;\n\n**(II)** FS and KD address similar objectives, and all losses operate on disjoint data partitions (nonPOI vs. POI);\n\n**(III)** The WM loss modifies only imperceptible features, preserving image quality and avoiding interference with other losses.\n\nWe also acknowledge the importance of each loss component introduced to address a specific task:\n\n**(I)** FS loss and KD loss: enhances visual fidelity in face-swapped images.\n\n**(II)** POI loss: enforces protection of POI identities.\n\n**(III)** WM loss: embeds watermark into the output of nonPOI images.\n\nTo evaluate their contributions, we took BlendFace as an example and conducted ablation studies by removing each loss independently in the table below. The results show significant performance degradation in the corresponding tasks on the removal of any loss term.\n\n|Loss|PSNR|SRmask|WM_Acc|\n|-|-|-|-|\n|w/o FS loss|32.3|0.93|0.98 |\n|w/o KD loss|27.7|1.00|0.99|\n|w/o POI loss|34.5|0.00|0.99|\n|w/o WM loss|35.2|1.00|0.51|", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength\n1. The task of secure swap is interesting and practically meaningful.\n2. The paper is well-organized and -written, so that it is easy to follow.\n3. The proposed method is technically reasonable.\n4. The experiments are solid and present insightful analysis\n\nWeakness\n1. The ID passport layer is not clearly explained how to work.\n2. How does IDConv perform? what is the structure of IDConv.\n3. How to balance each item of loss functions to make sure the proposed network can work properly..", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received all accept recommendations. AC agrees with this recommendation and therefore is happy to accept the paper. Please address reviewer comments to the camera-ready version of the paper.
Decision: Accept (poster)