Proactive Deepfake Detection via Training-Free Landmark Perceptual Watermarks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 07:50:10

Abstract: The Deepfake face manipulation technique has garnered significant public attention due to its impacts on both enhancing human experiences and posing security and privacy threats. Despite numerous passive Deepfake detection algorithms that have been attempted to thwart malicious Deepfake attacks, they mostly struggle with the generalizability challenge when confronted with hyper-realistic synthetic facial images contemporarily. To tackle the problem, this paper proposes a proactive Deepfake detection approach by introducing a novel training-free landmark perceptual watermark, LampMark for short. Firstly, we analyze the structure-sensitive characteristics of Deepfake manipulations and devise a secure and confidential transformation pipeline from the structural representations, i.e. facial landmarks, to binary landmark perceptual watermarks. Subsequently, we present an end-to-end watermarking framework that robustly and imperceptibly embeds and extracts watermarks concerning the images to be protected. Relying on promising watermark recovery accuracies, Deepfake detection is accomplished by assessing the consistency between the content-matched landmark perceptual watermark and the robustly recovered watermark of the suspect Deepfake image. Experimental results demonstrate the superior performance of our approach in watermark recovery and Deepfake detection compared to state-of-the-art methods across in-dataset, cross-dataset, and cross-manipulation scenarios.

Main paper: 
 introduction: 
 Deepfake, a deep neural network based facial manipulation technique, has yielded substantial effects on society from both positive and negative perspectives [40] . While being adopted in industries such as filmmaking and education for benign utilization, Deepfake attacks have severely jeopardized the privacy and security of human beings. To satisfy the demand of preventing the current and potential risks accordingly, abundant attempts for Deepfake detection have been conducted in the research domain.Most existing Deepfake detection approaches fall in the category of passive detection. In essence, the algorithms are designed to distinguish between real and fake facial images after Deepfake has manipulated the original real images. While fake images can mostly be detected by tracing the synthetic artifacts within image feature domains in early stages, most methods have struggled with bottlenecks when facing hyper-realistic Deepfake contents since no obvious manipulation trace can be explicitly or implicitly located. This is also reflected in the unsatisfactory and fluctuating generalizability of the passive detectors on unseen manipulations and datasets.Recently, the concept of proactive defense has been raised such that invisible signals are inserted into benign images in advance of potential manipulations and falsifications can be addressed regarding their existence. This concept encompasses two topics, distorting and watermarking. While the former [14, 35] directly adds learned noises into images to disable Deepfake manipulations regardless of malicious and benign purposes, the latter protects the images in relatively subtle manners. On the one hand, semi-fragile watermarks [26, 50] are vulnerable to Deepfake manipulations, and the detection is executed based on the absence of watermarks. On the other hand, robust watermarks [38, 51] are used for attribution, detection, and source tracing purposes, owing to their distinct characteristics and semantics. Despite preliminary attempts in the domain, considerable research gaps persist, particularly concerning the robustness and generalizability of proactive watermarks.To mitigate these problems, we delve into the common behaviors of Deepfake manipulations in the widely known two categories, face swapping and face reenactment. Face swapping modifies facial identities and maintains other facial attributes unchanged. Conversely, face reenactment reconstructs facial expressions and head poses but preserves the facial identities. Although they each follow a unique protocol in the synthetic pipeline, the facial structures are generally modified regardless of identities, expressions, and head poses. In this paper, we exploit the facial landmarks to represent the structural information of facial images and construct landmark perceptual watermarks accordingly for proactive Deepfake detection. As Figure 1 depicts, facial landmarks of the images after benign image processing operations and Deepfake manipulations are extracted and compared with those of the original raw images. In the left sub-figure, obvious offsets can be observed regarding face swapping (SimSwap [6] ) and face reenactment (StarGAN [7] ) manipulations. In contrast, only relatively imperceptible offsets are caused by the benign manipulation (GaussianNoise). Scientifically, 10K images are randomly sampled from the CelebA-HQ [18] dataset to demonstrate a prevalent pattern in landmark offsets caused by image manipulations. In particular, four benign manipulations (Jpeg, GaussianNoise, GaussianBlur, MedianBlur), two face swapping manipulations (SimSwap and InfoSwap [9] ), and two face reenactment manipulations (StarGAN and StyleMask [3] ) are adopted to produce images on the 10K ones. As exhibited in the right sub-figure of Figure 1 , a pellucid gap is observed between landmark offset distributions in Euclidean distances caused by benign and Deepfake manipulations. In a nutshell, structures of facial landmarks are unavoidably modified upon Deepfake manipulations, while benign image operations generally maintain the original image content.In this study, to proactively protect images against malicious Deepfake manipulations, we analyze the structural consistency of facial landmarks before and after image manipulations and propose a landmark perceptual watermarking framework, namely, LampMark. First, we introduce a training-free watermark construction pipeline that projects the coordinates of facial landmarks to binary watermarks with fixed lengths while preserving the characteristics and distributions as displayed in Figure 1 . Then, we devise a cellular automaton encryption system to securely encrypt the watermarks with unpredictable and complex manners, guaranteeing strong watermark confidentiality. Thereafter, we train an end-to-end watermarking framework that robustly embeds and recovers watermarks against benign image processing operations and Deepfake manipulations. For a watermarked image that is suspected to be fake, Deepfake detection is achieved by analyzing the similarity between the recovered watermark and the landmark perceptual watermark with respect to the suspect image. Extensive experiments demonstrate outstanding watermark robustness with average bit-wise watermark recovery accuracies of 91.83% and 91.86% on CelebA-HQ at 128 and 256 resolutions, respectively. Furthermore, obtaining 98.39% and 98.55% AUC scores on detecting a mixed set of seven Deepfake manipulations demonstrates the state-of-the-art performance of our approach. The contributions of this work can be summarized as follows:• We exploit the structure-sensitive characteristic of facial landmarks regarding Deepfake manipulations and devise novel training-free landmark perceptual watermarks with confidentiality to defend against Deepfake proactively. • We propose a framework that robustly inserts and extracts the landmark perceptual watermarks into and from facial images. To the best of our knowledge, we are the first to simultaneously detect face swapping and face reenactment Deepfake manipulations with a single robust watermark.• Extensive experiments under in-dataset, cross-dataset, and cross-manipulation settings demonstrate the promising watermark recovery and Deepfake detection performance of our method, outperforming the state-of-the-art algorithms. 
related work 2.1 deepfake generation: 
 Ever since the first occurrence raised on Reddit1 in 2017, the term 'Deepfake' has attracted abundant public attention. Throughout the evolution of Deepfake technology, the generative algorithms have been broadly classified into two categories: face swapping and face reenactment. Face swapping [6, 9, 19, 20, 23, 27, 29, 39, 47] aims to swap the facial identity from a source image onto the target one while preserving the remaining image semantics. On the contrary, face reenactment [2, 3, 7, 11, 45, 49, 52] transfers the facial attributes, including expressions and poses, from a source image onto the target one but maintains the original facial identity. Both categories reconstruct the target image with the desired content modifications that mostly happen on the facial structures. Recently, unlike the early ones that exhibit obvious synthetic traces, Deepfake algorithms employing generative adversarial networks [10] (GANs) have shown promising and efficient performance in producing hyper-realistic synthetic content. These advanced synthetic models have achieved seamless generations without visual artifacts by incorporating modules that implicitly update the underlying features and smoothly enforce changes within the facial areas. While benefiting the industry in several aspects, this has brought severe challenges to the domain of multimedia forensics, especially to the passive Deepfake detection algorithms.While the passive Deepfake detection algorithms progressively advance the steps to address the underlying synthetic artifacts, experiencing the evolution of CNN based approaches [1, 55] , CNN backbone based approaches [21, 28, 31, 37] , integrated architectures [24, 36, 53] , dataset enrichment strategies [5, 32, 33, 54] , and deep analysis on implicit feature domains [4, 8, 22] , they mainly lack generalization ability when encountering newly occurred Deepfake models and datasets. Moreover, the high-quality synthetic outputs of recent generative algorithms further aggravate the challenges in distinguishing the fakes. Recently, instead of passively engaging in the expensive competition, several studies have been undertaken to proactively defend against Deepfake and preemptively protect the raw images in advance. In 2020, Ruiz, Bargal, and Sclarof [30] raised the idea of 'disrupting deepfakes' with a gradient-based method that adds an invisible perturbation 𝜂 to the image 𝑥 following x = 𝑥 +𝜂 such that the resulting image x is able to disrupt the generation of Deepfake. Thereafter, several follow-up studies step further. Huang et al. [16] designed a two-stage training framework that superposes perturbations onto images to nullify face attribute editing and reenactment manipulations. While later approaches [14, 35, 41, 57] progressively advance the disruption performance, they unfortunately disable the benign utilization of Deepfake, and the visual qualities of images are unavoidably affected due to directly added perturbations. To resolve the above issues, subsequent attempts are made via watermarking. Yu et al. [51] trained a framework that inserts model-specific fingerprints for the attribution purpose. FaceGuard [50] and Face-Signs [26] determine the falsification based on the existence of the embedded semi-fragile watermarks. ARWGAN [15] contains an attention-guided robust watermarking framework that resists GAN model attacks. Wu, Liao, and Ou [46] proposed SepMark, a separable watermarking framework that incorporates semi-fragile and robust watermarks, to perform detection and source tracing via analyzing each watermark. Wang et al. [38] fulfilled the entire pipeline of Deepfake face swapping detection by training a robust watermarking framework with watermarks containing identity semantics. 
methodology 3.1 problem formulation: 
 Contrary to the passive detection methods which analyze explicit and implicit artifacts to identify Deepfake images, proactive defense approaches focus on safeguarding the original images before potential synthetic manipulations happen. This is achieved by inserting visually imperceptible information within the image contents. In this study, we introduce a training-free pipeline, denoted as 𝐺 𝑚 , designed to transform facial landmarks of facial images into landmark perceptual watermarks of fixed lengths. Subsequently, we devise an auto-encoder architecture to robustly embed and recover the watermarks into and from the images, respectively. Finally, we address falsifications by comparing the recovered watermarks with the landmark perceptual watermarks of the manipulated images that are watermark-protected.In practice, a real image 𝐼 raw is embedded with the corresponding landmark perceptual watermark 𝑚 raw , derived following the pipeline 𝐺 𝑚 , to provide proactive protection, and the watermarked image 𝐼 rec is then reconstructed. When a Deepfake manipulation occurs on 𝐼 rec , generating the synthetic image 𝐼 fake , two watermarks are obtained accordingly. Firstly, the embedded watermark can be recovered as 𝑚 rec , faithfully similar to 𝑚 raw due to its robustness. Meanwhile, the landmark perceptual watermark 𝑚 fake can be obtained via 𝐺 𝑚 based on the image content of 𝐼 fake . The bit-wise matching rate between 𝑚 rec and 𝑚 fake is expected to be relatively low since they are produced based on different images with unique facial landmarks. Additionally, 𝐼 rec typically undergoes visual quality degradation (e.g., compression, noising, and blurring) upon uploading and spreading on the internet, becoming the processed image 𝐼 benign . Consequently, the matching rate between 𝑚 rec and 𝑚 benign is expected to be high, as their visual contents are generally similar. Ultimately, these matching rate values can assist in determining falsifications.Motivated by the structural variations in facial images caused by Deepfake manipulations and the structure-sensitive characteristic regarding facial landmarks as demonstrated in Figure 1 , we construct landmark perceptual watermarks tailored to this purpose. The watermarks are crafted to differentiate Deepfake-manipulated facial images by verifying the watermark consistency in a secure and confidential manner. Generally, we aim to fit an optimum mapping H ( EQUATIONwhere 𝜇 𝐸 𝑙𝑚 refers to the mean vector of 𝐸 𝑙𝑚 . Thereafter, the corresponding eigenvectors v and eigenvalues 𝜆 are obtained by solving the eigendecomposition equationEQUATIONThe eigenvectors are sorted with respect to the eigenvalues in descending order, representing the principal components of 𝐸 𝑙𝑚 . Thenceforth, we preserve the top 𝑙 eigenvectors to form the projection matrix W and the transformed vectors with length 𝑙 are derived followingEQUATIONSince the eigenvectors with the best 𝑙 values are maintained, features with the least importance are eliminated during dimension reduction, preserving the uttermost information of features and the original distribution. Finally, to construct watermarks with binary values, we scale the 𝑙 features of each vector in 𝐸 trans to the range of [0, 1]. In detail, we normalize 𝐸 trans with respect to the upper and lower bounds of all the vector features,EQUATIONfor feature values at all 𝑙 indices where 0 <= 𝑖 < 𝑙. Lastly, setting the threshold at 0.5 enforces all vectors in 𝐸 norm contain only binary values, denoted as 𝐸 bin .. By having access to the pipeline of generating the landmark perceptual watermarks, an attacker may intentionally replace the embedded watermark with a content-matched one so that Deepfake detection is disabled. To ensure the confidentiality of the watermarks, in this study, we leverage the concept of cellular automata [34] , which refers to a mathematical modeling paradigm for complex systems, and design a cellular automaton [42, 43] encryption system with a specific transform rule to provide random and chaotic behaviors for the watermarks. Particularly, regarding a binary encryption key with length 𝑙 in the cellular automaton, the state of each bit at the next time step is determined by the transform rule and the neighboring bits, such thatEQUATIONwhere 𝑠 𝑡 +1 𝑖 denotes the state at bit index 𝑖 for time step 𝑡 +1 following the transform rule 𝑅.In this study, Rule 30 [44] is applied and the key set 𝐾 with an initial key 𝑘 0 for watermark encryption is derived followingEQUATIONThe encryption key 𝑘 𝑡 at time step 𝑡 contains bit values 𝑠 𝑡 𝑖 at bit index 𝑖, and the key set 𝐾 = {𝑘 0 , 𝑘 1 , ..., 𝑘 𝑛 } is then obtained by executing Eqn. ( 6) for 𝑛 iterations. We randomly select 𝑝 keys𝐾 ′ = {𝑘 ′ 0 , 𝑘 ′ 1 , ..., 𝑘 ′ 𝑝 -1 }from 𝐾 where 0 < 𝑝 <= 𝑛 + 1 and sequentially perform logical exclusive OR (XOR) operations on the corresponding raw binary watermark 𝑚 0 ∈ 𝐸 bin of image 𝐼 followingEQUATIONand 𝑚 𝑝 ∈ 𝑀 after 𝑝 sequential XOR denotes the ultimately encrypted watermark.The encrypted watermarks are unpredictable and complex following the encryption pipeline and are used to proactively detect Deepfake materials. On the other hand, while guaranteeing confidentiality by preventing deciphering the original information from unauthorized attackers, an authorized user who owns explicit information regarding the transform pipeline and has access to the value and order of 𝐾 ′ may recover the original watermarks via XOR operations in an inverse sequence.To fulfill the goal of Deepfake detection, the embedded watermarks are designed to have robust manners when facing both benign and Deepfake manipulations. On the one hand, in real-life scenarios, the visual quality of images is unavoidably degraded upon posting and spreading through different protocols (e.g., on Twitter or Instagram). Consequently, the watermarks are expected to resist benign image processing operations such as compression, noising, and blurring. Therefore, we construct a benign manipulation pool 𝑃 benign that contains the benign image processing operations to adversarially enforce the watermark robustness. On the other hand, to achieve the main objective of this study, we construct a Deepfake manipulation pool 𝑃 deepfake that consists of various Deepfake face manipulation algorithms.In specific, 𝑃 benign = {GaussianNoise(0, 0.1), GaussianBlur(2, 3), MedianBlur(3), Jpeg(50)} and 𝑃 deepfake = {SimSwap, InfoSwap, UniFace, E4S, StarGAN, StyleMask, HyperReenact} are constructed as the two pools during evaluation to validate that the watermarks are robust against a diverse of adversaries, while the model sees only Jpeg (50) and SimSwap during the training phase.We devise an auto-encoder framework and train it end-to-end for watermark embedding and recovery. The overall framework is illustrated in Figure 2 . In a nutshell, we adopt mainly convolutional neural networks (CNNs) in the framework. Specifically, following a convolutional block (Conv. Block) to expand the feature dimension, repeated convolutional attention blocks (Conv. Attn. Block) are constructed to analyze the intra-correlation within the feature domain, locating the proper feature positions to hide the watermarks. The attention mechanism is conducted separately and sequentially in channel-wise and spatial-wise perspectives via convolutional operations that preserve channel and spatial dimensions, respectively. The channel-wise attention is accomplished via squeeze-andexcitation networks (SENet) [12] and the spatial-wise attention is denoted by max pooling and average pooling along the channels.On the other hand, to ensure robustness, we first diffuse the watermark 𝑚 to match the dimension of image features, and then study the intra-correlation within the watermark features via a sequence of convolutional attention blocks. As the image and watermark features are both analyzed regarding the optimal strategies for embedding the watermarks, we concatenate the features and reconstruct the watermarked image 𝐼 rec . Image manipulation algorithms are adopted from 𝑃 benign and 𝑃 deepfake and applied upon the watermarked image before passing to the decoder for watermark recovery. This setting provides representations after image manipulations so that the encoder is aware of the types of adversaries it is battling with when staying robust and the decoder can precisely recover the watermarks accordingly. As for the decoder, after a convolutional block that expands the feature channels, consecutive convolutional squeeze-and-excitation blocks (Conv. SE. Block) are applied. Unlike blocks in the encoder, the watermark features are simply refined by expanding the channels while squeezing the spatial maps without attention mechanisms. In the end, the features are flattened and linearly projected to the length of the watermarks.In the training phase, a discriminator is established to adversarially tune the encoder for better image visual qualities. In particular, the discrimination is of similar architecture as the decoder without expanding the channel dimension, performing as a binary classifier to determine the existence of watermarks.Since the process of fitting the binary watermarks from facial landmarks is training-free, we focus on designing objective functions for the auto-encoder when embedding and recovering watermarks. In general, there are four objectives to be concurrently considered.The watermarks are expected to be invisible, therefore, we apply an 𝐿 2 constraint to ensure reasonable visual qualities of the reconstructed watermarked images followingEQUATIONwhere 𝐼 and 𝐼 rec are the raw and watermarked images. At the same time, the decoder is designed to faithfully recover the embedded watermarks. Therefore, we assign an 𝐿 2 constraint on the decoder followingEQUATIONwhere 𝑚 and 𝑚 rec denote the original and recovered watermarks.While training the watermarking framework end-to-end, a discriminator 𝐷 that tries to distinguish raw and watermarked images is established to adversarially improve the visual qualities of the watermarked images byEQUATIONLastly, we reserve the benign utilization of Deepfake in the industry and employ the 𝐿 2 constraint to preserve the synthetic quality of Deepfake even for the target images that are watermarked,EQUATION)where 𝐺 represents the Deepfake synthetic model and 𝐼 𝑠 denotes the source image that provides synthetic information. 
experiments 4.1 implementation details: 
 In this study, we adopted the popular CelebA-HQ [18] and LFW [13] datasets, which both contain sufficient diversity in facial images. Specifically, we leveraged CelebA-HQ with 30,000 images and followed the official split for training, validation, and testing. Meanwhile, CelebA-HQ is employed as the data corpus to fit the transform rule W for watermark construction in pipeline 𝐺 𝑚 . LFW with 5,749 unique facial identities is used for the cross-dataset validation. While the transformation from landmarks to watermarks is training-free, we train the auto-encoder end-to-end with a learning rate of 2𝑒 -2 on 4 Tesla A100 GPUs. [17] 128 × 128 30 33.01 0.775 RDA [51] 128 × 128 100 43.93 0.975 CIN [25] 128 × 128 30 43.37 0.967 ARWGAN [15] In this section, we validated the performance of the proposed method on CelebA-HQ [18] by evaluating the visual quality, bit-wise watermark recovery accuracy, and Deepfake detection AUC score. Contrastive methods include robust watermarking frameworks (HiDDeN [56] , MBRS [17] , and CIN [25] ), semi-fragile proactive Deepfake watermarking frameworks (FaceSigns [26] ), and robust proactive Deepfake watermarking frameworks (SepMark [46] and ARWGAN [15] ). Algorithms with source code available are reproduced in all experiments.Quality. The sampled images are exhibited in Figure 3 . Specifically, we visualized the original images, watermarked images, and manipulated watermarked images from bottom to top. The first four columns refer to the benign manipulations assigned with the listed parameters, and the remaining columns exhibit the effects of Deepfake manipulations, omitting the source images that provide desired identities, expressions, and head poses. It can be observed that the watermarks merely affect the visual qualities of images, and the Deepfake manipulations are regularly executed even on images with watermarks embedded.In quantitative experiments, we computed the average peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) regarding the raw and watermarked images. The two metrics evaluate the level of noise and structural similarity of the watermarking framework, respectively. As Table 1 lists, our method retains outstanding visual quality in both 128 and 256 resolutions. In specific, we achieved the best performance for the two resolutions, outperforming the previous state-of-the-art algorithms. The promisingly high PSNR and SSIM values imply the imperceptible visual perturbations brought by the watermarks towards raw images. Additionally, while the early approaches generally demonstrate mere advantages, RDA [51] and CIN [25] have shown likewise reasonable visual qualities with competitive statistics at the 128 resolution, and MBRS [17] is observed to be more reliable at the 256 resolution with the second best performance.We compared the similarity between the original watermark 𝑚 and the recovered watermark Table 2 : Quantitative comparison on CelebA-HQ regarding the bit-wise watermark recovery accuracy of the watermarks under benign manipulations. GausNoise, GausBlur, and Med-Blur are abbreviations of Gaussian Noise, Gaussian Blur, and Median Blur for space saving.GausNoise GausBlur MedBlur Jpeg HiDDeN [56] 51.36% 73.04% 82.72% 67.84% MBRS [17] 99.60% 99.99% 99.99% 99.49% RDA [51] 60.18% 99.95% 99.98% 66.85% CIN [25] 86.00% 99.99% 97.03% 96.86% ARWGAN [15] 53.45% 85.22% 96.66% 57.42% SepMark [46] 99 EQUATIONwhere 𝑚 𝑖 rec and 𝑚 𝑖 refers to the bit values at index 𝑖 of each watermark. In this section, to verify the model robustness, watermarks are recovered after each manipulation from 𝑃 benign and 𝑃 deepfake is executed.As illustrated in Table 2 and Table 3 , our proposed algorithm consistently outperforms the contrastive state-of-the-art ones. In Table 2 , the benign manipulations, GaussianNoise, GaussianBlur, Median-Blur, and Jpeg, are adopted for evaluation. It can be observed that, although most watermarking frameworks suffer accuracy damping due to the distortions and noises brought by the manipulations, SepMark [46] and our method generally maintain robustness in all circumstances for both resolution levels. In summary, Gaus-sianNoise and Jpeg are observed to be the most challenging ones for all models, including ours. It is also worth noting that, despite achieving satisfactorily high statistics at the 128 resolution, results of MBRS [17] are unreliable because of the unexpectedly low visual quality in Table 1 . Contrarily, while MBRS maintains reasonable visual quality at the 256 resolution, the corresponding watermark recovery accuracies in Table 2 are poor. As a result, combining the performance in Table 1 and Table 2 , our approach favorably achieves state-of-the-art average watermark recovery accuracies of 99.89% and 99.99% at the 128 and 256 resolutions, respectively, against benign image manipulations while concurrently ensuring satisfactory image visual qualities.Regarding the malicious Deepfake manipulations, watermarks are expected to stay robust so that proactive detection can be accomplished consistently. The watermark recovery accuracies at the 128 and 256 resolutions are reported in Table 3 . In general, except for SepMark [46] and our approach, all remaining comparative models have derived accuracies mostly around 50%. This implies Table 3 : Quantitative comparison on CelebA-HQ regarding the bit-wise watermark recovery accuracy of the watermarks under Deepfake manipulations. The top half refers to the 128 resolution and the bottom half refers to 256.SimSwap [6] InfoSwap [9] UniFace [47] E4S [23] StarGAN [7] StyleMask [3] HyperReenact [2] Average HiDDeN [56] 50.02% that their watermarks are ruined by Deepfake manipulations such that the decoders are unable to recover the correct messages. As a result, although most robust watermarking frameworks are able to achieve certain levels of robustness regarding some of the benign image manipulations, they are unfavorably unsatisfied against Deepfake manipulations. In addition, FaceSigns [26] , although designed as a semi-fragile watermarking framework that is supposed to be vulnerable and have low accuracies when facing Deepfake manipulations, fails to stay robust against benign image manipulations. Meanwhile, although trained against SimSwap, SepMark still suffers considerable watermark recovery errors with 86.17% and 92.09% accuracies for the two resolutions. Furthermore, strong fluctuations can be observed under the cross-manipulation scenario such that unseen Deepfake manipulations have caused significant challenges to SepMark by destroying the underlying watermarks.In conclusion, our proposed algorithm maintains the best robustness for benign and Deepfake manipulations and achieves 87.23% and 87.21% average accuracies in Table 3 for the 128 and 256 resolutions, respectively. At the same time, despite being trained against SimSwap [6] solely regarding Deepfake manipulations, our model demonstrates state-of-the-art cross-manipulation performance when tested with other Deepfake manipulations. Additionally, while statistics for face swapping models are all above 90%, those for face reenactment models are only around 75%. This is possibly caused by wiping out the background information in face reenactment results, making it more difficult to recover the original watermarks.In this study, the ultimate goal of maintaining watermark robustness is to ensure the reliability of watermarks and enforce proactive Deepfake detection accordingly. Particularly, for a watermarked image 𝐼 rec with embedded landmark perceptual watermark 𝑚 raw derived from the raw image 𝐼 raw following the pipeline 𝐺 𝑚 , falsification is addressed based on the recovered watermark 𝑚 rec and landmark perceptual watermark 𝑚 sus of a suspect image 𝐼 sus . If 𝐼 sus is derived via benign manipulations, the structural content is not modified and the similarity between 𝐼 sus and 𝑚 rec is, therefore, expectedly high. On the other hand, if 𝐼 sus is a Deepfake image, due to structural changes in the image content, 𝐼 sus and 𝑚 rec are dissimilar. Based on this rule, we conducted Deepfake detection on the seven Deepfake manipulation algorithms and computed the AUC scores by concurrently introducing benign image manipulations upon the raw images to produce real samples.In Table 4 , we compared the detection performance with four popular and state-of-the-art passive Deepfake detectors2 , namely, Xception [31] , SBIs [32] , RECCE [4] , and CADDM [8] . While the four passive detectors have demonstrated superior detection ability in lab-controlled scenarios on the FaceForensics++ [31] dataset, as listed in the last row of Table 4 , they are generally fooled by the upto-date synthetic algorithms with as low as less than 40% for AUC scores. Specifically, besides the lower resolution level bringing more difficulty with indistinguishable underlying artifacts and noises, the hyper-realistic output images by E4S, StarGAN, and StyleMask have limited the AUC scores of detectors all below 80%. On the other hand, SBIs and CADDM are observed to be the most powerful passive detectors by closely approaching 90% for AUC scores on some Deepfake manipulations.As for our proposed proactive approach, relying on the outstanding watermark recovery accuracy for all benign and Deepfake manipulations, the extracted watermarks by the decoder are faithfully similar to the originally embedded ones. Therefore, comparing 𝐼 sus and 𝑚 rec leads to reliable Deepfake detection results. As reported in Table 4 , our approach demonstrates superior detection ability with AUC scores all above 95% regardless of manipulation algorithms. In other words, the similarities between 𝐼 sus and 𝑚 rec for the benign manipulations are expectedly higher than those for the Deepfake manipulations. To conclude, our approach is proved to have the best performance of 98.39% and 98.55% for AUC scores even with a mixture of all seven Deepfake manipulations.Under the cross-dataset setting, we further evaluated the generalizability of our proposed LampMark on unseen datasets. In particular, we adopted the LFW dataset and conducted experiments at the 128 Experimental results are reported in Table 5 . It can be seen that the watermarking frameworks generally perform similarly to those on CelebA-HQ. Specifically, besides the visual qualities of RDA [51] , CIN [25] , and ours being generally acceptable, GaussianNoise and Jpeg are still the most challenging benign image manipulations that have caused the most trouble in watermark recovery. Meanwhile, the watermarks behave in like manners as in Table 2 when evaluated against Deepfake manipulations such that most of them become fragile when being manipulated by Deepfake. As a result, although slightly suffers the cross-dataset challenge when tested against the face reenactment models, the proposed method consistently reaches state-of-the-art performance with an 89.73% recovery accuracy on average for all manipulations. Furthermore, SepMark, besides consistently demonstrating fragile characteristics when facing unseen manipulations, encounters similar performance damping as ours under the cross-dataset setting. 
conclusion: 
 In this work, we exploit the structure-sensitive characteristic of Deepfake manipulations and present a proactive Deepfake detection approach, LampMark, that relies on landmark perceptual watermarks. We propose a training-free pipeline to transform facial landmarks into binary watermarks, securely protected by a sophisticated cellular automaton encryption system. Then, an end-to-end auto-encoder architecture is trained to embed and extract watermarks robustly. Extensive experiments demonstrate the state-ofthe-art performance of our method in watermark recovery and Deepfake detection. Lastly, despite successfully stepping forward with the innovative idea, the existing performance gap from perfect statistics points out the future direction of our research to continuously advance the model generalizability, especially for the cross-manipulation watermark recovery ability against face reenactment manipulations.  
 
 Appendix: https://www.reddit.com/The proactive approaches are excluded since the unsatisfactory watermark recovery accuracies make the corresponding detection unconvincing. In addition, since their watermarks are semantically void, the pipeline for Deepfake detection is incomplete. 


Decision: The reviewers generally are satisfied with the rebuttal, and recognize the contribution of proposed method along with the convincing evaluations.
Basically, the contributions of this manuscript include (1) a new solution to deepfake detection based on perceptual watermarking; (2) LampMark is training-free; (3) experimental evaluation is coprehensive.
Based on the above concern, it is recommended to accept this manuscript.

Reviews: review: paper_summary: paper_summary: The paper titled "Proactive Deepfake Detection via Training-Free Landmark Perceptual Watermarks" introduces a novel approach to Deepfake detection. The authors propose a proactive method named LampMark that leverages facial landmarks to create perceptual watermarks. These watermarks are binary representations generated from facial structures, which are then embedded into images to protect them against potential manipulations. The system is designed to be robust against both benign image processing operations and Deepfake manipulations. It employs an auto-encoder architecture for watermark embedding and recovery, ensuring the watermarks remain intact even after various manipulations. The paper claims state-of-the-art watermark recovery and Deepfake detection performance across different scenarios.
strength_weakness: strength_weakness: 1.	The paper presents a unique solution to Deepfake detection by using perceptual watermarks based on facial landmarks, which differs from traditional passive detection methods.
2.	LampMark's watermark generation process does not require training, which can save significant time and resources compared to machine learning-based methods.
3.	The paper reports high-performance metrics, with excellent watermark recovery accuracies and AUC scores for Deepfake detection, outperforming existing state-of-the-art methods.
limitations: limitations: 1.	LampMark utilizes auto-encoders, yet the experimental section lacks clarity on the auto-encoders' configuration. This ambiguity could hinder future researchers' ability to replicate the system and assess the robustness of its architecture.
2.	Lack of ablation experiments: LampMark framework consists of encoder and decoder but does not test the performance of the different auto-encoder configurations.
3.	The descriptive details of the article need to be checked. Such as line 458, ‘we adopt mainly…’.
paper_summary: paper_summary: In this paper, the authors proposed a proactive deepfake detection method by using a trainning-free landmark perceptual watermark. The end-to-end watermarking framework is proposed to improve the robustness of the extraction. The main contribution lies in the transformation from the facial landmarks to the perceptual watermarks. Overall, the manuscript is well organized and the experimental comparisons are sufficient.
strength_weakness: strength_weakness: The algorithm is designed in a sophisticated manner. Each step and modul are given with reasonable motivations.
limitations: limitations: The improvement over the SOTA method is quite limited. Is it valuable to put forward the investigation on this specific problem, as the current methods have shown a very good result?
paper_summary: paper_summary: This work addresses the problem of deepfake detection. The work introduces a novel training-free landmark perceptual watermark, which is used to detect two types of deepfakes: face swapping and face re-enactment. The experimental results show that both watermark extraction and deepfake detection are achieved with very good performance results.
strength_weakness: strength_weakness: - The writing is clear, easy to understand and the narrative is logical.
- This is an issue that has been on the minds of the community since depth modelling became active.
- The methodology and evaluation process is very detailed and solid.
limitations: limitations: I have to be honest and say that I am not very familiar with the subfield of depth forgery, and I have carefully studied the related work and the methods and experimental procedures presented in this entry. It may be difficult for me to make particularly nuanced and focused comments. One suggestion I have is to perhaps include more discussion in the experimental results evaluation section, e.g. why does your method achieve such good experimental results? Which steps played a key role?
paper_summary: paper_summary: The security and privacy issues have always been the key issue for the application of deepfake technique, especially in face manipulation. This paper proposes a proactive Deepfake detection approach based on watermarking scheme, this can not only perform forgery detection when the manipulation is maliciously, but also can preserve the quality of watermarked image and avoid hindering the benign application of deepfake technology. The main contribution is to introduce a landmark perceptual watermark transformation pipeline based on the structure-sensitive characteristics of face manipulation, which can satisfy the confidentiality, discrimination and robustness simultaneously. Evaluation experiments comprehensively cover the in-dataset, cross-dataset, and cross-manipulation settings, demonstrating the superior detection performance of the proposed method.
strength_weakness: strength_weakness: - This paper contributes a proactive deepfake detection framework based on watermarking scheme, which innovatively exploit the structure-sensitive characteristics of face manipulations, and combine the concept of cellular automata to ensure confidentiality.
- Datasets and baseline methods are comprehensive, and the results can fully support the proposed framework LampMark.
limitations: limitations: - The biggest limitation of LampMark framework is requiring the original face image in the inference detection, that is, it is a non-blind watermarking scheme, which I think is limited in real-world scenario.
- In the training stage, the decoder receives both benign and fake images, and if the forged image is also embedded in the landmark perceptual watermark, whether it will interfere with the recovery of the original image watermark.
- I wonder to know how to constrain and optimize $m_{rec}$, as described in Section 3.1, the objective functions seems lack of the constraint on $<m_{benigh},m_{rec}>$  and $<m_{fake},m_{rec}>$. In addition, if training end-to-end, there should be a total objective function and the  a description of weights for each part.
- I am curious about why the model sees only Jpeg(50) and SimSwap during the training phase, but it shows good robustness to more types of distortion at testing phase, please give a detailed explanation.
- The presentation of this paper needs to improve, and there are some notations are not uniform. Such as, the $m$ in Eq(9) and  $m_{raw}$ in Section 3.1 seems both refer to the landmark perceptual watermark of the original image, and $I$ in Eq(8) seems to be $I_{raw}$, please take care to make a uniform representation. Additionally, there is a lack of notation for the  input image of $P_{deepfake}$ in Figure 2 and there are still some reference citations problems, the authors need carefully check the manuscript.
paper_summary: paper_summary: This paper introduces a novel proactive Deepfake detection approach via training-free landmark perceptual watermarks. By analyzing the structure-sensitive characteristics and assigning landmark perceptual semantics to the robust watermarks, the proposed LampMark accomplishes the proactive detection pipeline. The authors also devise an encryption pipeline via a cellular automaton system to ensure watermark confidentiality. The paper reports promising watermark recovery and Deepfake detection results compared to the SOTA methods.
strength_weakness: strength_weakness: 1. The idea of the landmark perceptual watermark is novel, and it assigns semantics to the watermarks in a training-free manner, thus enhancing the explainability of Deepfake detection results at a low cost. 
2. This work innovatively accomplishes the entire pipeline of proactive Deepfake detection without knowing the groundtruth in advance, which is barely achieved in previous studies. 
3. This paper provides good statistics with sufficient experiments for in-dataset, cross-dataset, and cross-manipulation scenarios, demonstrating reliable model performance against various SOTA algorithms.
limitations: limitations: 1. There is an inconsistency of numerical statistics on page 2 in section 1 when reporting the average watermark recovery accuracies, 91.83% and 91.86%, compared to Table 3. Please double-check and clarify this in rebuttal. 
2. There are other commonly seen image processing operations such as brightness, contrast, saturation, and hue. Since they have been discussed in recent papers, how robust is LampMark when facing these operations?
3. Typo issue to be fixed: A missing percentage notation in Table 3 at performance for UniFace.