Advancing Generalized Deepfake Detector with Forgery Perception Guidance

Authors: ruiyang  xia
dawei  zhou
decheng  liu
lin  yuan
shuodi  wang
jie  li
nannan  wang
xinbo  gao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 03:08:12

Abstract: One of the serious impacts brought by artificial intelligence is the abuse of deepfake techniques. Despite the proliferation of deepfake detection methods aimed at safeguarding the authenticity of media across the Internet, they mainly consider the improvement of detector architecture or the synthesis of forgery samples. The forgery perceptions, including the feature responses and prediction scores for forgery samples, have not been well considered. As a result, the generalization across multiple deepfake techniques always comes with complicated detector structures and expensive training costs. In this paper, we shift the focus to real-time perception analysis in the training process and generalize deepfake detectors through an efficient method dubbed Forgery Perception Guidance (FPG). In particular, after investigating the deficiencies of forgery perceptions, FPG adopts a sample refinement strategy to pertinently train the detector, thereby elevating the generalization efficiently. Moreover, FPG introduces more sample information as explicit optimizations, which makes the detector further adapt the sample diversities. Experiments demonstrate that FPG improves the generality of deepfake detectors with small training costs, minor detector modifications, and the acquirement of real data only. In particular, our approach not only outperforms the state-of-the-art on both the cross-dataset and cross-manipulation evaluation but also surpasses the baseline that needs more than 3$\times$ training time. Code is available in the supplementary material.

Main paper: 
 introduction: 
 The development of artificial intelligence not only elevates performance in traditional visual tasks [5, 51] but also gives birth to massive novel and heuristic vision applications [21, 30, 48] . Deepfake, a novel technique used to generate believable media via deep neural networks [34] , has quickly developed and aroused social concerns due to the lifelikeness of the generation [11, 27, 66] and the simplicity of usage [7, 60, 63] . To ensure the safety and credibility of publicly-oriented media, in the field of computer vision and multimedia research, deepfake detection methods have recently been proposed to discern the authenticity of media automatically.Early investigations focus on several deepfake techniques [1, 62, 69] . Although these detectors achieve promising performance, their vulnerabilities are exposed immediately when facing the media manipulated by various advanced generative models. Maintaining the detection performance across a broad spectrum of deepfake techniques poses the primary challenge [22, 38] . To generalize the deepfake detector, as depicted in Fig. 1(a ) and (b), recent works mainly focus on the improvement on model structure and the synthesis of forgery samples. Model-centric methods modify the detector structure based on the public datasets for better perceiving the forgery samples [9, 43, 61, 72] , while sample-centric methods deliberately manipulate the common forgery traces such as boundary inconsistency to encourage the detector perceiving these forgery traces [6, 28, 42] . Although there are enhancements in the generalization, the trade-offs between costs and benefits have been overlooked. To be specific, the improved generalization entails integrating complex modules or constructing multiple parallel networks for the model-centric methods. In the case of sample-centric methods, enhancements encompass expensive training costs such as multiple training stages or lengthy training time. Therefore, the efficiency problem motivates us to consider a methodology that elevates the generalization of deepfake detectors with minor detector modifications and small training costs.To efficiently generalize the detector, one intuitive strategy is to investigate the deficiencies of the forgery perceptions, including the feature responses and prediction scores for forgery samples, and immediately feedback to the detector during training. Based on the insight, an efficient method called Forgery Perception Guidance (FPG) is proposed from the perspective of real-time perception analysis. As depicted in Fig. 1(c ), after the gradient computation through backpropagation, the sample is refined to emphasize the forgery traces that were previously unperceived. Subsequently, the detector is trained using the refined sample to strength its perception ability. More specifically, the gradients with respect to both the detector and inputs are respectively converted to the feature response and the adversarial perturbation for refining the shape and magnitude of the forgery traces. As a result, the refined forgery samples not only include forgery traces such as color mismatching but also reflect the deficiencies of the forgery perceptions. This training paradigm pertinently improves the perception ability related to the forgery samples and thus decreases the training costs. Furthermore, consider the input samples encompass irrespective factors, which are biased to the detectors and affect the perceiving of forgery traces. FPG additionally collects information on image qualities and forgery masks for extra explicit optimizations of the detector. Consequently, the generalization can be improved through FPG with less training costs, minor detector modifications, and the acquirement of real samples only. Experimental results show that FPG is not only superior to state-of-the-art on multiple evaluations but surpasses the baseline that needs more than 3× training time as well. These results demonstrate the high efficiency of our approach. The contributions are summarized as follows:• We shift the focus to the real-time analysis of the forgery perceptions and elevate the generalization of deepfake detectors through FPG with less training costs, minor detector modifications, and real sample acquirements only. • FPG refines the forgery samples by modifying the shape and magnitude of the forgery traces based on the feature responses and prediction scores. Moreover, explicit optimizations are further adopted during training by considering the image qualities and forgery masks, respectively. • Extensive experiments reveal that FPG significantly generalizes deepfake detectors under unknown datasets and deepfake techniques. Moreover, the training time can be saved around 3× compared with the comparable baseline method. 
related work 2.1 deepfake generation: 
 The face generation with malicious intention mainly includes identity replacement and expression reenactment. Identity replacement swaps the face of the target person in a video with the face of the source person. Early techniques focus on one-to-one identity replacement. For instance, FaceSwap uses autoencoders trained on faces with two persons for identity replacement [57] . The following techniques expand the paradigm to many-to-many identity replacement [7, 27, 58] . FaceShifter adopts Adaptive Attentional Denormalization layers (AAD) to transfer localized feature maps between the faces [27] . SimSwap increases the realism by injecting more identity information [7] . In addition, expression reenactment modifies the expression of the source person as arbitrary as the attacker wants. To be specific, the work in [3] applies an improved CycleGAN with two receptive field discriminators to execute one-to-one expression reenactment. Also, some works consider many-to-many expression reenactment [35, 52, 71] . ImaGINator not only considers the fusion between emotion and content but also adopts 3D convolutions to capture the distinct spatio-temporal relationships [52] .Model-centric Methods. Since the deepfake techniques mainly manipulate the local regions within the existing real images [34, 49] .To achieve automatic detection, deepfake detectors should carefully observe the image spatial regions for the sake of finding the nuanced artifacts generated through deepfake techniques. Besides, due to the lack of regular textures and semantic information in the forgery traces, conventional networks [10, 17, 44, 45] may not appropriate to deepfake detection. Therefore, some methods aim to improve the detection performance from the detector modifications. Specifically, a series of novel self-attention mechanisms [36, 64] , feature pyramid strategies [4, 14] , and graph network blocks [23, 53] are proposed in previous works to elevate the extraction ability of spatial forgery features for the networks. To further explore the samples, the other methods design the parallel networks which can simultaneously analyze the spatial and frequency information [8, 26, 39] . However, most datasets only provide coarse supervision such as binary labels, which makes it difficult for detectors to clearly learn the perceptions of forgery traces. Moreover, the perceptions learned from the previous datasets may be incompatible with the advanced deepfake techniques. Some methods can not achieve great detection performance under the newly generated forgery samples. Sample-centric methods. Although multiple deepfake techniques have appeared in recent years, their manipulation pipelines are similar. For example, identity replacement involves the generation of the source face and replacing the target face with blending. Some detection methods thus focus on sample synthesis to remain the common forgery traces like blending artifacts and avoid the traces generated from specific techniques. Besides, more information can be supplied, such as the forgery masks [42] , boundaries [28] , and bounding boxes [14] . Consequently, the generalization is improved with few [28] or even none [42] detector modifications. 
forgery perception guidance 3.1 preliminary: 
 Following the selection of a real sample, the attacker employs a deepfake technique to generate the designated content. This content is defined as the forgery trace, while the operational area of the deepfake technique can be succinctly represented with a forgery mask. Therefore, the essence of deepfake detection is to perceive these forgery traces, and the goal of generalizing a deepfake detector is to cultivate a strong perception ability for arbitrary forgery samples. This entails significant alignments between feature responses and forgery masks, along with notable consistencies between prediction scores and associated labels. To improve the perception ability of the detector, after the analysis of the forgery perception, the discrepancies of the consistencies and alignments should immediately feedback to the detector during training. An intuitive feedback path involves modifying the forgery samples based on these discrepancies, thereby encouraging the detector to learn the forgery traces that have been previously unperceived. However, this modification needs the same deepfake technique on the original forgery samples, which is complex and brings special forgery traces that are adverse to the generalization. To ease the generation of the original forgery samples and ensure these samples reflect the common forgery traces from the deepfake techniques, inspired by [42] , the forgery samples are generated as follows:EQUATIONwhere 𝒙 ′ ∈ R 3×𝐻 ×𝑊 indicates the forgery sample, which is generated by blending the real sample 𝒙 and its variant 𝑇 (𝑥) after geometric transformations. 𝐻 and 𝑊 denotes the height and width of the image. 𝑴 ∈ R 3×𝐻 ×𝑊 denotes the forgery mask. The forgery traces of the 𝒙 ′ within the 𝑴 > 0. According to Eq. ( 1), the learned perception of the detector from the forgery samples depends on the geometric transformations and the variations of the forgery mask. Since 𝑇 involves amount of discrete parameters, transformations, and non-differentiable during the detector training, we convert to explore the impacts from different forgery mask settings. From Fig. 2 , the optimal average result arises from the random mask setting, underscoring the diversities of forgery mask during training. This observation inspires the refinement of the forgery samples from the modifications of the forgery mask. Lastly, the cross-entropy loss is used to measures the difference between the prediction and the associate label of the input sample for guiding the training direction, which is expressed as follows:EQUATIONwhere 𝑦 𝑠 𝑖 and 𝑝 𝑠 𝑖 denote the binary label and the probability being fake for 𝑖-th sample 𝒔 𝑖 (𝒔 𝑖 may be real or fake), respectively.As shown in Fig. 3 , this procedure entails refining the shape and magnitude of the forgery traces by timely analyzing the forgery perceptions of the deepfake detector. The goal is to adopt pertinent training, thereby elevating the generalization with high efficiency. Forgery Shape Refinement. Since the feature response is closely related to the interested objects [41, 68] , the perceived forgery traces can be discerned through a comparative analysis of the distinctions between the responses and forgery masks. Given 𝑖-th fake sample 𝒙 ′ 𝑖 , the weight 𝛼 𝑐 𝑖 to the feature 𝑭 𝑥 ′ 𝑖 ∈ R 𝐶 ×𝐻 ′ ×𝑊 ′ in the last backbone layer of the detector at the 𝑐-th channel is computed as:EQUATION)where 𝑍 is the spatial size of the feature map. 𝐻 ′ , 𝑊 ′ , and 𝐶 denote the height, width, and channel of the feature map. The feature response 𝑹 ∈ R 𝐻 ′ ×𝑊 ′ of the 𝑖-th sample is computed as follow:EQUATION)As depicted in Fig. 3 , the shape refinement involves retaining the unperceived forgery traces while reverting the perceived regions to their real counterparts. After generating the feature response through Eq. ( 3) and ( 4), 𝑹 𝑖 is resized to the sample size and compared with the 𝑖-th forgery mask 𝑴 𝑖 . Given the potential differences in 𝑖 , The detector is frozen and the feature response 𝑹 𝑖 is computed to refine the shape of the forgery traces. After the shape refinement, the perturbation ε𝑖 of the mask M𝑖 is computed to further conduct the magnitude refinement. To the forgery shape refinement, the residual mask M𝑖 related to the unperceived forgery trace is acquired, and the perceived forgery traces are reverted to their real counterparts. To the forgery magnitude refinement, M𝑖 is added with the ε𝑖 to adjust the magnitude. The x′ 𝑖 is computed by blending the corresponding real image and its variant 𝑇 (𝒙 𝑖 ) after geometric transformations. The discrepancy between the shape-refined sample x′ 𝑖 and the x′ 𝑖 with both the shape and magnitude refinement is highlighted in the saliency map.orders of magnitude between feature responses and forgery masks, if the magnitude at each location for both the resized responses and masks exceeds 0, it will be binarized to 1. The binary mask with respect to the unperceived forgery trace is thus computed as:EQUATIONwhere for the 𝑖-th forgery sample, with the binarization, M𝑏 , 𝑴 𝑏 , and 𝑹 ′ 𝑏 ∈ {0, 1} 3×𝐻 ×𝑊 denote the residual mask of the unperceived trace, original forgery mask, and the resized feature response. However, using the M𝒃 directly leads to the noticeable boundary inconsistency in the refined forgery samples. To smooth the boundary, we adopt the GaussianBlur transformation to get the forgery mask M. Consequently, the sample x′ 𝒊 with forgery shape refinement is generated as follow:EQUATIONA straightforward method entails refining each forgery sample throughout the detector training process. Nevertheless, due to the diversities of forgery traces, it is important to notice the potential variability of the forgery perceptions. To the original hard forgery samples, the detector may still not predict them accurately in the early stage. Fewer forgery traces in these samples after refinement will further increase the training difficulties. To the original easy forgery samples, the resized feature responses are close to the forgery masks, the corresponding refined forgery samples will contain fewer forgery traces and are similar to the real counterparts. In the extreme case, x′ 𝑖 will approximate 𝒙 𝑖 if 𝑹 ′ 𝑏 𝑖 = 𝑴 𝑏 𝑖 , which is also adverse to the training. Therefore, to make the strategy adaptively refine the forgery samples, the refinement probability is dependent on the sample prediction score, which means the refinement to the hard original forgery samples can be lower than the easy ones. Besides, a forgery sample will be refined if the overlapped size [15, 24, 32] . Since these perturbations are closely related to the discrepancies between the prediction scores and the corresponding labels, adding these perturbations to the samples can enlarge the discrepancies and encourage the detector to further reduce the discrepancies during which the forgery perceptions pertinently. Following the insight, the work in [55] introduces adversarial perturbations as the parameters of GaussianBlur transformation and elevates the generalization of deepfake detection with a large margin. As we focus on refining the forgery mask, for the 𝑖-th shape-refined sample x′ 𝑖 , we further compute the gradient respect to the forgery mask𝜕 L 𝐶𝐸 ( x′ 𝑖 ) 𝜕 M𝑖. After that, M𝑖 is added with the perturbations computed from the gradients to refine the magnitude of the forgery mask. However, the inherent irregularities from gradient computations will lead to the refined forgery samples being unnatural and inconsistent with the real-world forgery samples. Considering both the adversarial property and the similarity with the real-world forgery samples, for the 𝑖-th forgery mask M𝑖 , we collect the gradients with sign function from each pixel and compute a unifying perturbation as follows:EQUATIONwhere 𝜖 is the magnitude of the perturbation. Since we focus on the regions within the forgery traces, the perturbations should not affect the real regions. The forgery mask added with the unifying perturbation at the (𝑢, 𝑣)-th location is expressed as follows:EQUATIONwhere Clip is the truncation function that constrains the magnitude of the M𝑖 within [0, 1]. Then the forgery sample x′ 𝑖 with both the shape and magnitude refinement is computed by following Eq. (1).Moreover, considering the training difficulties if the process includes the adversarial forgery samples only, the adversarial perturbation is added to the forgery mask with a 0.5 probability.Considering the sample diversities, certain attention modules are employed to make the detector adaptively focus on the forgery traces. However, these modules are updated based on the binary labels only, which are implicit and lack additional explicit supervision. This implicitness hampers the improvement in forgery perception. Therefore, two extra supervisions are introduced as explicit optimizations for the detector, i.e., image quality and forgery mask.Given a forgery sample, various image qualities can yield disparities within the forgery traces. It becomes essential to adjust the detector to accommodate such quality variations. In Fig. 4 , based on the advanced quality assessment network related to the facial images [37] , an extra regression network branch is built into the detector to output the regression results which are supervised by the output of the quality assessment network as follows:EQUATION)where 𝑄 is the pretrained quality assessment network. ∥•∥ 2 denotes Euclidean norm. Reg is the extra regression branch that consists of several convolutional layers. Subsequently, a preset lightweight attention module called Squeeze-and-Excitation [20] is supervised by the image quality information directly, which enables the detector to search for optimal detection patterns for the samples exhibiting diverse image qualities. The details of the regression branch and the attention module are described in the supplementary material. Furthermore, considering the differences between the 𝒙 𝑖 and the corresponding fake sample x′ 𝑖 located in M𝑖 > 0, contrasting the differences of the paired samples is beneficial to the detector perceiving the forgery traces more directly. The mask M𝑖 is hence introduced to extract the local features within the forgery region.Specifically, the M𝑖 is resized to the feature size and binarized to get M′ 𝑏 𝑖 ∈ {0, 1} EQUATION)Based on these extra explicit optimizations, the whole loss function during the detector training can hence be expressed as follow:EQUATIONwhere 𝐵 indicates the batch size, and 𝛼 denotes the scale factor. 
experiments 4.1 settings: 
 Dataset. To evaluate the generalization of deepfake detectors, our experiments are conducted on multiple large-scale and widely used datasets: FaceForensics++ (FF++) [40] , DeepfakeDetection (DFD) [18] , CelebDF (CDF) [29] , Deepfake Detection Challenge (DFDC) [19] , preview version of DFDC (DFDCP) [13] , and Face Forensics in the Wild (FFIW) [70] . FF++ is a large-scale dataset comprising 720 [47] and NeuralTexture (NT) [46] . We use the fake videos from the FF++ dataset for evaluating cross-manipulation performance. Following previous works [4, 39, 53] , the c23 version of FF++ is adopted for training. Moreover, since the CDF-v2 is more challenging than the v1 version, the results in CDF come from the v2 version in default. Implementation Details. Each pristine video is sampled 8 frames as the training sample. By default, we employ EfficientNetB4 [45] as the deepfake detector, which is initialized through pre-training on the ImageNet [12] . The batch and image size are the same as the baseline settings [42] . The threshold 𝑡 and 𝜖 in forgery sample refinement is set to 0.2 and 0.01, respectively. Empirically, the scale factor 𝛼 is set as 0.075. The maximum of training epochs is 50. The learning rate equals 1e-4. The optimizer is AdamW [31] with 0.9 and 0.98 betas, alongside 0.2 weight decay. The experiments are conducted on four RTX 3090 GPUs and the Pytorch framework.Evaluation. Following previous works in [2, 16, 42] , we use videolevel predictions to evaluate detectors. Each video are predicted through sampling 32 frames and averaging their predictions for comprehensive evaluation. We use the Average Precision (AP) and Area Under the Receiver Operating Characteristic Curve (AUC) as the metrics to evaluate the detection performance.Cross-Dataset Evaluation. The generalization constitutes the primary determinant for a deepfake detector. Since the public forgery datasets involve multiple deepfake techniques, complex real-world scenes, and various personal characteristics, the detection results across multiple datasets hold significant reference value for assessing the generalization performance. Therefore, we first evaluate the AUC results under five prevalent and challenging datasets that remain unknown to the deepfake detector. Table 1 presents our cross-dataset evaluation results. Compared with previous promising methods, the detector with FPG is trained on real samples only and outperforms the second-best method on CDF, DFDCP, and FFIW by 1.31%, 1.08%, and 3.10% points, respectively. To the DFD, we can also achieve a comparable result. As a result, leveraging the refinement strategy and the explicit optimizations, our approach attains 88.16 % AUC on average for five datasets and surpasses the baseline method [42] by 1.33% (88.16 % vs. 86.83%). These outcomes demonstrate the effectiveness of our approach. Cross-Manipulation Evaluation. Given the difference in deepfake techniques significantly affects the generalization, the next experiments are hence conducted to evaluate the detection results under different deepfake techniques. We follow the previous protocols [2, 25, 28, 42, 54, 65] by using the raw version of FF++ for evaluation. To be specific, our approach is evaluated through four deepfake techniques: DF, F2F, FS, and NT. Similar to the above evaluation, we compare the performance with both the video-level and frame-level methods. The evaluation results are listed in Table 2 . As seen, the results evaluated on DF and FS nearly equal the best performance (with only 0.02% and 0.25% AUC discrepancy).On the other hand, the detector attains the best results on F2F and NT and outperforms state-of-the-art methods on average. The promising detection results exhibited by the detector underscore the enhancements in generalization achieved by FPG.Training Efficiency Analysis. As stated, FPG is proposed to not only elevate the generalization of deepfake detectors but also the training efficiency. To analyze the efficiency in detail, the detector with FPG is trained with three different epochs, i.e., 10, 25, 50. As comparison, the baseline is also trained with the corresponding epochs and the same training settings for fairness. Fig. 5 (a) lists the average AUC results on CDF, DFDCP and FFIW. As seen, since the diversities of forgery traces are increased through FPG, too short training epochs make the detector under-fitting and insufficient for generalization. However, the advantage of utilizing FPG appears apparent when continues to increase the epoch. When the epoch number equals 50, all metrics from our approach surpass the baseline with significant margins. To further explore the number of epochs required for the baseline to approximate the performance achieved by FPG, we extend the maximum of training epochs to 100 and 250, respectively. Apparently, the improvement rate in detection results is sluggish. We conjecture the reason is the lack of a compact correlation between forgery sample generation and the cultivation of forgery perception. Consequently, using the original forgery samples to cultivate a comprehensive forgery perception proves to be inefficient. Even when the epoch number equals 250, the average detection results with the baseline method remain inferior to FPG trained with only 50 epochs.Table 3 presents more details related to the detection performance and the costs. As we bridge the forgery sample generation and the cultivation of forgery perception through the FPG. From the third row of Table 3 , after adopting the refinement strategy Method Applicability Analysis. To evaluate the effectiveness of FPG on the other deepfake detectors, i.e., Xception [10] , ResNet50 [17] , and VGG [44] . These detectors are trained with the baseline method and FPG, respectively. As shown in Table 4 , with the help of the refinement strategy and explicit optimizations, FPG outperforms the baseline method on multiple detectors. For example, FPG can elevate the detection results of Xception to 84.41%, which surpasses the baseline method by over 2.46%. Moreover, these results produced by FPG can effectively demonstrate its applicability, which is important in scenarios where the computing source, technique compatibility, or model scalability are constrained. Moreover, the experiments related to the applicability of different image distortions are listed in the supplementary material.Hyperparameter Setting Analysis. We first analyze the impact brought by the scale factor 𝛼. in the local forgery traces. Increasing the 𝛼 from 0.025 to 0.075 leads to an improved detection performance. This is because the direct contrast of the local features elevates the sensitivity of the detector to the forgery traces. The decline appears when 𝛼 beyond 0.075. This may be attributed to the respective field of the feature map. L 𝑠𝑖𝑚 uses the local features that are selected from the resized forgery mask. Since these deep layer features possess high receptive fields, indicating that the selected features related to the image area outside local forgery regions and increase the 𝛼 thus enlarging the biases. To get correlated features, a more precise feature selection strat egy will be considered in our future work.In the forgery sample refinement, 𝑡 is the overlapped threshold between the forgery masks and the corresponding resized feature responses. 𝜖 involves the modifications to the magnitude within the forgery trace. To investigate the impact of the hyperparameter, as depicted in Fig. 6 (a), the upward trend appears when 𝑡 is increased from 0 to 0.2, which means the forgery samples generated through the shape refinement can reflect the detector omissions to the forgery traces and then conduct more efficient training. Further increasing leads to fewer forgery traces within the samples, exacerbating the detection difficulties and hard to converge during training. Fig. 6 (b) illustrates the detection result related to 𝜖. Similarly, increasing the perturbation improves the detection result. Nevertheless, since the magnitude of the mask is adjusted through a one-step attack strategy [15] , perturbations are generated by linear approximation of the gradients. This approximation introduces biases and decreases the detection result when 𝜖 is further increased.Forgery Perception Analysis. To investigate the perception of detector to the forgery samples more intuitively, the saliency map of forgery samples with different deepfake techniques is shown in Fig. 7 . It can be apparently observed that, with the help of FPG, there are higher salient values than the baseline method within the forgery traces, which means the detector can perceive the forgery traces more completely. As a result, the deepfake detector can better adapt to the forgery samples with various deepfake techniques so that the generalization can be improved remarkably. More qualitative results are shown in the supplementary material. Sample Distribution Analysis. To make the detector adapt to the sample diversities, extra explicit optimizations are introduced to further consider the facial image qualities and local forgery traces during the detector training. Fig. 8 illustrates the impact brought by the optimizations to the sample distributions when facing unknown deepfake techniques. In the absence of explicit optimizations, the distributions of samples with the same class present dramatic variations such as sparse scattered or long trails, suggesting that the detector is sensitive to the irrespective factors. With explicit optimizations, the detector can better adapt the various samples and focus on the features related to the forgery traces. Due to the similar characteristics of samples with the same deepfake techniques, these distributions thus becomes clustered. 
conclusion: 
 In this paper, an efficient method dubbed Forgery Perception Guidance (FPG) is proposed to elevate the generalization of deepfake detectors from the perspective of real-time perception analysis, which has inadequate attention in previous studies. Specifically, FPG encompasses the forgery sample refinement and the extra explicit optimizations. The refinement strategy pertinently facilitates perceiving the forgery traces by modifying the samples based on the feature responses and prediction scores. The extra explicit optimizations promote adapting sample diversities by considering facial image qualities and local forgery masks, thereby further improving the forgery perceptions. Experiments demonstrate that FPG is superior to state-of-the-art methods on the cross-dataset and cross-manipulation protocols. Notably, the enhanced generalization is achieved with small training costs, minor detector modifications, and the acquirement of real samples only. 
 
 Appendix:  


Decision: All reviewers rated the paper to be fine, the authors are suggested to take all the comments into considerations for the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes an efficient method to finetune the shape and magnitude of the forgery mask. The author conducted relatively comprehensive experiments to show its efficacy and effectiveness.
strength_weakness: strength_weakness: 1. Overall this paper is well-motivated. I think the forgery response is a good starting point to design an algorithm.

2. The paper is well-written and easy for me to understand. There are no apparent logic gaps or truth errors.

3. The experiments conducted in this work are relatively sufficient.

4. This work shows a comprehensive discussion of related works, especially some generation techniques that are not well known in the detection field.
limitations: limitations: A significant limitation of this work is its heavy reliance on forgery masks. All the modules proposed in this work are heavily dependent on these masks, which is why this work uses SBI as their baseline. However, the improvement compared to the baseline (SBI) is minimal, with only about a 1% improvement on CDF-v2. Therefore, the primary contribution to the performance seems to be from SBI itself.

When considering the "efficacy module" proposed in this work, it relies on masks as their "golden annotations," making some supervised learning methods unfeasible and only allowing for SBI-like self-supervised methods. If we need to train on forgery data without any mask label, it would be challenging to apply your method in practice. In reality, we often do not know the forgery mask and its corresponding forgery region in a given deepfake video/image. Moreover, as mentioned in the related work section on generation techniques, most current state-of-the-art generation methods are many-to-many and do not involve a blending process but directly generate a whole image. So, in these cases, does your method still have potential applicability? I understand that many existing detection methods rely on blending data for training instead of deepfake data. However, the concern I've mentioned is crucial and increasingly significant for deepfake detectors. Therefore, I need more clarification on this point.

Another observation is that the refined shape of the mask appears visually strange. Intuitively, the forgery mask should resemble the original SBI's mask, with a convex hull or other shapes. However, after your refinement, the shape does not seem to resemble the actual deepfake forgery mask. So, how does your refinement achieve better performance?

My last minor point pertains to the t-SNE visualization. Based solely on the visualization, I cannot determine why your method is superior to the baseline. Could you provide more insights or explanations regarding this figure?
paper_summary: paper_summary: The paper deals with deepfake detection (facial images) by resorting to a forgery perception guided approach.
strength_weakness: strength_weakness: The paper addresses an important open issue such as generalization in deepfake detection.
It introduces the idea to force the attention on specific parts of the face related to the specific forgery by using a masking approach.
It only considers real sample for training the forgery perception part. 
Experimental results appear to achieve the sota.
limitations: limitations: The work is not so well presented and it is quite difficult for the reader to follow the whole adopted pipeline.
Also the way experiments are carried out is not often so clear.
Furthermore, it is not evident the motivation behind: which is the effective relation between a fake sample generated according to Equation (1) starting from a transformed real one and the generated fake samples obtained through the diverse DF techniques?
Often, it seems that the process also uses fake sample to train, please be careful when adopting the term "forgery sample" or similar.
It is not clear (Table 1), if all the method have been trained in the same way due to the fact that results are taken from their paper statements (see caption of Table 1), please explain this crucial issue? Furthermore, how is the DF detector (EfficientNetB4) trained on FF++? All the different forgeries together?
Moreover, in Table 2 (cross-forgery), same question as before? Table 2,  is it on raw FF++ images (see raw 670) or c23 as said in the caption?
paper_summary: paper_summary: In this paper, the authors propose an efficient method, i.e., Forgery Perception Guidance (FPG), to generalized deepfake detectors. Specifically, FPG adopts a sample refinement strategy to improve diversity of synthesized samples via the feedback of the optimization objective. Moreover, FPG introduces more sample information as explicit optimizations, which makes the detector further adapt the sample diversities. Extensive experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The proposed FPG could be exploited by existing method of synthesizing forgery artifacts, e.g., SBI, Face X-ray, and PCL, and further improve their performance.
2. FPG leverages differences of gradients in difference regions to further regenerate more targeted forgery samples, which guides the detector to mine more discriminative artifacts. Furthermore, the authors introduce the frozen quality assessment network with designed loss functions to guarantee the quality of synthesized samples.
3. Extensive experiments illustrate that SBI-based FPG method outperforms most existing methods in terms of cross-dataset and cross-manipulation generalization evaluations.
limitations: limitations: 1. RFM (Representative Forgery Mining for Fake Face Detection, CVPR 2021) also exploits the gradient feedback to guide the detector capture more comprehensive artifacts. It is encouraged that the authors compare this method and describe detailed differences. Moreover, in my opinion, FPG could be applied to any synthesis-based methods, e.g., PCL, Face X-ray, etc. I encourage the authors to include more basic methods and demonstrate the effectiveness of FPG, comprehensively.
2. As described in method part, forgery sample refinement exploits the gradient back-propagation of the cross-entropy loss function to regenerate diverse fake samples. However, in early training stage, the detector is unable to provide accurate forgery regions via gradient back-propagation, how the authors tackle this issue? Warm-up the detector with few epochs, or ? I expect the authors provide implementation details.
3. For a good detector, we also expect it to perform well under several real-world perturbations, e.g., Gaussian Noise, compression, color mismatch, etc. Therefore, I encourage the authors to conduct extensive experiments on robustness performance.
paper_summary: paper_summary: This paper presents a Forgery Perception Guidance (FPG) based method for face forgery detection. It builds on SBI[1], and proposed to refine the shape and magnitude of the manipulation region to improve the quality of generated images.
strength_weakness: strength_weakness: (1) The motivation is clear, and I think the proposed method is feasible.

(2) The results shown in Tables 1 and 2 are promising.
limitations: limitations: (1) Face X-ray and SBI are conducted on the FF++ raw data, so Table 1 is somewhat not right. As I know, the issue with these blending-based methods is that they do not perform well on compressed data. I have some doubts that the proposed method can achieve this result on the FF++ C23. Though the authors have released their code, I think it is better to provide the evaluation code and the trained model to verify its effectiveness.

(2) Another problem in blending-based methods is their poor robustness performance, can the author provide the robustness performance? 

[1] Detecting deepfakes with self blended images, CVPR 2022.