SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1) Horizontal Flip: This augmentation randomly flips the image horizontally with a probability of 0.5, simulating mirror images.
2) Rotation: This augmentation randomly rotates the image within a range of -10 to 10 degrees with a probability of 0.5. By applying random rotations, it introduces diversity in object orientations, making the model more robust to different angles and orientations.
3) Isotropic Resize: This augmentation resizes the image while maintaining isotropy, ensuring that the aspect ratio of the image is preserved. It randomly selects one interpolation method (INTER_AREA, INTER_CUBIC, or INTER_LINEAR) for resizing. The maximum side length is determined by the configured value. Isotropic resizing is particularly useful when dealing with objects that have varying scales and proportions, allowing the model to learn from different object sizes and maintain the aspect ratio of the objects.
4) Random Brightness and Contrast: This augmentation randomly adjusts the brightness and contrast of the image with a probability of 0.5. By applying random brightness and contrast variations, it introduces changes in the illumination and contrast levels of the images. This helps the model generalize better to different lighting conditions and improves its robustness to variations in brightness and contrast.
5) FancyPCA: This augmentation applies the FancyPCA algorithm with a probability of 0.5. FancyPCA performs Principal Component Analysis (PCA) on the pixel values of the image and perturbs the components to introduce color variations. By altering the principal components of the image, it can change the color distribution, leading to more diverse training samples.
6) Hue Saturation Value (HSV) Adjustment: This augmentation randomly adjusts the hue, saturation, and value of the image. While the probability is not specified in the code snippet, it allows for variations in the color representation of the images. Adjusting the hue changes the overall color tone, saturation controls the intensity of colors, and value adjusts the brightness.
7) Image Compression: This augmentation applies image compression with a probability of 0.5. It reduces the quality of the image by compressing it. The lower and upper limits, set to 40 and 100 respectively, control the compression quality. Image compression introduces artifacts and reduces the image quality, simulating real-world scenarios where images may be of lower quality or have compression artifacts. This augmentation helps the model learn to handle such variations and improves its robustness in practical applications.
Full Experimental Results
Overview In the main paper, our focus is on presenting the experimental results obtained from selecting the models that achieve the highest performance on each individual testing dataset. The primary metric utilized for evaluation in the main paper is the Area Under the Curve (AUC). In order to provide a more comprehensive view of our experimental results, we present the complete set of results here. We have incorporated three different widely utilized metrics for assessment: AUC, Average Precision (AP), and Equal Error Rate (EER). These metrics are dynamically recorded throughout the training process as part of our benchmark. Additionally, we have stored the prediction results along with their corresponding labels, which facilitates the computation of additional metrics. In this paper, we compare the 3 aforementioned metrics as a means to compare the performance of the 15 detectors across the 14 testing datasets.
Comprehensive Metrics In addition to saving the best-performing model throughout the training process, we also save the last model to evaluate its performance at the completion of all training epochs. This allows us to assess the models' effectiveness after undergoing the entire training duration. Furthermore, by recording the predictions and corresponding labels, we are able to calculate additional metrics such as Precision and Recall, in addition to the previously mentioned metrics.
Figure 11: Illustration and visualization of within-dataset evaluation. We draw the ROC-AUC curve using the models at the last trained epoch.
Figure 12: Illustration and visualization of cross-dataset evaluation. We draw the ROC-AUC curve using the models at the last trained epoch.
Here, we present the ROC-AUC curve and Precision-Recall curve for all detectors. These detectors are trained on the  $\mathrm{FF}++$  (c23) dataset and evaluated on a total of 14 testing datasets, encompassing both within-dataset and cross-dataset evaluations (see Fig. 11, Fig. 13, Fig. 12, Fig. 14). These visualizations provide a more comprehensive understanding of the experimental outcomes, allowing for a more detailed analysis of the detectors' performance. Moreover, as a benchmark, our proposed approach facilitates the computation of additional evaluation metrics based on user requirements, thereby demonstrating the convenience and versatility of our benchmarking framework.
Full Testing Results During the Training Process To facilitate the monitoring of model performance during the training process, we utilize TensorBoard to record various metrics. These metrics include training loss, training accuracy, AUC, AP, and EER, as well as testing loss and testing metrics (AUC, AP, EER). By visualizing these metrics, users gain insight into the performance trends during training, enabling them to debug issues and optimize parameters as needed.
In this section, we present visualizations of testing metrics plotted against the training steps. The metrics of interest include AUC, AP, and EER, which provide a comprehensive assessment of the detectors' performance across different datasets (see Fig. 16, Fig. 19, Fig. 15, Fig. 18, Fig. 17, Fig. 20). By comparing the curves, we can analyze the relative performance of the detectors using different evaluation metrics.
Furthermore, we can observe the stability of the testing results. Some detectors may exhibit volatility and lack stability in their metrics, which introduces uncertainty. In such cases, while the overall
Figure 13: Illustration and visualization of within-dataset evaluation. We draw the Precision-Recall curve using the models at the last trained epoch.
Figure 14: Illustration and visualization of cross-dataset evaluation. We draw the Precision-Recall curve using the models at the last trained epoch.
results may not be consistently good, there may be instances where individual metrics perform well. To address this, we adopt an average-based approach, computing the average values for each testing metric to determine the final results (Top-3). By examining the provided figures, we can also discern the stability of each detector's performance.
Note that due to the differing training batch sizes of DSP-FWA and Face X-ray detectors compared to the other 13 detectors, we visualize them separately. This distinction allows for a more clear comparison within their respective groups.
A.4 Other Analysis Results
Artifacts of deepfake forgeries in Frequency Inspired by [48], we adopt a similar approach to visualize the average frequency spectra of each dataset. The purpose is to examine the artifacts generated by deepfake forgeries. Our methodology involves computing the average frequency spectrum of a selected set of images, specifically 2000 randomly sampled images. To mitigate computational complexity, a random subset of both real and fake images is chosen for analysis. The process begins by converting the images to grayscale and applying a high-pass filter. Subsequently, a Fourier transform is performed, with the zero frequency component shifted to the center of the spectrum. Finally, the spectra are summed and averaged to obtain the final result.
The resulting visualization comprises three subplots for each deepfake forgery. The first subplot illustrates the average spectrum of the real image, the second subplot represents the average spectrum
Figure 15: Illustration and visualization of all testing results during the training process. The metric is AUC. We compare 13 detectors (except for the Face X-ray and DSP-FWA) on different datasets using the AUC metric.
of the fake image, and the third subplot showcases the difference between the spectra of the real and fake images.
Our findings align with those reported in [48]. We observe that deepfake forgeries do not exhibit obvious artifacts, as observed in other images generated by GANs. This consistency with the findings in [48] can be attributed to the various pre-processing and post-processing steps involved in the creation of deepfake images. These steps, which include resizing, blending, and MPEG compression of the synthesized face region, introduce perturbations in the low-level image statistics. As a result, the frequency patterns may not emerge distinctly in our visualization method.
Visualizations of GAN-generated and diffusion-generated artifacts in Frequency Following the similar process in Sec. A.4, we also visualize the artifacts generated by GANs and diffusion models. Specifically, we utilize the GenImage dataset [54] and apply the frequency analysis tool in our benchmark for analysis. The visualizations are shown in Fig. 22. This analysis has unearthed intriguing observations specific to diffusion-generated images when contrasted with GAN-generated images. Particularly, diffusion-generated images exhibit fewer artifacts, while GAN-generated images display a noticeable checkerboard pattern of artifacts.
Figure 16: Illustration and visualization of all testing results during the training process. The metric is AP. We compare 13 detectors (except for the Face X-ray and DSP-FWA) on different datasets using the AP metric.
Cross-data evaluation and the importance of phase spectrum In Tab. 3 of the manuscript, we highlight the SPSL detector, which achieves an impressive average score of  $78.75\%$  in cross-domain evaluation. A distinctive feature of SPSL compared to Xception is the incorporation of the phase spectrum, which is concatenated with the spatial image in the channel dimension. As mentioned in the original SPSL paper, the phase spectrum can capture up-sampling artifacts present in many forgery processes. Motivated by this finding, we explore the potential benefits of incorporating the phase spectrum feature into blending-based detectors. We hypothesize this would enhance performance in both cross-data and cross-manipulation evaluations.
- Cross-data evaluation: To validate our hypothesis, we first conduct an experiment in which we integrate the spectrum feature into the FWA detector, resulting in an improved FWA (iFWA). The experimental results, summarized in Tab 6, show a significant improvement achieved by iFWA (from  $73.16\%$  to  $80.35\%$  in average AUC).
- Cross-manipulation evaluation: Second, we conduct experiments and show the cross-manipulation outcomes achieved by iFWA in Tab. 7. These visualizations serve to strengthen our argument about the consistent performance of iFWA.
Figure 17: Illustration and visualization of all testing results during the training process. The metric is EER. We compare 13 detectors (except for the Face X-ray and DSP-FWA) on different datasets using the EER metric.
These two analyses and experimental validations aim to explain the phenomena observed in our evaluations, ensuring our experimental evaluations are not only fair and comprehensive, but also insightful.
Model	FF++.c23	FF++.c40	CDF-v2	DFDCP	DFD	Average
FWA	0.8765	0.7357	0.6680	0.6375	0.7403	0.7316
iFWA	0.9557	0.7496	0.7612	0.7104	0.8408	0.8035
Improvement	+7.92%	+1.39%	+9.32%	+7.29%	+10.05%	+7.19%
Table 6: Cross-data evaluation between iFWA (with the spectrum feature) and FWA (without the spectrum feature). The models are trained on  $FF++\_c23$  and tested on other datasets. The metric is the frame-level AUC.
Why do Naive detectors work can perform as well as more advanced in certain settings? From results in Tab. 3, we have observed that some Naive detectors (i.e., Xception and EfficientNetB4) can exhibit competitive performance compared to more complex methods, which might be surprising given the advancements in the field. We then explain this phenomenon from the following aspects.
Figure 18: Illustration and visualization of all testing results during the training process. The metric is AUC. We compare Face X-ray and DSP-FWA on different datasets using the AUC metric.
Model	Training	FF-DF	FF-F2F	FF-FS	FF-NT	Average
FWA	FF-DF	0.90	0.91	0.92	0.90	0.91
iFWA	FF-DF	0.97	0.97	0.98	0.90	0.96
Improvement	-	+7%	+6%	+6%	+0%	+5%
Table 7: Cross-manipulation evaluation between iFWA (with the spectrum feature) and FWA (without the spectrum feature). The models are trained on FF-DF and tested on other forgeries in  $FF++\_c23$ . The metric is the frame-level AUC.
First, Naive detectors, despite their simplicity, may have inherent strengths that are yet to be fully understood and harnessed. However, few previous studies have deeply explored the capabilities of these baseline methods or identified the conditions under which they can be particularly effective. Second, previous works have shown that some strategies or tricks could bolster the performance of Naive detectors, e.g., pre-training or data augmentation. To this end, we conduct an experiment to compare the performance of the Naive detector and the complex one under the conditions with or without tricks. By adding the tricks, we find the gap between the Naive detector and complex detector
Figure 19: Illustration and visualization of all testing results during the training process. The metric is AP. We compare Face X-ray and DSP-FWA on different datasets using the AP metric.
Model	Number of Layers	Number of Parameters
Xception	71	22.9M
ResNet 34	34	21.8M
EfficientNet-B4	~75	19M
Table 8: Summary of the statistics for Xception, ResNet 34, and EfficientNet-B4.
is reduced (see Tab. 10). Showing that the performance of the Naive detector is effectively mined by using these tricks.
Apart from these two aspects, we also find that the problem of consistency in experimental procedures and evaluation metrics is also notable. It is worth noting that comparison methodologies can vary across studies, and directly adopting results from prior papers could sometimes lead to discrepancies due to differences in experimental conditions and evaluation metrics. For instance, current studies often directly cite the results of Xception from the original paper [33], but different training settings used in different works can inevitably result in disparities.
Figure 20: Illustration and visualization of all testing results during the training process. The metric is EER. We compare Face X-ray and DSP-FWA on different datasets using the EER metric.
Figure 21: Frequency analysis on each dataset. We present the average spectra of high-pass filtered images, focusing on both real and fake images. Our findings align with those reported in work [48]. We observe that the shown deepfake forgeries do not display obvious artifacts in the average spectra. This underscores the similarity of our results with [48].
Is it standard practice to use Adam optimizer for deepfake detection algorithms? In our experiments, we want to clarify it as follows:
Figure 22: Frequency analysis on each dataset using the frequency analysis tool within our benchmark. We present the average spectra of high-pass filtered images, focusing on both real and fake images.
Dataset	ResNet 34	ResNet 50	ResNet 152
CDF-v2	0.7027	0.7491	0.7514
DFDCP	0.6170	0.6658	0.7078
DFD	0.6464	0.7002	0.7005
FF+_c23	0.8493	0.8928	0.9119
FF++_c40	0.7846	0.7933	0.8167
Average	0.7199	0.7602	0.7776
Table 9: Comparing the performance of different variants of ResNet. The models are trained on  $FF + +\_ c23$  and tested on other datasets. The metric is the frame-level AUC.
- Using Adam as the optimizer can be considered a common setting, and a substantial number of deepfake detection methods in existing literature have adopted this configuration (paper [20, 27, 26, 3, 12]).
- It is important to note that our benchmark is versatile and supports various optimizers, including three mainstream ones: Adam, SGD, and AdamW. In deepfakebench, to ensure fairness and consistency across evaluations, all detectors are trained using the Adam optimizer. Other hyperparameters, such as learning rate, are also unified as much as possible.
Overall, employing the Adam optimizer aligns with common practices in deepfake detection and serves to facilitate a consistent and equitable comparison of different algorithms.
Detector	Condition	FF++_c23	CDF-v2	DFD	DFDCP	FF++_c40	Average
RECCE	Before Tricks	0.9881	0.6083	0.5894	0.5128	0.5792	0.6556
RECCE	After Tricks	0.9621	0.7319	0.8119	0.7419	0.8190	0.8134
RECCE	Improvement (%)	-2.63%	20.34%	37.84%	44.72%	41.37%	24.08%
Xception	Before Tricks	0.9893	0.5175	0.5870	0.4894	0.5420	0.6250
Xception	After Tricks	0.9637	0.7365	0.8163	0.7374	0.8261	0.8160
Xception	Improvement (%)	-2.59%	42.31%	39.08%	50.67%	52.40%	30.56%
SRM	Before Tricks	0.9882	0.5993	0.6029	0.5995	0.5754	0.6731
SRM	After Tricks	0.9576	0.7552	0.8120	0.7408	0.8114	0.8154
SRM	Improvement (%)	-3.10%	26.00%	34.74%	23.56%	40.99%	21.13%
Table 10: Comparison of performance among three detectors: Naive detector (Xception), frequency detector (SRM), and spatial detector (RECCE) under the conditions of without vs. with tricks. We demonstrate that the performance of the Naive detector can be effectively enhanced through the use of tricks: data augmentation and pre-training. The models are trained on  $FF++\_c23$ . The metric is the frame-level AUC.
Training Config	Face X-ray & FWA	Other Detectors
Image Size	256, 256	256, 256
Weight Initialization	ImageNet Pre-trained	ImageNet Pre-trained
Optimizer	Adam	Adam
Base Learning Rate	2e-4	2e-4
Weight Decay	5e-4	5e-4
Optimizer Momentum	B1, B2=0.9, 0.999	B1, B2=0.9, 0.999
Batch Size	16	32
Training Epochs	10	10
Learning Rate Schedule	None, Constant	None, Constant
Flip Probability	0.5	0.5
Rotate Probability	0.5	0.5
Rotate Limit	[-10, 10]	[-10, 10]
Blur Probability	0.5	0.5
Blur Limit	[3, 7]	[3, 7]
Brightness Probability	0.5	0.5
Brightness Limit	[-0.1, 0.1]	[-0.1, 0.1]
Contrast Limit	[-0.1, 0.1]	[-0.1, 0.1]
Quality Lower	40	40
Quality Upper	100	100
Table 11: The results of our main experiments (shown in Table. 2 and Figure. 2 of the manuscript) are generated using the following settings.