SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiyuan Yan $^{1}$ , Yong Zhang $^{2}$ , Xinhang Yuan $^{1}$ , Siwei Lyu $^{3}$ , Baoyuan Wu $^{1*}$
$^{1}$ School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China  $^{2}$ Tencent AI Lab
$^{3}$ Department of Computer Science and Engineering, University at Buffalo, State University of New York, USA
A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. This issue leads to unfair performance comparisons and potentially misleading results. Specifically, there is a lack of uniformity in data processing pipelines, resulting in inconsistent data inputs for detection models. Additionally, there are noticeable differences in experimental settings, and evaluation strategies and metrics lack standardization. To fill this gap, we present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions: 1) a unified data management system to ensure consistent input across all detectors, 2) an integrated framework for state-of-the-art methods implementation, and 3) standardized evaluation metrics and protocols to promote transparency and reproducibility. Featuring an extensible, modular-based codebase, DeepfakeBench contains 15 state-of-the-art detection methods, 9 deepfake datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations. Moreover, we provide new insights based on extensive analysis of these evaluations from various perspectives (e.g., data augmentations, backbones). We hope that our efforts could facilitate future research and foster innovation in this increasingly critical domain. All codes, evaluations, and analyses of our benchmark are publicly available at https://github.com/SCLBD/DeepfakeBench.
Deepfake, widely recognized for its facial manipulation, has gained prominence as a technology capable of fabricating videos through the seamless superimposition of images. The surging popularity of deepfake technology in recent years can be attributed to its diverse applications, extending from entertainment and marketing to more complex usages. However, the proliferation of deepfake is not without risks. The same tools that enable creativity and innovation can be manipulated for malicious intent, undermining privacy, promoting misinformation, or eroding trust in digital media, etc.
Responding to the risks posed by deepfake contents, numerous deepfake detection methods [53, 22, 33, 32, 52, 3] have been developed to distinguish deepfake contents from real contents, which are generally categorized into three types: naive detector, spatial detector, and frequency detector. Despite rapid advancements in deepfake detection technologies, a significant challenge remains due to the lack of a standardized, unified, and comprehensive benchmark for a fair comparison among different detectors. This issue causes three major obstacles to the development of the deepfake detection field. First, there is a remarkable inconsistency in the training configurations and evaluation standards utilized in the field. This discrepancy inevitably leads to divergent outcomes, making a fair
*Corresponding author: Baoyuan Wu (wubaoyuan@cuhk.edu.cn)
Model Type	Detectors	Backbone	Repositories	Reference
Naive Detector	MesoNet [1]	Designed CNN	https://github.com/DariusAf/MesoNet	WIFS-2018
Naive Detector	MesoInception [1]	Designed CNN	https://github.com/DariusAf/MesoNet	WIFS-2018
Naive Detector	CNN-Aug [48]	ResNet [16]	https://peterwang512.github.io/CNNDetection/	CVPR-2020
Naive Detector	EfficientNet-B4 [40]	EfficientNet [40]	https://github.com/Lukemelas/EfficientNet-PyTorch	ICML-2019
Naive Detector	Xception [33]	Xception [5]	https://github.com/ondyari/FaceForensics	ICCV-2019
Spatial Detector	Capsule [29]	Designed Capsule [34]	https://github.com/nii-yamagishilab/Capsule-Forensics-v2	ICASSP-2019
Spatial Detector	DSP-FWA [22]	Xception [5]	https://github.com/dammohaha/CVPRW2019_Face_Artifacts	CVPRW-2019
Spatial Detector	Face X-ray [20]	HRNet [46]	Unpublished code, reproduced by us	CVPR-2020
Spatial Detector	FFD [6]	Xception [5]	cvlab.cse.msu.edu/project-fdd.html	CVPR-2020
Spatial Detector	CORE [30]	Xception [5]	https://github.com/niyunsheng/CORE	CVPRW-2022
Spatial Detector	RECCE [2]	Designed Networks	https://github.com/VISION-SJTU/RECCE	CVPR-2022
Spatial Detector	UCF [50]	Xception [5]	Unpublished code, reproduced by us	ICCV-2023
Frequency Detector	F3Net [32]	Xception [5]	Unpublished code, reproduced by us	ECCV-2020
Frequency Detector	SPSL [26]	Xception [5]	Unpublished code, reproduced by us	CVPR-2021
Frequency Detector	SRM [27]	Xception [5]	Unpublished code, reproduced by us	CVPR-2021
Table 1: Summary of the compared deepfake detectors. For detectors without publicly available repositories, we undertake careful re-implementation, adhering to the instructions specified in the original papers.
comparison difficult. Second, the source codes of many methods are not publicly released, which could be detrimental to the reproducibility and comparability of their reported results. Third, we find that the detection performance can be significantly influenced by several seemingly inconspicuous factors, e.g., the number of selected frames in a video. Since the settings of these factors are not uniform and their impacts are not thoroughly studied in most existing works, the reported results and corresponding claims may be biased or misleading.
To bridge this gap, we build the first comprehensive benchmark, called DeepfakeBench, offering a unified platform for deepfake detection. Our main contributions are threefold. 1) An extensible modular-based codebase: Our codebase consists of three main modules. The data processing module provides a unified data management module to guarantee consistency across all detection inputs, such that alleviating the time-consuming data processing and evaluation. The training module provides a modular framework to implement state-of-the-art detection algorithms, facilitating direct comparisons among different detection algorithms. The evaluation and analysis module provides several widely adopted evaluation metrics and rich analysis tools to facilitate further evaluations and analysis. 2) Comprehensive evaluations: We evaluate 15 state-of-the-art detectors with 9 deepfake datasets under a wide range of evaluation settings, providing a holistic performance evaluation of each detector. Moreover, we establish a unified evaluation protocol that enhances the transparency and reproducibility of performance evaluation. 3) Extensive analysis and new insights: We provide extensive analysis from various perspectives, not only analyzing the effects of existing algorithms but also uncovering new insights to inspire new technologies. In summary, we believe DeepfakeBench could constitute a substantial step towards calibrating the current progress in the deepfake detection field and promoting more innovative explorations in the future.
Deepfake Generation Deepfake technology, which generally centers on the artificial manipulation of facial imagery, has made considerable strides from its rudimentary roots. Starting in 2017, learning-based manipulation techniques have made significant advancements, with two prominent methods gaining considerable attention: Face-Swapping and Face-Reenactment. 1) Face-swapping constitutes a significant category of deepfake generation. These techniques typically involve autoencoder-based manipulations, which are based on two autoencoders with a shared encoder and two different decoders. The autoencoder output is then blended with the rest of the image to create the forgery image. Notable face-swapping datasets of this approach include UADFV [21], FF-DF [7], CelebDF [23], DFD [9], DFDC [8], DeeperForensics-1.0 [17], and ForgeryNet [31]. 2) Face-reenactment is characterized by graphics-based manipulation techniques that modify source faces imitating the expressions of a different face. NeuralTextures [41] and Face2Face [42], utilized in FaceForensics++, stand out as standard face-reenactment methods. Face2Face uses key facial points to generate varied expressions, while NeuralTexture uses rendered images from a 3D face model to migrate expressions.
Deepfake Detection Current deepfake detection can be broadly divided into three categories: naive detector, spatial detector, and frequency detector. 1) Naive detector employs CNNs to directly distinguish deepfake content from authentic data. Numerous CNN-based binary classifiers have been proposed, e.g., MesoNet [1] and Xception [33]. 2) Spatial detector delves deeper into specific representation such as forgery region location [28], capsule network [29], disentanglement learning [50, 24], image reconstruction [2], erasing technology [45], etc. Besides, some other
Feature / Paper	DeepfakeBench	Paper [25]
Scope of Deepfake	Face-swapping + Diffusion + GAN	Face-swapping
Number of Detectors	15	11
Number of Datasets	9	8
Code Open Source	✓	Not yet
Modular and Extensible Codebase	✓	-
User-Friendly APIs	✓	-
Customizable Preprocessing Module	✓	-
Unified Training Framework	✓	-
Rich Analysis Tools	✓	✓
Analysis of FLOPs	-	✓
Evaluation Metrics	AUC, AP, ACC, EER, Precision, Recall	AUC
Table 2: Comprehensive comparison of our benchmark with existing benchmark [25].
methods specifically focus on the detection of blending artifacts [22, 20, 3], generating forged images during training in a self-supervised manner to boost detector generalization. 3) Frequency detector addresses this limitation by focusing on the frequency domain for forgery detection [13, 32, 26, 27]. SPSL [26] and SRM [27] are other examples of frequency detectors that utilize phase spectrum analysis and high-frequency noises, respectively. Qian et al. [32] propose the use of learnable filters for adaptive mining of frequency forgery clues using frequency-aware image decomposition.
Related Deepfake Surveys and Benchmarks The growing implications of deepfake technology have sparked extensive research, resulting in the establishment of several surveys and dataset benchmarks in the field. 1) Surveys provide a detailed examination of various facets of deepfake technology. For instance, Westerlund et al. [49] present a thorough analysis of deepfake, emphasizing its legal and ethical dimensions. Tolosana et al. [43] furnish a comprehensive review of face manipulation techniques, including deepfake methods, along with approaches to detect such manipulations. 2) Benchmarks in this field have emerged as essential tools to provide realistic forgery datasets. For instance, FaceForensics++ (FF++) [33] serves as a prominent benchmark, offering high-quality manipulated videos and a variety of forgery types. The Deepfake Detection Challenge Dataset (DFDC) [10] introduces a diverse range of actors across different scenarios.
While these benchmarking methodologies have made significant contributions, they specifically focus on their own datasets, without offering a standardized way to handle data across different datasets, which may lead to inconsistencies and obstacles to fair comparisons. Also, the lack of a unified framework in some benchmarks could lead to variations in training strategies, settings, and augmentations, which may result in discrepancies in the outcomes. Furthermore, the provision of comprehensive analytical tools is not always prominent, which might restrict the depth of analysis on the potential impacts of different factors. One notable work [25] aims to build a benchmark for evaluating various detectors under different datasets. Another recent work [18] introduces a benchmark centered around detecting GAN-generated images using continual learning. However, these two benchmarks still lack a modular, extensible, and comprehensive codebase that includes data preprocessing, unified settings, training modules, evaluations, and a series of analytical tools. DeepfakeBench, on the other hand, presents a concise but comprehensive benchmark. Its contributions are threefold: introducing a unified data management system for consistency, offering an integrated framework for implementing advanced methods, and analyzing the related factors with a series of analysis tools. Detailed comparisons between our DeepfakeBench and [25] are shown in Tab.2.
3 Our Benchmark
3.1 Datasets and Detectors
Datasets Our benchmark currently incorporates a collection of 9 widely recognized and extensively used datasets in the realm of deepfake detection: FaceForensics++ (FF++) [33], CelebDF-v1 (CDFv1) [23], CelebDF-v2 (CDFv2) [23], DeepFakeDetection (DFD) [9], DeepFake Detection Challenge Preview (DFDC-P) [11], DeepFake Detection Challenge (DFDC) [10], UADFV [21], FaceShifter (Fsh) [19], and DeeperForensics-1.0 (DF-1.0) [17]. Notably, FF++ contains 4 types of manipulation methods: Deepfakes (FF-DF) [7], Face2Face (FF-F2F) [42], FaceSwap (FF-FS) [14], NeuralTextures (FF-NT) [41]. There are three versions of FF++ in terms of compression level, i.e.,
Figure 1: The general structure of the modular-based codebase of DeepfakeBench.
raw, lightly compressed (c23), and heavily compressed (c40). The detailed descriptions of each dataset are presented in the Sec. A.3 of the Appendix. Typically,  $\mathrm{FF} + +$  is employed for model training, while the rest are frequently used as testing data. However, our benchmark allows users to select their combinations of training and testing data, thus encouraging custom experimentation.
It is notable that, although these datasets have been widely used in the community, they are not usually provided in a readily accessible and combined format. It often requires a substantial investment of time and effort in data sourcing, pre-processing (e.g., frame extraction, face cropping, and face alignment), and organization of the raw datasets, which are often organized in diverse structures. This considerable data preparation overhead often diverts researchers' attention away from the core tasks like methodology design and experimental evaluations. To tackle this challenge, our benchmark offers a collection of well-processed and systematically organized datasets, allowing researchers to devote more time to the core tasks. Additionally, our benchmark enriches some datasets (e.g.,  $\mathrm{FF} + +$  [33] and DFD [9]), by including mask data (i.e., the forgery region) that is aligned with the respective facial images in these datasets. It could facilitate more comprehensive deepfake detection studies. In summary, our benchmark provides a unified, user-friendly, and diversified data resource for the deepfake detection community. It eliminates the cumbersome task of data preparation and allows researchers to concentrate more on innovating effective deepfake detection methods.
Detectors Our benchmark has implemented a total of 15 established deepfake detection algorithms, as detailed in Tab. 1. The selection of these algorithms is guided by three criteria. First, we prioritize methods that hold a classic status (e.g., Xception), or those considered advanced, typically published in recent top-tier conferences or journals in computer vision or machine learning. Second, our benchmark classifies detectors into three categories: naive detectors, spatial detectors, and frequency detectors. Our primary emphasis is on image forgery detection, hence, temporal-based detectors have not yet been incorporated. Moreover, we have refrained from including traditional detectors (e.g., Headpose [51]) due to their limited scalability to large-scale datasets, making them less suitable for our benchmark's objectives. Third, we aim to include methods that are straightforward to implement and reproduce. We notice that several existing methods involve a series of steps, some of which are reliant on third-party algorithms or heuristic strategies. These methods usually have numerous hyper-parameters and are fraught with uncertainty, making their implementation and reproduction challenging. Therefore, these methods without open-source codes are intentionally excluded from our benchmark. However, it is important to note that there are also some non-open-source methods we employed that are derived from the code directly provided by their respective authors.
3.2 Codebase
We have built an extensible modular-based codebase as the basis of DeepfakeBench. As shown in Fig. 1, it consists of three core modules, including Data Processing Module, Training Module, and Evaluation and Analysis Module.
Data Processing Module The Data Processing Module includes two pivotal sub-modules that automate the data processing sequence, namely the Data Preprocessing and Data Arrangement sub-modules. 1) Data preprocessing sub-module presents a streamlined solution. First, Users are provided with a YAML configuration file, enabling them to tailor the preprocessing steps to their specific requirements. Second, we furnish a unified preprocessing script, which includes frame extraction, face cropping, face alignment, mask cropping, and landmark generation. 2) Data arrangement sub-module further augments the convenience of data management. This sub-module comprises a suite of JSON files for each dataset. Users can execute a rearranged script to create a unified JSON file for each dataset. This unified file provides access to the corresponding training, testing, and validation sets, along with other information such as the frames, landmarks, masks, etc, related to each dataset.
Training Module The Training Module currently accommodates 15 detectors across three categories: naive detector, spatial detector, and frequency detector, all of which are shown in Tab. 1. 1) Naive detector leverages various CNN architectures to directly detect forgeries without relying on additional manually designed features. 2) Spatial detector builds upon the backbone of CNNs used in the Naive Detector and further explores manual-designed algorithms to detect deepfake. 3) Frequency detector focuses on utilizing information from the frequency domain and extracting frequency artifacts for detection. Each detector implemented in our benchmark is managed in a streamlined and efficient way, with a YAML config file created for each one. This allows users to easily set their desired parameters, e.g., batch size, learning rate, etc. These detectors are trained on a unified trainer that records the metrics and losses during the training and evaluation process. Thus, the training and evaluation processes, logging, and visualization are handled automatically, eliminating the need for manual specification.
Evaluation and Analysis Module For evaluation, we employ 4 widely used evaluation metrics: accuracy (ACC), the area under the ROC curve (AUC), average precision (AP), and equal error rate (EER). Besides, it is notable that there is an inconsistency in the usage of these evaluation metrics in the community, some are at the frame level, while others are at the video level, leading to unfair comparisons. Our benchmark currently adopts the frame level evaluation to build a fair basis for comparison among detectors. In addition to the evaluation values of these metrics, we also provide several visualizations to facilitate performance comparisons, e.g., the ROC-AUC curve, radar chart, and histogram. For analysis, we provide various visualization tools to gain deeper insights into the detectors' performance. For example, Grad-CAM [36] is used to highlight the potential forgery regions detected by the models, providing interpretability and assisting in understanding the underlying reasoning for the model's predictions. To explore the learned features and representations, we employ t-SNE visualization [44]. Furthermore, we offer custom visualizations tailored to specific detectors. For example, for Face X-ray [20], we provide visualizations of the detection boundary of the face, as described in its original paper (see the top-right corner of Fig. 1).
4 Evaluations and Analysis
In the data processing, face detection, face cropping, and alignment are performed using DLIB [35]. The aligned faces are resized to  $256 \times 256$  for both the training and testing. In the training module, we employ the Adam optimization algorithm with a learning rate of 0.0002. The batch size is fixed at 32 for all experiments. We sample 32 frames for each video for training and testing. We primarily leverage pre-trained backbones from ImageNet if feasible. Otherwise, we resort to initializing the remaining weights using a normal distribution. We also apply widely used data augmentations, i.e., image compression, horizontal flip, rotation, Gaussian blur, and random brightness contrast. In terms of evaluation, we compute the average value of the top-3 metrics (e.g., average top-3 AUC) as our evaluation metric. We also report other metrics (i.e., AP, EER, Precision, and Recall) in the Sec. A.3 of the Appendix. Further details of dataset configuration, algorithms implementation, and full training details can be seen in the Sec. A.1, Sec. A.2, and Sec. A.3 of the Appendix, respectively.
Type	Detector	Backbone	Within Domain Evaluation	Cross Domain Evaluation
FF-c23	FF-c40	FF-DF	FF-F2F	FF-FS	FF-NT	Avg.	Top3	CDFv1	CDFv2	DF-1.0	DFD	DFDC	DFDCP	Fsh	UADFV	Avg.	Top3
Naive	Meso4 [1]	MesoNet	0.6077	0.5920	0.6771	0.6170	0.5946	0.5701	0.6097	0	0.7358	0.6091	0.9113	0.5481	0.5560	0.5994	0.5660	0.7150	0.6551	1
Naive	MesoIncep [1]	MesoNet	0.7278	0.8542	0.8087	0.7421	0.6517	0.7571	0	0.7366	0.6966	0.9233	0.6069	0.6226	0.7561	0.6438	0.9049	0.7364	3
Naive	CNN-Aug [48]	ResNet	0.8493	0.7846	0.9048	0.8788	0.9026	0.7313	0.8419	0	0.7420	0.7027	0.7993	0.6464	0.6361	0.6170	0.5985	0.8739	0.7020	0
Naive	Xception [33]	Xception	0.9637	0.8261	0.9799	0.9785	0.9833	0.9385	0.9450	4	0.7794	0.7365	0.8341	0.8163	0.7077	0.7374	0.6249	0.9379	0.7718	2
Naive	EfficientB4 [40]	Efficient	0.9567	0.8150	0.9757	0.9758	0.9797	0.9308	0.9389	0	0.7909	0.7487	0.8330	0.8148	0.6955	0.7283	0.6162	0.9472	0.7718	3
Spatial	Capsule [29]	Capsule	0.8421	0.7040	0.8669	0.8634	0.8734	0.7804	0.8217	0	0.7909	0.7472	0.9107	0.6841	0.6465	0.6568	0.6465	0.9078	0.7488	2
Spatial	FWA [22]	Xception	0.8765	0.7357	0.9210	0.9000	0.8843	0.8120	0.8549	0	0.7897	0.6680	0.9334	0.7403	0.6132	0.6375	0.5551	0.8539	0.7239	1
Spatial	X-ray [20]	HRNet	0.9592	0.7925	0.9794	0.9872	0.9871	0.9290	0.9391	3	0.7093	0.6786	0.5531	0.7655	0.6326	0.6942	0.6553	0.8989	0.6985	0
Spatial	FFD [6]	Xception	0.9624	0.8237	0.9803	0.9784	0.9853	0.9306	0.9434	1	0.7840	0.7435	0.8609	0.8024	0.7029	0.7426	0.6056	0.9450	0.7733	1
Spatial	CORE [30]	Xception	0.9638	0.8194	0.9787	0.9803	0.9823	0.9339	0.9431	2	0.7798	0.7428	0.8475	0.8018	0.7049	0.7341	0.6032	0.9412	0.7694	0
Spatial	Recce [2]	Designed	0.9621	0.8190	0.9797	0.9779	0.9785	0.9357	0.9422	1	0.7677	0.7319	0.7985	0.8119	0.7133	0.7419	0.6095	0.9446	0.7649	2
Spatial	UCF [50]	Xception	0.9705	0.8399	0.9883	0.9840	0.9896	0.9441	0.9527	6	0.7793	0.7527	0.8241	0.8074	0.7191	0.7594	0.6462	0.9528	0.7801	5
Frequency	F3Net [32]	Xception	0.9635	0.8271	0.9793	0.9796	0.9844	0.9354	0.9449	1	0.7769	0.7352	0.8431	0.7975	0.7021	0.7354	0.5914	0.9347	0.7645	0
Frequency	SPSL [26]	Xception	0.9610	0.8174	0.9781	0.9754	0.9829	0.9299	0.9408	0	0.8150	0.7650	0.8767	0.8122	0.7040	0.7408	0.6437	0.9424	0.7875	3
Frequency	SRM [27]	Xception	0.9576	0.8114	0.9733	0.9696	0.9740	0.9295	0.9359	0	0.7926	0.7552	0.8638	0.8120	0.6995	0.7408	0.6014	0.9427	0.7760	2
Table 3: Within-domain and cross-domain evaluations using the AUC metric. All detectors are trained on FF-c23 and evaluated on other data. "Avg." donates the average AUC for within-domain and cross-domain evaluation, and the overall results. "Top3" represents the count of each method ranks within the top-3 across all testing datasets. The best-performing method for each column is highlighted in red.
Figure 2: Visualization of heat maps showing the cross-manipulation evaluation results. The color represents the AUC performance index of the corresponding detector under specific test data, and the darker the color, the better the performance. All heat maps use a uniform color scale for performance comparison.
4.2 Evaluations