SlowGuess/NeurIPS_2023_OCR huggingface dataset:
facial analysis algorithms. Several algorithms, such as Face X-ray [20], FWA [22], SLADD [3], etc, rely on landmarks to perform operations and analysis on facial images. By extracting landmarks during the preprocessing step, we aim to provide a comprehensive dataset that includes both the aligned face images and the corresponding landmark coordinates. Users can leverage landmarks to develop and train models without the need for additional face-detection steps during training, thereby reducing computational overhead and improving training speed.
Mask Extraction (Optional) In some deepfake datasets (i.e., FaceForensics++ [33] and DFD [9]), an additional mask is provided, indicating the regions of the face that are manipulated or modified. Also, we see that there are several works that rely on mask data for the detection, e.g., Multi-task [28], Face X-ray [20], M2TR [47], etc. Thus, if the dataset includes masks, our script also extracts and saves these masks. Since we have performed the face alignment and cropping operations in the previous steps, we need to do the same operations for the mask data. To extract masks, we utilize the additional video files provided by the authors that contain the mask information for each frame. Note these mask video files have the same frame count and frame rate as the original video. During the face cropping step, if a mask video file is provided, we extract the corresponding frames and masks for each video. The mask data is saved as a separate folder but with the same dictionary as the videos and frames. The masks can be used to identify specific areas of interest for further analysis or to train models that specifically focus on detecting manipulated regions.
Frame Sampling In the preprocessing pipeline, we incorporate frame sampling techniques to strike a balance between computational requirements and maintaining a diverse set of examples. This step aims to extract a subset of frames from each video in the dataset. The frame sampling process depends on the specified mode, which can be either "fixed_num_frames" or "fixed_stride". In the "fixed_num_frames" mode, we extract a fixed number of frames from each video. This approach ensures that the resulting dataset contains a consistent number of frames for each video, regardless of the video's duration. By selecting a predetermined number of frames, we obtain a manageable dataset size that is suitable for subsequent analysis or model training. In the "fixed_stride" mode, we sample frames with a fixed stride. This means that we skip a certain number of frames between each frame that is selected. This approach allows us to capture frames at regular intervals throughout the video, providing a representative sampling of the temporal dynamics. By choosing an appropriate stride, we can control the density of the selected frames and adjust the amount of temporal information included in the dataset. Frame sampling serves two primary purposes. Firstly, it reduces the computational requirements for subsequent steps in the pipeline, such as face detection and alignment, by operating on a subset of frames rather than the entire video. This improves the overall efficiency of the preprocessing process, particularly when dealing with large-scale datasets. Secondly, frame sampling ensures that the resulting dataset maintains a diverse set of examples. By selecting frames at regular intervals or a fixed number of frames per video, we capture different facial expressions, poses, and actions exhibited by individuals. This diversity enhances the generalizability of models trained on the dataset, enabling them to handle a wide range of scenarios and variations encountered in real-world applications. Note in this paper we only choose the "fixed_num Frames" mode.
Parallel Processing To improve the processing speed, we use parallel processing techniques. We leverage the concurrent.futures library, which provides a high-level interface for asynchronously executing callables. By using multiple processes, we can process multiple videos simultaneously, significantly reducing the overall processing time. The number of processes used is determined based on the CPU capabilities of the system. We assign one process per CPU core to maximize the utilization of available resources.
Saving Processed Data After completing the preprocessing steps, we save the processed data for future use. The cropped face images, extracted landmarks, and masks (if available) are saved in a structured directory format. Each video is associated with a separate directory, containing the processed frames, landmarks, and masks (if applicable). This organization allows for efficient data retrieval and analysis during subsequent stages.
Arrangement The process of rearranging the dataset structure is motivated by the need for a unified and convenient way to load different datasets. Each dataset typically has its own distinct structure and organization, making it hard and troublesome to handle them uniformly. This could involve writing separate input/output (I/O) code for each dataset, leading to duplication of effort and potential difficulties in managing the data.
To this end, we adopt a unified approach by organizing and managing the dataset information using a JSON file. This enables a standardized structure that subsequent algorithms and models can easily process. By leveraging the JSON file format, we provide a comprehensive and adaptable representation of the dataset, accommodating the specific requirements and characteristics of each dataset. The rearranged structure organizes the data in a hierarchical manner, grouping videos based on their labels and data splits (i.e., train, test, validation). Each video is represented as a dictionary entry containing relevant metadata, including file paths, labels, compression levels (if applicable), etc. This unified representation facilitates streamlined dataset loading and handling, eliminating the need for dataset-specific I/O code.
The JSON file serves as a centralized repository of dataset information, providing a consistent and easily accessible format. Users can leverage existing code and tools to parse and analyze the JSON file, promoting reproducibility and facilitating collaborations across different datasets. Additionally, the JSON file simplifies the data preprocessing pipeline, reducing duplication of effort and enhancing the efficiency of subsequent data analysis and model training processes.
The whole process of data preprocessing and arrangement can be summarized in the following Algorithm. 1.
Algorithm 1 Data Preprocessing and Arrangement
1: Input: Video dataset
2: Output: Preprocessed dataset with rearranged structure
3: Procedure:
4: Perform the following preprocessing steps for each video in the dataset:
5: Extract a subset of frames from each video using frame sampling techniques.
6: Detect faces in each video frame using the Dlib face detection algorithm.
7: Align and crop the faces based on the detected facial landmarks using Dlib shape predictor model.
8: (Optional) Extract and save masks for each aligned face if provided.
9: Extract landmarks for each detected face using Dlib shape predictor model.
10: Save the processed face images, landmarks, and masks (if applicable) in separate folders.
11: Use parallel processing to speed up the overall processing time by processing multiple videos simultaneously.
12: Save the processed data in a structured directory format with a JSON file containing metadata.
13: Return the rearranged dataset structure with metadata stored in the JSON file.
Configuration The provided config file contains settings for two different preprocessing tasks: "preprocess" and "rearrange". We will go through each section and explain the available settings and their advantages in this section.
For the Preprocess:
- dataset_name: This setting allows the user to specify the name of the dataset. Users can choose from a list of supported dataset names such as FaceForensics++ [33], Celeb-DF-v1 [23], Celeb-DF-v2 [23], DFDCP [11], DFDC [10], DeeperForensics-1.0 [17], and UADFV [21]. Each dataset has its own characteristics and purpose.
- dataset_root_path: This setting defines the root path where the dataset is located. Users need to provide the path to the dataset directory.
- comp: This setting is specific to the FaceForensics++ dataset and determines the compression level of the videos. Users can choose from "raw", "c23", or "c40". Different compression levels have different trade-offs between video quality and file size.
- mode: This setting determines the mode of preprocessing, either "fixed_num_frames" or "fixed_stride". In "fixed_num Frames" mode, users can specify the number of frames to extract from each video using the "num_frames" setting. In "fixed_stride" mode, users can specify the number of frames to skip between each frame extracted using the "stride" setting.
- stride: This setting is used when the mode is set to "fixed_stride". It determines the number of frames to skip between each frame extracted. A higher stride value will result in fewer extracted frames.
- num_frames: This setting is used when the mode is set to "fixed_num Frames". It specifies the number of frames to extract from each video. Extracting a fixed number of frames allows for consistent and manageable data sizes.
For the Arrangement:
- dataset_name: This setting allows users to specify the name of the dataset users want to rearrange.
- dataset_root_path: This setting defines the root path where the dataset is located.
- output_file_path: This setting specifies the path where the output JSON file will be saved. The JSON file contains information about the rearranged dataset.
- comp: This setting is specific to the FaceForensics++ dataset and determines the compression level of the videos. Users can choose from "raw", "c23", or "c40".
- perturbation: This setting is specific to the DeeperForensics-1.0 dataset and allows users to select different levels of perturbations to apply to the dataset. There are options such as "end_to_end", "end_to_end_level_1", "end_to_end_mix_2_distortions", etc.
Dataset rearrangement is specifically designed for rearranging datasets. It provides the flexibility to modify and rearrange the dataset according to specific needs. The script generates a JSON file that contains information about the rearranged dataset. This file can be used for further analysis or as input to other scripts or models. By using this config file, users can easily customize the preprocessing and rearrangement tasks to suit their specific dataset and requirements. The flexibility offered by this file enables efficient and consistent preprocessing of various deepfake datasets.
A.2 Details of Algorithms Implementation and Visualizations
Algorithms Implementation In addition to the basic information in Tab. 2 of the main manuscript, here we describe the general idea of the 15 implemented detection algorithms in the DeepfakeBench, as follows.
1) Meso4 [1]: is a CNN-based deepfake detection method targeting the mesoscopic properties of images. The model is trained on unpublished deepfake datasets collected by the authors. We evaluate two variants of MesoNet, namely, Meso4 and MesoIncep. Meso4 uses conventional convolutional layers.
2) MesoIncep [1]: this detector, similar to Meso4, utilizes a designed CNN architecture and is also implemented in the MesoNet repository. Note that MesoIncep is based on the more sophisticated Inception modules [39].
3) CNN-Aug [48]: detects GAN-generated images using a ResNet [16] with widely-used augmentations. In the DeepfakeBench, we employ a ResNet-34 [16] with JPEG compression and Gaussian
blurring augmentations, etc. The effect of augmentations we used in this work has been explored in Sec. 4 in the main paper. The specific settings of the augmentations can be found in the following section in Sec. A.3.
Figure 8: Illustration and visualization of the DSP-FWA algorithm. We use the data from FaceForensics++ [33] and apply some augmentations to the source image, as well as the blending image.
13) F3Net [32]: uses cross-attention two-stream networks to collaboratively learn frequency-aware clues from two branches: FAD and LFS, where the FAD module partitions the input image in the frequency domain based on learnable frequency bands and represents the image with frequency-aware components to learn forgery patterns through frequency-aware image decomposition, and the LFS module extracts localized frequency statistics to describe statistical discrepancies between real and fake faces, allowing for effective mining through CNNs and revealing unusual statistics of forgery images at each frequency band while sharing the structure of natural images. This detector utilizes the Xception backbone [5]. The code for this detector is not publicly available, we re-implement it carefully following the instructions and settings in the original paper.
14) SPSL [26]: combines spatial image and phase spectrum to capture the up-sampling artifacts of face forgery to improve the transferability (generalization ability), for face forgery detection. This paper theoretically analyzes the validity of utilizing the phase spectrum. Moreover, this paper notices that local texture information is more crucial than high-level semantic information for face forgery detection. This detector utilizes the Xception backbone [5]. The code for this detector is not publicly available, we re-implement it carefully following the instructions and settings in the original paper.
15) SRM [27]: extracts high-frequency noise features and fuses two different representations from RGB and frequency domains to improve the generalization ability. This detector utilizes the Xception architecture [6]. This detector utilizes the Xception backbone [5]. The code for this detector is not publicly available, we re-implement it carefully following the instructions and settings in the original paper.
Visualizations We implement all 15 detectors mentioned above. However, not all of them have publicly available code, so we implement some of them ourselves following the settings and instructions provided in the original papers. This allowed us to verify the correctness of our implementation and gain a better understanding of these detectors. To further assess the performance and behavior of the detectors, we conduct visualizations of the results for 2 specific detectors: DSP-FWA [22] and Face X-ray [20].
1) DSP-FWA: Note that the official code for DSP-FWA does not include the training code or the code for dynamically generating forgery data using self-blending in each iteration during training. To this end, we make use of certain parts of the code provided in the official repository and implement the training process and forgery data generation ourselves. In our implementation of DSP-FWA, we use the Xception network [5] as the backbone. This choice is to ensure consistency in the benchmark by using the same backbone network across different detectors. By doing so, we could focus solely on evaluating the algorithmic performance of DSP-FWA itself. By incorporating our own implementation of the training process and forgery data generation, we are able to overcome the absence of these components in the official code. This allows us to thoroughly evaluate DSP-FWA and ensure a fair comparison with other detectors in our benchmark. We visualize the original images, blending masks, and blending images in Fig. 8.
2) Face X-ray: Note that the official code for Face X-ray is not available. So we re-implement the data manipulation and training process carefully following the instructions of the original paper. The visualizations can be seen in Fig. 10.
Figure 9: t-SNE visualization of FWA-generated data. By assigning distinct labels to various forgeries, we enhance the clarity of their representation within the feature space.
Figure 10: Illustration and visualization of the Face X-ray algorithm. We use the data from FaceForensics++ [33] and apply some augmentations to the source image, as well as the blending image.
Furthermore, we conduct a t-SNE analysis for FWA, visualizing labels in the feature space. Our findings suggest that images generated through blending technology (new data generated by FWA) exhibit distinctiveness, distancing them from images generated by alternative manipulation methodologies. This characteristic enlarges the forgery space, culminating in enhanced generalization capabilities.
A.3 Training Details and Full Experimental Results
Datasets Our benchmark currently incorporates a collection of 9 widely recognized and extensively used datasets in the realm of deepfake forensics: FaceForensics++ (FF++) [33], CelebDF-v1 [23], CelebDF-v2 [23], DeepFakeDetection (DFD) [9], DeepFake Detection Challenge Preview (DFDC-P) [11], DeepFake Detection Challenge (DFDC) [10], UADFV [21], FaceShifter [19], and DeeperForensics-1.0 (DF-1.0) [17]. The detailed descriptions of each dataset are presented in Tab. 5.
The dataset splitting for different datasets used in deepfake detection is described as follows:
1) FaceForensics++ (FF++): The FF++ dataset is divided into several subsets, including FF-DF, FF-F2F, FF-FS, FF-NT, and FF-all. Each subset corresponds to a combination of deepfake and real videos from YouTube. In the real dataset, the data is duplicated and split into three sets: train, test,
Dataset	Real Videos	Fake Videos	Total Videos	Rights Cleared	Total Subjects	Synthesis Methods	Perturbations	Download Link
FF++[33]	1000	4000	5000	NO	N/A	4	2	Hyper-link
FaceShifter [19]	1000	1000	2000	NO	N/A	1	-	Hyper-link
DFD [9]	363	3000	3363	YES	28	5	-	Hyper-link
DFDC-P [11]	1131	4119	5250	YES	66	2	3	Hyper-link
DFDC [10]	23,654	104,500	128,154	YES	960	8	19	Hyper-link
CelebDF-v1 [23]	408	795	1203	NO	N/A	1	-	Hyper-link
CelebDF-v2 [23]	590	5639	6229	NO	59	1	-	Hyper-link
DF-1.0 [17]	50,000	10,000	60,000	YES	100	1	7	Hyper-link
UADFV [21]	49	49	98	NO	49	1	-	Hyper-link
Table 5: Summary of the datasets used for deepfake detection. The table provides information on the number of real and fake videos, the total number of videos, whether rights have been cleared, the number of agreeing subjects, the total number of subjects, the number of synthesis methods, and the number of perturbations.
and validation. For the fake dataset, the train, test, and validation splits are determined based on the information provided in the corresponding JSON files used in the arrangement process (see Sec. A.1). Masks are also included in the dataset.
2) DeepFakeDetection: Since the dataset does not have the official splitting, the fake and real data are duplicated and split into the train, test, and validation sets in our benchmark. Masks are included in this dataset as well.
3) FaceShifter: The real data is duplicated and split into the train, test, and validation sets, similar to the FF++ dataset. The train, test, and validation splits for the fake dataset are determined using the FF++ JSON files used in the arrangement process.
4) Celeb-DF-v1/v2: All the real and fake videos are used as the training dataset, and a subset of real and fake videos is selected as the test dataset based on a text file provided by the author. The validation set is set to be the same as the test set.
5) DFDCP: The dataset contains real videos and fake videos generated by two different methods: method A and method B. The train and test splits are determined based on the given method. The validation set is set to be the same as the test set.
6) DFDC: The train and test splits are determined based on the given method, similar to DFDCP. The validation set is set to be the same as the test set.
7) DeeperForensics-1.0: The dataset includes various perturbation methods in the fake data subset. One perturbation method is considered a separate category of fake videos. In the fake dataset, the train, test, and validation splits are determined based on the provided text file. The real dataset is duplicated and split into train, test, and validation sets.
8) UADFV: The strategy used for the UADFV dataset involves duplicating the real and fake parts of the dataset three times to create the train, test, and validation sets.
Experimental Setup In the training module, we utilize the Adam optimization algorithm with a learning rate of 0.0002. The batch size is set to 32 for most experiments. However, for the DSP-FWA [22] and Face X-ray [20] detectors, the batch size is adjusted to 16 due to the input data being pairs. Specifically, for DSP-FWA and Face X-ray, which generate forgery images dynamically during training, the input size is doubled.
For the naive detectors (e.g., ResNet, Xception, and EfficientNet), we employ their official models, initializing the parameters through pre-training on the ImageNet. The pre-trained backbones from ImageNet are used to initialize the remaining weights. However, Meso4 [1] and MesoIncep [1] do not have pre-training weights in ImageNet, so pre-training is not utilized for them. The effect of pre-training is evaluated in Sec. 4.3 of the main paper.
Regarding evaluation, we compute the average value of the top-3 metrics, such as the average top-3 Area Under the Curve (AUC), as our primary evaluation metric. Additionally, we report the top-1 results. Other widely used metrics, including Average Precision (AP) and Equal Error Rate (EER), are also computed and presented in the following sections. Furthermore, it is important to note that the validation set is not utilized in our experiments. Following previous works [3, 4], we adopt the practice of selecting the model that achieves the highest performance on the test set rather than the validation set for final evaluation.
To ensure fair and consistent evaluation, all experiments are conducted in a standardized environment using the NVIDIA A100 GPU. More software library dependencies can be seen on our GitHub website (https://github.com/SCLBD/DeepfakeBench).
Data Augmentation Our benchmark utilizes a series of widely used data augmentation methods for image processing. We describe each augmentation method as follows: