dataesr/scientific-paragraphs-categorization huggingface dataset:
DeepFake Detection Approaches Given the growing threat of tampered media to society, a lot of methods have been proposed for DeepFake detection. One of the earliest works in the field is MesoNet  [1] , where a relatively shallow Convolutional Neural Network (CNN) with five layers was proposed. In their landmark work  [61] , the researchers benchmarked the performance of several state-of-the-art CNNs on their proposed novel FaceForensics++ dataset, showing that an XceptionNet network  [7]  outperformed the competition. Research since then has evolved by combining CNNs with other architectures such as Recurrent Neural Networks (RNNs)  [19] , Long Short-Term Memories (LSTMs)  [36, 47]  or Attention heads  [14, 30, 73, 80, 81, 84]. In  [3]  the authors propose an ensemble of numerous CNN classifiers, based on the popular EfficientNet network  [66]  in tandem with attention mechanisms and Siamese training with the goal of accurately detecting DeepFakes.
DeepFake Detection Services Seeing the threat that DeepFakes pose on society, several companies and academics have developed DeepFake detection web services. DeepWare 6  developed an online DeepFake scanner as well as an Android application for the identification of DeepFake videos. Their approach uses an EfficientNet-B7  [66]  pre-trained on ImageNet  [11]  and fine-tuned on the DFDC dataset  [12]  that operates at frame level. Since the dataset is imbalanced, containing approximately 20K real and 100K fake videos, they balanced it at training time by randomly selecting equal number of real or fake videos. DuckDuckGoose 7  has created the DeepDetector, which is a Deep-Fake detection system, as well as a browser detector plugin named DeepfakeProof. Additionally, they have created the so-called Replicant DeepFake creation system that can be used to test the reliability of biometric authentication systems.
Poisoning attacks  [2]  attempt to alter the dataset used to train an AI model. This type of attack occurs prior to the deployment of the AI system. Extraction attacks  [23]  operate on a different dimension than previous attacks. These aim at stealing the underlying parameters of AI models and thus reproducing the same model at very little cost compared to the one invested for development. Inference attacks  [8]  finally consist in identifying the characteristics of specific samples that were used to train an AI model. This can be particularly problematic when personal information was used to train a system, which could be breached and damage individuals' privacy. A noteworthy publication is  [17]  where the authors evaluate the robustness of DeepFake detectors against multiple DeepFake attacks and subsequently experiment with defense methodologies against them.
Processing Pipeline Once the service receives the link of an image or video as input by the user, the following processing pipeline takes place. Download Media: The image/video at the URL is identified and downloaded by our custom download module that supports popular file sharing services such as Dropbox  9  and Google Drive 10  , as well as social media platforms like YouTube 11  and Twitter  12. Media Type: If the downloaded resource is an image, then only the Face Detection and Inference steps that are described below are applied to get the final results. Therefore, the following steps are described below as if the resource is a video. Video Segmentation: During this step, a video similarity network is used in order to segment the video in multiple shots. We follow the feature extraction and similarity calculation process described in  [35].
3. 2. 2 Training process. Models 1-4 were trained on the Facebook DeepFake Detection Challenge (DFDC) dataset  [12]  while Model 5 was trained on the WildDeepFake (WDF) dataset  [84]. For the former models, we used the Adam optimizer  [33]  with a learning rate of 10 -4 and 32 batch size for 25 epochs respectively. The networks are initialized with pre-trained weights on ImageNet-1k  [11]. For the latter model, we trained an EfficientV2-M-in21k pre-trained on ImageNet-21k  [60]  and fine-tuned using the Adam optimizer with 10 -4 learning rate and 32 batch size for 2 epochs. Furthermore, during training, we employ the following augmentations using the Albumentations library  [4] : Geometric augmentations (Rotate, HorizontalFlip), Color augmentations (ColorJitter, ToGray), Blurring (MotionBlur, GaussianBlur), Image Corruption (ISONoise, Coarse-Dropout), External Effects (RandomSunFlare, RandomRain).
We used FastAPI  [71]  to implement the controller's REST API, and Python-RQ  [13]  to dispatch and monitor jobs asynchronously. Each job is implemented as a blocking HTTP call to the core service, which provides a synchronous REST API and orchestrates the necessary computations. First, it dispatches a request to the media download service that is implemented on top of Youtube-dlp  [78]. Once the video is successfully downloaded, we use OpenCV  [54]  to load it and extract frames. The extracted frames are then fed to a segmentation model running on a Triton inference server  [53]. Triton is an open-source optimized inference server for executing deep learning models on CPU or GPU. GPU memory management, batching, and model versioning are seamlessly handled. To load our models on Triton, we use the torchscript serialization of PyTorch  [57].
Model Card We have documented our DeepFake detection service using less formal language in a Model Card format  13. The model card includes a description of our service's intended use, an account of caveats and recommendations that potential novice users should take into consideration when interpreting the service results, as well as a performance evaluation over three datasets accompanied with a clear explanation of the reported metrics. The compiled model card has been reviewed by AI experts from different disciplines. Based on these reviews, the current version is intended for experts having technical experience, i. e. , other researchers working on the problem of DeepFake detection or media verification companies/organizations/groups. Yet, there is room for improvement for other non-technical audiences.
Evaluation settings 4. 1. 1 Datasets. We employ three evaluation datasets to assess the performance of our DeepFake Detection service: • FaceForensics++ (FF++)  [61]  This is organized in two manipulation categories, Identity Swap, implemented based on FaceSwap and DeepFakes, and Expression Swap, implemented using NeuralTextures and Face2Face. FF++ contains 1000 real videos and 4000 fake videos derived by applying the four models on each real video. Evaluation on FF++ provides a performance indicator on different manipulation categories and methods. Compared to more recent datasets (e. g. CelebDF, DFDC) the DeepFake quality in FF++ is visibly worse. • CelebDF-V2 (CelebDF)  [42]  This comprises videos from celebrity interviews that have been manipulated using improved versions of the DeepFake manipulation methods used in FF++.
It consists of 590 real and 5639 fake videos. • WildDeepFake (WDF)  [84]  In contrast to the above datasets where manipulations were generated by the dataset creators, this contains real-world DeepFakes sourced from various video-sharing websites and their corresponding real versions. It consists of 3800 real and 3500 fake videos. Due to its real-world nature, it is considered a challenging dataset. one class or alter the datasets via sampling. Hence, we choose to report the Balanced Accuracy (BA) rather than raw Accuracy. BA is defined as the mean of the recall computed on each class. Its possible values are in the range 0%-100% (higher is better). Moreover, we report the Area Under the Curve (AUC) as it is the most often used metric in the literature. It is defined as the area under the Receiver Operating Characteristic (ROC) curve with possible values ranging from 0 to 1 (higher is better).
Adversarial robustness. To set up the adversarial robustness evaluation, we used IBM's Adversarial Robustness Toolbox 14  (ART). More specifically, we used ART's PyTorchClassifier class to wrap our model ensemble and subsequently used the Projected Gradient Descent adversarial attack  [44]  attack class. Regarding its hyperparameters, we use  = 0. 2 and _ = 5, due to computational constraints. To benchmark our model, for each video in each dataset, (i) we feed it to the PyTorchClassifier unaltered, (ii) we generate an adversarial example on a frame-by-frame basis, and (iii) feed it through our classifier again. 1  presents our evaluation results on the three aforementioned datasets. Our system performs much better on the CelebDF and WDF datasets rather than the FF++, in which we observe an average 13% performance drop in terms of BA.
Specifically, on the WDF dataset, we achieve an 84. 94% BA which is close to other state-of-the-art methods  [84]  and is expected since our ensemble's fifth model was trained on this dataset (see Section 3. 2). In the case of the CelebDF dataset, even though none of our models have been trained with this dataset, we achieve an 82. 75% BA. Additionally, we compare our system with the publicly available model by DeepWare  15. We follow the same settings as used for our models for pre-processing, which are slightly different from those used by the original authors. The two systems perform comparably in FF++ and CelebDF, with our service having a small but clear edge. Our system significantly outperforms its competitor in WDF, which is expected since it has been used for training.
Evaluation on different manipulations. To delve into the performance discrepancy between FF++ and the other two datasets, we performed a more extensive evaluation on the FF++ dataset in terms of manipulation type; this is presented in Table  2   3  illustrates the service performance in terms of its robustness to adversarial attacks with the PGD attack with three different output normalization settings. Even though PGD is a white-box attack, meaning that the attacker would need access to the weights of all the ensemble models, we maintain that in the spirit of Reliable AI, it is preferable to consider such a worst-case scenario. All attacks try to fool the detector into assessing that the input media are real. The norm-1 attack does not have any noticeable effect on the performance in comparison to the original performance from Table  1.
Availability We have chosen to keep the code and the model's weights private and to require user credentials for granting access to our UI. This is due to two main reasons. First, we argue that since our models are vulnerable to adversarial attacks -as shown in Section 4. 2. 3 -it is essential to protect the service from white-box attacks that would be very easy if the internals were public. We believe that this action will deter the large majority of malicious actors from performing white-box adversarial attacks. Second, given the scarcity of publicly accessible DeepFake detection services, we lack the computational resources to handle the potentially high traffic from end users, and other complications including Denial of Service (DoS) attacks. For the above reasons, we grant access to the service and UI only to trusted partners upon request.
CONCLUSION In this work, we introduced the MeVer DeepFake detection service, a complex multi-model system that detects DeepFake videos and images. We discussed the overall processing pipeline, including a number of pre-processing steps. Also, we presented the model architectures and training processes for the deployed models as well as implementation details for the service. The service has been evaluated on three well-known datasets: FaceForensics++, Celeb-DF, and WildDeepFake. For Celeb-DF and WildDeepFake, our service performed robustly and better compared to the publicly available DeepWare model. For FF++, evaluation by manipulation type revealed that our service performed robustly only in Identity Swap manipulations. In the spirit of Trustworthy AI, we also performed an Adversarial Robustness evaluation, and we provided a model card for the service.
From the results of the adversarial evaluation, we observe a vulnerability to the Projected Gradient Descent attack, which opens new directions for future research. Last but not least, we discuss at length the practical challenges we faced moving from a research codebase to a real-world system that would need to be robust to arbitrary media content from the Internet. In the future, we plan to improve the detection accuracy by continuously employing the most recent advancements in the field, i. e. , by using better datasets for training and evaluation and using state-of-the-art model architectures. In addition, we plan to experiment with various promising Deep Learning architectures and training techniques in order to keep up with the ever-increasing visual quality of DeepFakes. Also, we plan to enhance our service 19 https://www.
[46]n three datasets attacked with the PGD adversarial attack with three output normalization setting. The former two belong to the Identity Swap manipulation category while the latter two are examples of Expression Swapping[46]. It can be argued that this is due to our training data lacking Expression Swapping examples; therefore, we expect our service to perform better on Identity Swap manipulations. 4. 2. 3 Adversarial robustness. Table.
ACKNOWLEDGEMENTS This work has been supported by the  AI4Media H2020  project, partially funded by the  European Commission  under contract number  951911 , and the  US Paris Tech Challenge  award, which was funded by the  US Department of State Global Engagement Center  under contract number  SGECPD18CA0024. Also, we would like to thank  Birgit Gray  from  DW  and  Beat Buesser  from  IBM  for their valuable feedback and support.
For example, this is the case for most datasets  [10] [11] [12]. From time to time, thermal cameras are used as complementary sensors to video or lidar devices, but they have almost never been used alone during degraded conditions. Regarding the combined use of thermal images and object detection/identification software, some works in the literature present fusions between RGB and thermal representations to improve detection in a Faster-RCNN model  [5]. Works that show the implementation of the YOLO model and thermal images for the detection of objects under different adverse meteorological conditions also exist  [3, 4] , although they lack formality regarding the relationship between the detection rate and the meteorological conditions that exist. Unlike what is presented in the literature, our proposal presents a detailed analysis of the response of the detection software (in this case the YOLO model) concerning the intensity of the fog, the angle of view of the camera, and the distance between the camera and the objects.
This unique proprietary feature is responsible for the camera's high image quality, eliminating the need for moving parts and resulting in cost reduction, extended camera life, smaller camera size, and lower power consumption. About this, Table  1  summarizes the technical specifications of ADASKY's camera. As a side note-shuttered cameras lose on average 0. 5 s of data every 2-3 min for calibration. This implies that the vehicle will "lose its sight" for about 11 m while moving at a speed of 80 KM/H. Moreover, shuttered cameras exhibit image quality degradation between the shutter cycles, whereas ADASKY's camera maintains high image quality continuously. ADASKY's Video Out streams can be broadcast simultaneously in a 14-bit grayscale optimized for computer vision and an 8-bit grayscale optimized for viewing.
Unlike the thermal targets, the temperatures of the pedestrian and the car were not uniform, making it impossible to execute the same analysis carried out for the static tests. To overcome this, object recognition software was used for the assessment of the observability of the pedestrian and the car. For this, the famous YOLO software was used  [9, 26] -deep learningbased software for object detection and recognition, including pedestrians and cars. To obtain a result that was independent of the recognition method, it was decided to use the YOLOv5s pre-trained model without any modification, only pre-trained by the COCO dataset  [27]. Thus, without model training, the recognition rate depends mostly on the contrast level in the images, resulting in the use of the same indicator as in the static scenarios.
• Use case (a)-Reference data. Videos recorded without the presence of fog (MOR above 1000 m); • Use case (b)-Data from foggy weather. Videos recorded with a MOR (continuous variable) varied from an initial value of 10 m (denser fog) to 160 m (dissipated fog). Figure  8  shows an example of each of these two groups. On the left side there is an image with visibility of 10 m, and the image in the middle corresponds to visibility of 160 m; both images come from a video belonging to the use case (b). The image on the right corresponds to the reference data (use case (a)), where the MOR is around 1000 m. As this study was aimed at analyzing the capabilities of the thermal cameras under foggy weather conditions, most of the analysis was oriented toward the images from the use case (b). Thus, unless otherwise indicated, from now on, the presented results belong to the videos from use case (b).
For visualization purposes, these data were squeezed to 8-bit format, since this is the pixel range that is supported by all display screens. There are numerous techniques to compress a 14-bit histogram to an 8-bit for display purposes. The transformation technique from 14 to 8 bits used by ADASKY is a unique IP, targeting the best experience for the human observer. It is important to emphasize that there is an inevitable loss of information when compressing 14 bits to 8 bits. Hence, the observability results are driven by the ability to distill the relevant details when compressing the data from 14-to 8-bit format and can be improved by specific adjustments to the desired use case, which was not done during this campaign. This means that the results might vary based on the compression method.
Then, assuming that objects and the background have different pixel values, easy differentiation of these two elements can be performed. These two indicators, intensity and contrast, are intimately linked and are both needed to analyze the content of grayscale images to locate objects in them. For the analysis of data collected from static scenarios, the computation of these two indicators was carried out as follows: 1. The mean intensity in a target is the mean of the intensities (i. e. , the values) of the pixels that belong to a target.
Contrast Attenuation Due to Foggy Conditions A study of attenuation in contrast caused by MOR is presented next. This parameter quantifies the impact of MOR degradation on the target's appreciation. Attenuation was calculated per target, distance, AOV, and MOR, and it was estimated as follows: Attenuation in interval A = NGC (median) in interval A NGC reference * 100 -100 (2) Here, interval A denotes a MOR interval (e. g. , MOR from 10 to 15 m), and NGC reference is the median in contrast obtained from reference data (i. e. , images taken without the presence of fog). Thus, when referring to the attenuation in the interval of 10 to 15 m, we refer to the difference (in percentage) between the contrast (median) in images taken by an MOR between 10 and 15 m and the contrast (median) in images taken from the reference data.
Note that a negative attenuation value indicates loss of contrast (and therefore, loss of target observability) with respect to the reference images. On the contrary, a positive attenuation value indicates that the contrast between the target and the background is greater than in the reference images. The results of this analysis are presented in Figure  19. On the one hand, the left image shows the percentage of attenuation for the full dataset. There, we can observe a median in attenuation below 10% when MOR is greater than 20 m, between 10% and 20% when MOR is between 16 and 20 m, and above 50% when MOR is between 10 and 15 m. An ANOVA confirms that the effect of MOR is statistically significant (F(5, 378) = 10. 90, p < 0. 001). The right side of Figure  19  shows the percentage of attenuation grouped by targets (temperature).
Analysis of Dynamic Scenarios From the static analysis, the operability limits for the four different cameras (AOV 18°, 30°, 60°, and 90°) were obtained. These limits of operability represent the meteorological optical ranges up to which a negligible impact on contrast is ensured. Below these limits, the NGC drops drastically. Now, in the dynamic analysis, we seek to expand these results by indicating not only from which value of MOR the contrast is affected, but also from which MOR the objects stop being "detected" by object recognition software. For this, we have decided to use YOLO (You Only Look Once) software, a popular family of object recognition models  [9]. As the YOLOv5s pre-trained model is used without any extra training, object recognition is directly linked to the NGC. Thus, the results given in this section are general and independent of the recognition method used.
Validation by Human Observers As a first phase of the dynamic data analysis, a series of visual validations of the videos from the dynamic scenarios were carried out. For this, five people visually verified the videos from protocol (i) (described in Section 3. 2) recorded by 18°and 30°cameras, in which a car and a pedestrian were involved. All videos (from 18°and 30°cameras) were distributed among the five volunteers, so each video was verified by a single person. During this process, observers registered the moment at which they saw or no longer saw the objects in the scenario (car/pedestrian). The reason why only the videos from the 18°a nd 30°cameras were analyzed is because those narrow AOVs can be easier interpreted by a human observer due to higher resolution of the objects in the scene.
Validation Using YOLO For the analysis of the dynamic scenarios (including targets in motion (protocol (ii) described in Section 3. 2 or motionless (protocol (i)) using YOLO software, the following procedure was carried out: 1. Each of the dynamic images sequences was passed through YOLOv5s software using the "yolov5s" model, 2. The outputs from YOLOv5s were: (1) the same image with the bounding boxes that delimit the found/detected objects (see Figure  23 ), and (2) a semantic classification and a confidence rate corresponding to the objects found. For this study, only the first output was considered. This means that we only considered whether the object was detected or not, regardless of its semantic classification or confidence rate.
For the first part, five people visually verified the 8-bit videos in which a car and a pedestrian were involved. During this process, observers registered the moment at which they started or stopped to see any of the targets present in the scenario (car/pedestrian). From this analysis, we concluded that, with a camera-to-target distance between 10 and 25 m, and using the mean (rounded value) as a decisive parameter, the observers were able to distinguish: (1) the pedestrian in the 18°camera when MOR ≥ 15 m; (2) the pedestrian in the 30°camera when MOR ≥ 14 m; (3) the car in the 18°camera when MOR ≥ 15 m; and (4) the car in the 30°camera when MOR ≥ 15 m. Finally, for validation using object recognition software, we decided to use YOLO (You Only Look Once). As the YOLOv5s pre-trained model was used without any extra training, object recognition was directly linked to the NGC.
In the case of the 90°camera, detection rates below the operability limit established were obtained in all the cases. Finally, it is important to emphasize that the performance of YOLO can be improved if the model is trained taking into account the thermal data. An interesting particular case would be the use of the uncompressed 14-bit images for both model training and object detection. This idea is exploited in ADASKY's AI perception layer. However, this product is not part of the AWARD program and thus was not part of this research.