Query: Deepfake Detection

title: Learning to Recognize Patch Wise Consistency for Deepfake Detection
abstract: We propose to detect Deepfake generated by face manipulation based on one of their fundamental features: images are blended by patches from multiple sources, carrying distinct and persistent source features. In particular, we propose a novel representation learning approach for this task, called patch wise consistency learning (PCL) It learns by measuring the consistency of image source features, resulting to representation with good interpretability and robustness to multiple forgery methods. We develop an inconsistency image generator (I2G) to generate training data for PCL and boost its robustness. We evaluate our approach on seven popular Deepfake detection datasets. Our model achieves superior detection accuracy and generalizes well to unseen generation methods. On average, our model outperforms the state of the art in terms of AUC by 2% and 8% in the in and cross dataset evaluation, respectively.

title: Leveraging edges and optical flow on faces for deepfake detection
abstract: Deepfakes can be used maliciously to sway public opinion, defame an individual, or commit fraud. Hence, it is vital for journalists and social media platforms, as well as the general public, to be able to detect deepfakes. Existing deepfake detection methods, while highly accurate on datasets they have been trained on, falter in open world scenarios due to different deepfake generations algorithms, video formats, and compression levels. In this paper, we seek to address this by building on the XceptionNet based deepfake detection technique that utilizes convolutional latent representations with recurrent structures. In particular, we explore how to leverage a combination of visual frames, edge maps, and dense optical flow maps together as inputs to this architecture. We evaluate these techniques using the FaceForensics+ and DFDC mini datasets. We also perform extensive studies to evaluate the robustness of our network against adversarial post processing as well as the generalization capabilities to out of domain datasets and manipulation strategies. Our methods, which we call XceptionNet* achieve 100% accuracy on the popular Face Forensics s+ dataset and set new benchmark standards on the difficult DFDC mini dataset. The XceptionNet* models are shown to exhibit superior performance on cross domain testing and demonstrate surprising resilience to adversarial manipulations.

title: Identifying Invariant Texture Violation for Robust Deepfake Detection
abstract: Existing deepfake detection methods have reported promising in distribution results, by accessing published large scale dataset. However, due to the non smooth synthesis method, the fake samples in this dataset may expose obvious artifacts (e.g. stark visual contrast, non smooth boundary) which were heavily relied on by most of the frame level detection methods above. As these artifacts do not come up in real media forgeries, the above methods can suffer from a large degradation when applied to fake images that close to reality. To improve the robustness for high realism fake data, we propose the Invariant Texture Learning (InTeLe) framework, which only accesses the published dataset with low visual quality. Our method is based on the prior that the microscopic facial texture of the source face is inevitably violated by the texture transferred from the target person, which can hence be regarded as the invariant characterization shared among all fake images. To learn such an invariance for deepfake detection, our InTeLe introduces an auto encoder framework with different decoders for pristine and fake images, which are further appended with a shallow classifier in order to separate out the obvious artifact effect. Equipped with such a separation, the extracted embedding by encoder can capture the texture violation in fake images, followed by the classifier for the final pristine/fake prediction. As a theoretical guarantee, we prove the identifiability of such an invariance texture violation, i.e. to be precisely inferred from observational data. The effectiveness and utility of our method are demonstrated by promising generalization ability from low quality images with obvious artifacts to fake images with high realism.

title: Investigating the Impact of Pre processing and Prediction Aggregation on the DeepFake Detection Task
abstract: Recent advancements in content generation technologies (widely known as DeepFakes) along with the online proliferation of manipulated media and disinformation campaigns render the detection of such manipulations a task of increasing importance. Even though there are many DeepFake detection methods, only few focus on the impact of dataset preprocessing and prediction aggregation on the detection performance of the models. In this paper, we propose a preprocessing step to improve the quality of training corpus for the problem. We also examine its effects on the DeepFake detection performance and investigate the effect of video level prediction aggregation approaches. Experimental results show that the proposed preprocessing approach leads to considerable improvements in the performance of detection models, and the prediction aggregation scheme further boosts the detection efficiency in cases where there are multiple faces in a video.

title: Deepfake detection through PRNU and logistic regression analyses
abstract: Deepfakes, as in creating a video of an impersonated target by replacing the face of an actor with the face of said target, have evolved in recent years to reach convincing levels of perceived authenticity. Generated forged imagery of faces has become so advanced that it is getting more complicated to differentiate a manipulated video from an original by watching it. If Deepfakes can not be told apart from original videos, undesirable effects, such as media distrust or forensic misevaluation, may occur. Therefore, Deepfake detection methods are required to ensure that one can mitigate potentially devastating effects of fake media. In this paper, we consider multiple approaches to distinguish between authentic and Deepfake videos. Among the approaches are Photo Response Non Uniformity (PRNU) visual artifact, and frequency domain analysis, three methods that do not make use of neural nets. We discovered that none of the methods show definite proof of Deepfake presence or absence, and even discovered detection evasion methods. As an alternative, we propose a conceptual system to authenticate original media instead of discovering Deepfake forgeries.  Deepfake, Detection, PRNU, Visual Artifacts, Media Authenticity

title: Identity Driven DeepFake Detection
abstract: DeepFake detection has so far been dominated by `artifact driven' methods and the detection performance significantly degrades when either the type of image artifacts is unknown or the artifacts are simply too hard to find. In this work, we present an alternative approach: Identity Driven DeepFake Detection. Our approach takes as input the suspect image/video as well as the target identity information (a reference image or video) We output a decision on whether the identity in the suspect image/video is the same as the target identity. Our motivation is to prevent the most common and harmful DeepFakes that spread false information of a targeted person. The identity based approach is fundamentally different in that it does not attempt to detect image artifacts. Instead, it focuses on whether the identity in the suspect image/video is true. To facilitate research on identity based detection, we present a new large scale dataset `Vox DeepFake" in which each suspect content is associated with multiple reference images collected from videos of a target identity. We also present a simple identity based detection algorithm called the OuterFace, which may serve as a baseline for further research. Even trained without fake videos, the OuterFace algorithm achieves superior detection accuracy and generalizes well to different DeepFake methods, and is robust with respect to video degradation techniques a performance not achievable with existing detection algorithms.

title: Deepfake Detection with Clustering based Embedding Regularization
abstract: In recent months, AI synthesized face swapping videos referred to as deepfake have become an emerging problem. False video is becoming more and more difficult to distinguish, which brings a series of challenges to social security. Some scholars are devoted to studying how to improve the detection accuracy of deepfake video. At the same time, in order to conduct better research, some datasets for deepfake detection are made. Companies such as Google and Facebook have also spent huge sums of money to produce datasets for deepfake video detection, as well as holding deepfake detection competitions. The continuous advancement of video tampering technology and the improvement of video quality have also brought great challenges to deepfake detection. Some scholars have achieved certain results on existing datasets, while the results on some high quality datasets are not as good as expected. In this paper, we propose new method with clustering based embedding regularization for deepfake detection. We use open source algorithms to generate videos which can simulate distinctive artifacts in the deepfake videos. To improve the local smoothness of the representation space, we integrate a clustering based embedding regularization term into the classification objective, so that the obtained model learns to resist adversarial examples. We evaluate our method on three latest deepfake datasets. Experimental results demonstrate the effectiveness of our method.

title: Deepfake Detection through Deep Learning
abstract: Deepfakes allow for the automatic generation and creation of (fake) video content, e.g. through generative adversarial networks. Deepfake technology is a controversial technology with many wide reaching issues impacting society, e.g. election biasing. Much research has been devoted to developing detection methods to reduce the potential negative impact of deepfakes. Application of neural networks and deep learning is one approach. In this paper, we consider the deepfake detection technologies Xception and MobileNet as two approaches for classification tasks to automatically detect deepfake videos. We utilise training and evaluation datasets from FaceForensics+ comprising four datasets generated using four different and popular deepfake technologies. The results show high accuracy over all datasets with an accuracy varying between 91 98% depending on the deepfake technologies applied. We also developed a voting mechanism that can detect fake videos using the aggregation of all four methods instead of only one.

title: Deepfake detection: humans vs. machines
abstract: Deepfake videos, where a person's face is automatically swapped with a face of someone else, are becoming easier to generate with more realistic results. In response to the threat such manipulations can pose to our trust in video evidence, several large datasets of deepfake videos and many methods to detect them were proposed recently. However, it is still unclear how realistic deepfake videos are for an average person and whether the algorithms are significantly better than humans at detecting them. In this paper, we present a subjective study conducted in a crowdsourcing like scenario, which systematically evaluates how hard it is for humans to see if the video is deepfake or not. For the evaluation, we used 120 different videos (60 deepfakes and 60 originals) manually pre selected from the Facebook deepfake database, which was provided in the Kaggle's Deepfake Detection Challenge 2020. For each video, a simple question: "Is face of the person in the video real of fake?" was answered on average by 19 naive subjects. The results of the subjective evaluation were compared with the performance of two different state of the art deepfake detection methods, based on Xception and EfficientNets (B4 variant) neural networks, which were pre trained on two other large public databases: the Google's subset from FaceForensics+ and the recent Celeb DF dataset. The evaluation demonstrates that while the human perception is very different from the perception of a machine, both successfully but in different ways are fooled by deepfakes. Specifically, algorithms struggle to detect those deepfake videos, which human subjects found to be very easy to spot.

title: A Face Preprocessing Approach for Improved DeepFake Detection
abstract: Recent advancements in content generation technologies (also widely known as DeepFakes) along with the online proliferation of manipulated media and disinformation campaigns render the detection of such manipulations a task of increasing importance. There are numerous works related to DeepFake detection but there is little focus on the impact of dataset preprocessing on the detection accuracy of the models. In this paper, we focus on this aspect of the DeepFake detection task and propose a preprocessing step to improve the quality of training datasets for the problem. We also examine its effects on the DeepFake detection performance. Experimental results demonstrate that the proposed preprocessing approach leads to measurable improvements in the performance of detection models.