Video Forgery Detection Using Multiple Cues on Fusion of EfficientNet and Swin Transformer

1 INTRODUCTION . Video synthesis techniques based on deep learning Karras et al . ( 2017 ; 2018 ) ; Thies et al . ( 2019 ) have reduced the threshold for people to manipulate videos , and the high-quality deepfake videos that are indistinguishable to the human eye are increasingly available Deepfakes ( 2020 ) ; Faceswap ( 2019 ) ; Fakeapp ( 2020 ) . Although this technology has been applied to Cineflex , Crytek , and driverless technology , which has had a positive effect on society . Inevitably , it is illegally abused by some malicious users to forge indistinguishable false information , which seriously affects the level of public trust in the government and society , causes irreparable social problems , and can even pose a serious threat to politics . Many deepfake detection methods Wang et al . ( 2019 ) ; Kim et al . ( 2021 ) ; Jeon et al . ( 2020 ) have been proposed to address this problem and significant progress has been made . Previous works explore forgery patterns based on manual features Ferrara et al . ( 2012 ) ; Pan et al . ( 2012 ) ; Cozzolino et al . ( 2015 ) and then classify them by zooming in on the subtle differences between the real and forged images . While most of the recent learning-based approaches Cozzolino et al . ( 2017 ) ; Chollet ( 2017a ) define the problem as a binary classification of an image or video with a process of first extracting the global features of the image using a neural network and then feeding them to a classifier to distinguish between genuine and faked images . However , with the increasing refinement of deep generative methods , the generated forged videos are becoming increasingly difficult to discriminate , especially some low-quality ones with only a few subtle visual artifacts . Most existing detection methods still focus on specific artifacts , leading to significant challenges in the generalization and effectiveness of detection systems in realistic scenarios . This creates an urgent need to discover subtle cues that are more critical for detection , and then to extract and incorporate appropriate modules with efficient backbone networks to enhance these artifact features and combine them effectively for forgery detection . After systematic analysis , we find three fundamental reasons for the low generalization of the model . The first is the difficulty of capturing general artifact cues and the limitations of the dataset in terms of quantity and quality . The second is the limitations of selecting a suitable network model for specific feature extraction . The third is the limitations of fully and effectively utilizing the extracted features . In terms of feature cues , we find that although the generated videos are realistic with few artifacts in the texture , the frequency domain , especially at high frequency , still reveals significant differences , even for low-quality videos , and many forgery methods are designed to generate videos that are slightly defective to human vision , with few methods targeting the frequency domain . In addition , it is found that texture information located at the shallow level of the detection network rather than the higher-level semantic information is particularly critical for detection . To eliminate the effect of luminance in RGB on texture feature extraction , we convert the images from RGB to YCbCr and remove the luminance ( Y ) channel , and combine Cb and Cr channels with high frequencies . In addition , we find a characteristic that the optical flow of the real video changes significantly while the optical flow of the deepfake video has little change , and coupled with the temporal consistency of the continuous frames of the real video , we choose the high frequency , texture , and optical flow as the feature cues for video forgery detection . For features such as high frequency and texture , which require attention to local details , we choose EfficientNet-B5 Tan & Le ( 2019 ) , a CNN class network that is superior in local feature extraction , as one of the backbone networks , which has higher accuracy and performance compared with other CNNs . For features such as optical flow , which reflect the inter-frame temporality , we choose Swin Transformer Liu et al . ( 2021 ) , a Transformer class network that is superior in global representation , as another backbone network , which applies to all scales of images , simple and flexible in computation , and shows excellent performance compared with other Transformers . In addition , we introduce the attention mechanism and a new loss function for EfficientNet-B5 to extract more robust face features . In summary , in this paper , we propose a novel multiple-cue video forgery detection model ENST using a combination of high frequency , low-level texture , and optical flow cues , based on the fusion of EfficientNet-B5 and Swin Transformer , which fully integrates multiple key features and uses an advanced backbone network for extraction . To demonstrate the effectiveness of ENST , we conduct extensive experiments on FaceForensics++ Rssler et al . ( 2019 ) and Celeb-DF ( v2 ) Li et al . ( 2019 ) , as well as a comparison of various settings in an ablation study . Comprehensive experiments indicate that ENST has excellent competitive properties . The contributions of this paper are as follows : 1 . We investigate the current video forgery detection methods in-depth and systematically , and find three reasons for the poor generalization ability of these methods , as well as discover several artifact cues in forgery detection , and use these cues to design an efficient network structure for the above reasons . 2 . We propose a novel network structure ENST , which can utilize the local feature extraction ability of EfficientNet-B5 and the global relationship sensing ability of Swin Transformer , and integrate them to complement their abilities during data processing . In addition , we introduce the attention module and a new loss function for EfficientNet-B5 to enhance the ability to extract features . 3 . Our experiments on FaceForensics++ and Celeb-DF ( v2 ) demonstrate that ENST achieves superior classification performance and generalization compared to other state-of-the-art methods . The rest of this paper is organized as follows . Section 2 presents related work in the field of video forgery detection and describes the motivation for the proposed ENST . Section 3 details the specifics of the proposed model . Section 4 describes the experimental results and analysis . Section 5 concludes this paper . 2 RELATED WORK . 2.1 VIDEO FAKE CUES . In recent years , various deep learning forgery methods have produced synthetic images or videos of faces Suwajanakorn et al . ( 2017 ; 2015 ) that have caused some degree of harm to society . The problem of deep forgery detection has received extensive attention in the vision field , and many forgery detection methods have been proposed to detect such forged products . Some methods focus on mining specific artifacts generated by the forgery process , such as color space Li et al . ( 2018 ) ; McCloskey & Albright ( 2018 ) and shape cues . Many deep learning methods Marra et al . ( 2018 ) ; Yu et al . ( 2018 ) ; Zhou et al . ( 2018 ) use deep neural networks to extract high-level semantic information from the spatial domain and then classify a given image or video . Some methods convert the image from the spatial domain to the frequency domain Stuchi et al . ( 2017 ) ; Franzen ( 2018 ) , capturing some information that is valuable for forgery detection . Stuchi et al . ( 2017 ) employ a fixed set of filters to extract different ranges of frequency information and then adopt a fully connected layer to obtain classification results . Durall et al . ( 2019 ) extract the frequency domain information using the DFT transform and average the amplitudes of the different frequency bands . Other methods extract statistical features , such as capturing features from spatial textures Pan et al . ( 2009 ) ; Conotter ( 2011 ) and transform domain coefficient distributions Chen et al . ( 2009 ) ; Lyu & Farid ( 2005 ) . Video detection methods can discover differences in the temporal dimension of video frames Sabir et al . ( 2019 ) ; Guera & Delp ( 2018 ) , such as Wang & Dantcheva ( 2020 ) using spatial and motion information , and Chintha et al . ( 2020 ) ; Amerini et al . ( 2020 ) ; Caldelli et al . ( 2021 ) using optical flow for video forgery detection . While the above methods are limited to specific cues , we take full advantage of these cues . 2.2 DEEPFAKE DETECTION STRUCTURES . Many deep learning models have been proposed to extract features and discover forgery patterns directly without targeting a specific forgery cue or artifact , classify real and fake images or videos , and implement deep forgery detection created by random methods . EfficientNet Tan & Le ( 2019 ) achieves higher accuracy with a larger network by combining a composite model expansion method with neural structure search techniques to optimize the dimensions of network depth , network width , and input image resolution , which has higher accuracy and performance compared to other advanced CNNs . Each layer of Swin Transformer computes self-attention within the window of the transform , enabling cross-window connectivity for all scales of the image . Swin Transformer is flexible and efficient , with only linear complexity for image size . Convolution module in CNN is good at extracting local details , and as the layers get deeper , the attention field will be wider and more suitable for image processing . But to capture global information , detection model often requires stacking many convolutional layers . While attention in Transformer is good at grasping the whole , but requires a large amount of data for training . Conformer Peng et al . ( 2021 ) uses a parallel and complementary approach , combining both CNN and Transformer branches to preserve the features extracted by both branches , to achieve performance beyond CNN and ViT Dosovitskiy et al . ( 2020 ) with comparable parameter complexity . 3 PROPOSED METHOD . In this section , we outline the principles and details of the different parts of the proposed method according to the data processing process . It is challenging for most methods to extract sufficiently comprehensive features with suitable network models and then integrate them effectively for forgery detection . Inspired by Conformer Peng et al . ( 2021 ) , we choose two models EfficientNet-B5 and Swin Transformer with more efficient performance among CNNs and Transformers , as baselines and combine them for video forgery detection . EfficientNet-B5 extracts local features of frames , such as high frequency features and texture features by using the characteristic of focusing well on local features . Swin Transformer extracts artifact features from the optical flow map between frames to obtain the global representation of frames , where the optical flow features responding to the temporal consistency between frames are used . We design a parallel interactive deep forgery detection architecture ENST to learn a potential representation that can distinguish between real and forged faces , and the pipeline of ENST is shown in Figure 1 . 3.1 PRE-PROCESSING . We first extract the frames of the input video , and then use the Multitask Cascaded CNNs ( MTCNN ) Zhang et al . ( 2016 ) to detect and extract the faces present in each frame , crop the faces of each frame , then resize to 224×224 pixels . Moreover , we normalize them to zero mean and unit variance , and finally obtain the extracted faces . After that , we first operate on a single frame in the video , and the face in frame i is input to two branches after the basic feature extraction process .

Summary: The author proposes a method for detecting deepfakes that makes use of EfficientNet B5 and Swin Transformer. Additionally, the authors propose a loss function and an attention mechanism for EfficientNet B5. The proposed methods demonstrate increased accuracy and generalizability when compared to the baselines.