Video Forgery Detection Using Multiple Cues on Fusion of EfficientNet and Swin Transformer

1 INTRODUCTION . Video synthesis techniques based on deep learning Karras et al . ( 2017 ; 2018 ) ; Thies et al . ( 2019 ) have reduced the threshold for people to manipulate videos , and the high-quality deepfake videos that are indistinguishable to the human eye are increasingly available Deepfakes ( 2020 ) ; Faceswap ( 2019 ) ; Fakeapp ( 2020 ) . Although this technology has been applied to Cineflex , Crytek , and driverless technology , which has had a positive effect on society . Inevitably , it is illegally abused by some malicious users to forge indistinguishable false information , which seriously affects the level of public trust in the government and society , causes irreparable social problems , and can even pose a serious threat to politics . Many deepfake detection methods Wang et al . ( 2019 ) ; Kim et al . ( 2021 ) ; Jeon et al . ( 2020 ) have been proposed to address this problem and significant progress has been made . Previous works explore forgery patterns based on manual features Ferrara et al . ( 2012 ) ; Pan et al . ( 2012 ) ; Cozzolino et al . ( 2015 ) and then classify them by zooming in on the subtle differences between the real and forged images . While most of the recent learning-based approaches Cozzolino et al . ( 2017 ) ; Chollet ( 2017a ) define the problem as a binary classification of an image or video with a process of first extracting the global features of the image using a neural network and then feeding them to a classifier to distinguish between genuine and faked images . However , with the increasing refinement of deep generative methods , the generated forged videos are becoming increasingly difficult to discriminate , especially some low-quality ones with only a few subtle visual artifacts . Most existing detection methods still focus on specific artifacts , leading to significant challenges in the generalization and effectiveness of detection systems in realistic scenarios . This creates an urgent need to discover subtle cues that are more critical for detection , and then to extract and incorporate appropriate modules with efficient backbone networks to enhance these artifact features and combine them effectively for forgery detection . After systematic analysis , we find three fundamental reasons for the low generalization of the model . The first is the difficulty of capturing general artifact cues and the limitations of the dataset in terms of quantity and quality . The second is the limitations of selecting a suitable network model for specific feature extraction . The third is the limitations of fully and effectively utilizing the extracted features . In terms of feature cues , we find that although the generated videos are realistic with few artifacts in the texture , the frequency domain , especially at high frequency , still reveals significant differences , even for low-quality videos , and many forgery methods are designed to generate videos that are slightly defective to human vision , with few methods targeting the frequency domain . In addition , it is found that texture information located at the shallow level of the detection network rather than the higher-level semantic information is particularly critical for detection . To eliminate the effect of luminance in RGB on texture feature extraction , we convert the images from RGB to YCbCr and remove the luminance ( Y ) channel , and combine Cb and Cr channels with high frequencies . In addition , we find a characteristic that the optical flow of the real video changes significantly while the optical flow of the deepfake video has little change , and coupled with the temporal consistency of the continuous frames of the real video , we choose the high frequency , texture , and optical flow as the feature cues for video forgery detection . For features such as high frequency and texture , which require attention to local details , we choose EfficientNet-B5 Tan & Le ( 2019 ) , a CNN class network that is superior in local feature extraction , as one of the backbone networks , which has higher accuracy and performance compared with other CNNs . For features such as optical flow , which reflect the inter-frame temporality , we choose Swin Transformer Liu et al . ( 2021 ) , a Transformer class network that is superior in global representation , as another backbone network , which applies to all scales of images , simple and flexible in computation , and shows excellent performance compared with other Transformers . In addition , we introduce the attention mechanism and a new loss function for EfficientNet-B5 to extract more robust face features . In summary , in this paper , we propose a novel multiple-cue video forgery detection model ENST using a combination of high frequency , low-level texture , and optical flow cues , based on the fusion of EfficientNet-B5 and Swin Transformer , which fully integrates multiple key features and uses an advanced backbone network for extraction . To demonstrate the effectiveness of ENST , we conduct extensive experiments on FaceForensics++ Rssler et al . ( 2019 ) and Celeb-DF ( v2 ) Li et al . ( 2019 ) , as well as a comparison of various settings in an ablation study . Comprehensive experiments indicate that ENST has excellent competitive properties . The contributions of this paper are as follows : 1 . We investigate the current video forgery detection methods in-depth and systematically , and find three reasons for the poor generalization ability of these methods , as well as discover several artifact cues in forgery detection , and use these cues to design an efficient network structure for the above reasons . 2 . We propose a novel network structure ENST , which can utilize the local feature extraction ability of EfficientNet-B5 and the global relationship sensing ability of Swin Transformer , and integrate them to complement their abilities during data processing . In addition , we introduce the attention module and a new loss function for EfficientNet-B5 to enhance the ability to extract features . 3 . Our experiments on FaceForensics++ and Celeb-DF ( v2 ) demonstrate that ENST achieves superior classification performance and generalization compared to other state-of-the-art methods . The rest of this paper is organized as follows . Section 2 presents related work in the field of video forgery detection and describes the motivation for the proposed ENST . Section 3 details the specifics of the proposed model . Section 4 describes the experimental results and analysis . Section 5 concludes this paper . 2 RELATED WORK . 2.1 VIDEO FAKE CUES . In recent years , various deep learning forgery methods have produced synthetic images or videos of faces Suwajanakorn et al . ( 2017 ; 2015 ) that have caused some degree of harm to society . The problem of deep forgery detection has received extensive attention in the vision field , and many forgery detection methods have been proposed to detect such forged products . Some methods focus on mining specific artifacts generated by the forgery process , such as color space Li et al . ( 2018 ) ; McCloskey & Albright ( 2018 ) and shape cues . Many deep learning methods Marra et al . ( 2018 ) ; Yu et al . ( 2018 ) ; Zhou et al . ( 2018 ) use deep neural networks to extract high-level semantic information from the spatial domain and then classify a given image or video . Some methods convert the image from the spatial domain to the frequency domain Stuchi et al . ( 2017 ) ; Franzen ( 2018 ) , capturing some information that is valuable for forgery detection . Stuchi et al . ( 2017 ) employ a fixed set of filters to extract different ranges of frequency information and then adopt a fully connected layer to obtain classification results . Durall et al . ( 2019 ) extract the frequency domain information using the DFT transform and average the amplitudes of the different frequency bands . Other methods extract statistical features , such as capturing features from spatial textures Pan et al . ( 2009 ) ; Conotter ( 2011 ) and transform domain coefficient distributions Chen et al . ( 2009 ) ; Lyu & Farid ( 2005 ) . Video detection methods can discover differences in the temporal dimension of video frames Sabir et al . ( 2019 ) ; Guera & Delp ( 2018 ) , such as Wang & Dantcheva ( 2020 ) using spatial and motion information , and Chintha et al . ( 2020 ) ; Amerini et al . ( 2020 ) ; Caldelli et al . ( 2021 ) using optical flow for video forgery detection . While the above methods are limited to specific cues , we take full advantage of these cues . 2.2 DEEPFAKE DETECTION STRUCTURES . Many deep learning models have been proposed to extract features and discover forgery patterns directly without targeting a specific forgery cue or artifact , classify real and fake images or videos , and implement deep forgery detection created by random methods . EfficientNet Tan & Le ( 2019 ) achieves higher accuracy with a larger network by combining a composite model expansion method with neural structure search techniques to optimize the dimensions of network depth , network width , and input image resolution , which has higher accuracy and performance compared to other advanced CNNs . Each layer of Swin Transformer computes self-attention within the window of the transform , enabling cross-window connectivity for all scales of the image . Swin Transformer is flexible and efficient , with only linear complexity for image size . Convolution module in CNN is good at extracting local details , and as the layers get deeper , the attention field will be wider and more suitable for image processing . But to capture global information , detection model often requires stacking many convolutional layers . While attention in Transformer is good at grasping the whole , but requires a large amount of data for training . Conformer Peng et al . ( 2021 ) uses a parallel and complementary approach , combining both CNN and Transformer branches to preserve the features extracted by both branches , to achieve performance beyond CNN and ViT Dosovitskiy et al . ( 2020 ) with comparable parameter complexity . 3 PROPOSED METHOD . In this section , we outline the principles and details of the different parts of the proposed method according to the data processing process . It is challenging for most methods to extract sufficiently comprehensive features with suitable network models and then integrate them effectively for forgery detection . Inspired by Conformer Peng et al . ( 2021 ) , we choose two models EfficientNet-B5 and Swin Transformer with more efficient performance among CNNs and Transformers , as baselines and combine them for video forgery detection . EfficientNet-B5 extracts local features of frames , such as high frequency features and texture features by using the characteristic of focusing well on local features . Swin Transformer extracts artifact features from the optical flow map between frames to obtain the global representation of frames , where the optical flow features responding to the temporal consistency between frames are used . We design a parallel interactive deep forgery detection architecture ENST to learn a potential representation that can distinguish between real and forged faces , and the pipeline of ENST is shown in Figure 1 . 3.1 PRE-PROCESSING . We first extract the frames of the input video , and then use the Multitask Cascaded CNNs ( MTCNN ) Zhang et al . ( 2016 ) to detect and extract the faces present in each frame , crop the faces of each frame , then resize to 224×224 pixels . Moreover , we normalize them to zero mean and unit variance , and finally obtain the extracted faces . After that , we first operate on a single frame in the video , and the face in frame i is input to two branches after the basic feature extraction process .

Summary: This paper faces the problem of video deepfake detection. The idea is to rely on high frequency, texture and optical flow cues to gain generalization. To this end it is proposed a dual-branch detection approach: low-level features are extracted by EfficientNet-B5 that also includes an attention mechanism, while temporal inconsistencies are captured by Swin Transformer. Experiments are carried out on FaceForensics++ and Celeb-DF (v2) and show better generalization ability with respect to state-of-the-art.