Deep forgery discriminator via image degradation analysis

IET Image ProcessingVolume 15, Issue 11 p. 2478-2493 ORIGINAL RESEARCH PAPEROpen Access Deep forgery discriminator via image degradation analysis Miaomiao Yu, Miaomiao Yu Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorJun Zhang, Corresponding Author Jun Zhang zhangjun1975@nudt.edu.cn Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China Correspondence Zhang Jun and Li Shuohao, Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China. Email: zhangjun1975@nudt.edu.cn; lishuohao@nudt.edu.cnSearch for more papers by this authorShuohao Li, Corresponding Author Shuohao Li lishuohao@nudt.edu.cn orcid.org/0000-0003-4958-8573 Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China Correspondence Zhang Jun and Li Shuohao, Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China. Email: zhangjun1975@nudt.edu.cn; lishuohao@nudt.edu.cnSearch for more papers by this authorJun Lei, Jun Lei Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorFenglei Wang, Fenglei Wang Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorHao Zhou, Hao Zhou Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this author Miaomiao Yu, Miaomiao Yu Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorJun Zhang, Corresponding Author Jun Zhang zhangjun1975@nudt.edu.cn Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China Correspondence Zhang Jun and Li Shuohao, Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China. Email: zhangjun1975@nudt.edu.cn; lishuohao@nudt.edu.cnSearch for more papers by this authorShuohao Li, Corresponding Author Shuohao Li lishuohao@nudt.edu.cn orcid.org/0000-0003-4958-8573 Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China Correspondence Zhang Jun and Li Shuohao, Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China. Email: zhangjun1975@nudt.edu.cn; lishuohao@nudt.edu.cnSearch for more papers by this authorJun Lei, Jun Lei Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorFenglei Wang, Fenglei Wang Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorHao Zhou, Hao Zhou Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, ChinaSearch for more papers by this author First published: 07 May 2021 https://doi.org/10.1049/ipr2.12234AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Generative adversarial network-based deep generative model is widely applied in creating hyper-realistic face-swapping images and videos. However, its malicious use has posed a great threat to online contents, thus making detecting the authenticity of images and videos a tricky task. Most of the existing detection methods are only suitable for one type of forgery and only work for low-quality tampered images, restricting their applications. This paper concerns the construction of a novel discriminator with better comprehensive capabilities. Through analysis of the visual characteristics of manipulated images from the perspective of image quality, it is revealed that the synthesized face does have different degrees of quality degradation compared to the source content. Therefore, several kinds of image quality-related handicraft features are extracted, including texture, sharpness, frequency domain features, and deep features, to unveil the inconsistent information and modification traces in the fake faces. In this way, a 1065-dimensional vector of each image is obtained through multi-feature fusion, and it is then fed into RF to train a targeted binary classification detector. Extensive experiments have shown that the proposed scheme is superior to the previous methods in recognition accuracy on multiple manipulation databases including the Celeb-DF database with better visual quality. 1 INTRODUCTION It is known that DeepFake technology is developing rapidly, in which deep learning models are used to generate and manipulate images, videos, or audio contents. The technology has attracted widespread public attention [1-3]. A non-professional user without any deep learning knowledge background can easily swap the face of one person for the one of another person in a video just by utilizing easily available tools, such as ZAO, FaceApp and FakeApp. DeepFake has many application scenarios, such as video games, film and television program production. In addition to these entertainment-related scenarios, there are huge risks behind such technology. Its malicious used may bring serious consequences [4]. Particularly in business and politics, a tampered image, video or audio may damage a company's long-established brand image, destroy the reputation of celebrities [5], or even cause political conflicts. In April 2018, a video about former US President Barack Obama mocking Trump in a critical tone received more than two million views on Twitter, but the video was actually tampered by a public statement made by a director in the United States, as shown in Figure 1. Katie Jones, a network character created through AI, is active in the political and business circles of the United States with a very convincing fictional image. Actually, it is just one of a large number of "spies" lurking on the social networking site LinkedIn. In the US election in November 2020, fake videos and news manipulated by DeepFake were used to confuse voters in various ways, which had a serious impact on the credibility of the news media. FIGURE 1Open in figure viewerPowerPoint An example of DeepFake The widespread dissemination of such forged contents truly breaks the conventional perception of "Seeing is believing." For face forgery, in order to reduce the various risks and challenges that DeepFake may bring about to individuals, societies and the country, it is vital to develop a deep forgery discriminator that can effectively distinguish real images from fake images. According to our findings, most of the existing detection methods [5-8] are mainly targeted at masking forgery, and a small part of researches is adapted to unmasking DeepFake [9-11], while few methods are suitable for both types at the same time. Thus, the application range of these single-type forgery detection models is greatly restricted. What is more, most models [4, 12–14] achieve high accuracy on rough and low-quality (LQ) forged databases, while the detection of high-quality (HQ) forged contents becomes particularly difficult. The target of this research is to build a deep forgery discriminator with better comprehensive performance. With the continuous optimization of computer performance and the development of some fields such as computer vision and deep learning, deep generative models based on deep neural networks (DNN) are widely used to produce fake images and videos consistent with the distribution of real data. However, since the encoder used to generate the fake face is not familiar with the skin texture and scene area other than the face-changing region in the source image, and the size and resolution of the generated synthetic face are very limited, visually visible boundary artefacts would appear when the face-changing area is merged with the rest of the area. In order to match the fake face with the surrounding background, further blur and smoothing is necessary. The positions of the bounding boxes over frames are smoothed by blurring the colour intensity of the pixels near the border to eliminate jitter generated by swapped face, resulting in the lack of texture details of the skin in the swapped region, as shown in Figure 2. In addition to changes made to the skin, blurring and smoothing can also change the presentation of facial organs, as shown in Figure 3. Additionally, in the process of masking DeepFake, the source face is covered with a mask. If the position and shape of the mask are not accurately estimated, artefacts visible to the naked eye can be exposed along the border of the mask, such as eyebrows and face contours, as shown in Figure 4. FIGURE 2Open in figure viewerPowerPoint Examples of face manipulation on the DeepFake Detection (DFD) dataset [15]. (a) is the real face, (b) and (c) are fake ones, where the face-changing area is blurred, so the detail information is lost, and the texture is changed FIGURE 3Open in figure viewerPowerPoint Examples of face manipulation on the Celeb-DF database [16]. (a) is the real face, (b) and (c) are fake ones, where the texture structure of the teeth is lost FIGURE 4Open in figure viewerPowerPoint Examples of face manipulation on the DFD dataset From the above observations, it is clear that the synthesized faces do have different degrees of quality degradation compared to the source contents. The detection of synthetic faces can be analyzed by the following two factors: (1) the appearances of visual artefacts; (2) the changes of texture structure. Here, a deep forgery discriminator via image degradation analysis is presented, capable of effectively capturing the inconsistent information generated by forgery process by extracting multiple features related to image quality, thus realizing the distinction between the original face and the manipulated face. In summary, the major contributions of this paper are as follows: Combining the concepts of DeepFake detection with image quality analysis, the visual characteristics of the synthetic face are analyzed from the aspect of image quality degradation. The well-built detection model can not only effectively identify low-quality forged images, but also achieve higher detection accuracy for images with high resolution and better visual quality. Meanwhile, an alternative is proposed that can effectively reduce the computational complexity. A novel texture features descriptor is proposed and improved on the basis of the classic local binary pattern (LBP) operator. The local structure map is constructed and adopted as a weighting factor to represent the local texture information and highlight the structural loss in the forged image when counting histogram features, so as to expose the changes in texture distribution that may occur during the forgery process. The strategy of multi-feature fusion is adopted to combine the four handpicked and representative features extracted from spatial and frequency domains. The experimental results indicate that the combined feature vector is valid enough to maximize the representation of the facial image. The established model is not only suitable for detecting the entire face synthesis forgery, but also for the identity swap manipulation. To the best of our knowledge, this is the first method designed for both unmasking and masking DeepFake detection tasks at the same time. The rest of this paper is organized as follows: Section 2 briefly summarizes some typical works most relevant to the research of this article, including the generation and detection methods of forged images or videos; Section 3 describes in detail the establishment process of our proposed deep forgery discriminator; Section 4 demonstrates the validity and reliability of the proposed scheme through extensive experiments. The last Section is the conclusion and outlook. 2 RELATED WORKS 2.1 Face forgery methods Recently, the excellent ability of deep generative models, such as generative adversarial networks (GAN) [17] and variational autoencoders (VAE) [18], are widely used to produce forged faces with realistic appearance. GAN and its variants are one of the most important achievements in the field of machine learning in the past 10 years. They consist of two neural networks: a generator and a discriminator. The generator is to fool the discriminator by learning the data distribution and producing seemingly convincing fake samples, while the discriminator attempts to distinguish between the real samples and the fake samples generated by the generator. The learning processes of these two networks are adversarial. Similar to the GAN generator, the VAE is composed of encoder and decoder. The encoder is mainly used to reduce the dimensionality of the input layer to obtain the compressed representation of the image, whereas the decoder serves to create a new output closely resembling the original input. Figure 5 depicts DeepFake generation pipeline based on the combination of VAE and GAN. It is worth mentioning that the reason why GAN generator is superior to VAE network is that GAN discriminator rejects some bad samples. FIGURE 5Open in figure viewerPowerPoint DeepFake generation pipeline via VAE-GAN Face forgery mainly embraces four types: entire face synthesis, identity swap, attribute manipulation, and expression swap. The first one applies GAN-based architecture [19] to create high-definition faces that do not exist in the real world. Identity swap is the most popular face tampering technology, but unlike the entire face synthesis, it replaces the face of the subject in the source video with the face of the target subject. Typically, the Deepfake-AE [20] is good at generating facial images with more realistic visual quality on the basis of autoencoder architecture with a shared encoder and two decoders of the source and target faces. Methods AttGAN [21], GDWCT [22], StarGAN [23], PGGAN [24] and CycleGAN [25] are all designed for the last two types of forgery to modify facial attributes or reshape facial expressions. 2.2 Face manipulation detection 2.2.1 Traditional methods Previous studies on face-swapped image/video detection approaches were conducted respectively on three bases: the consistency of imaging equipment [26, 27], the remnants of the forgery process [5, 10, 11], and the statistical properties of an image or video itself. The first one attempts to find out the related artefacts generated by the processing unit inside a camera (such as sensor pattern noise, lens distortion, colour filter array artefacts, compression artefacts) or related artefacts generated by processing outside the camera (such as geometric transformations, contrast adjustment, blurring). The second type seeks to determine the authenticity of the image by detecting the specific fingerprints left by GAN. The last detection method is a very common strategy. On the one hand, it is used to detect unreal biological signals appearing in the forged content, including unnatural head movement [28], absence of reflex in the eye area [29], abnormal heart rate [30], mismatched eyebrows [31].. On the other hand, it utilizes hand-craft features and adopts shallow learning techniques to perform detection task. Local features like LBP, SIFT, and SURF have been used in prior works [3, 32, 33]. Akhtar et al. [4] selected 10 traditional local feature descriptors to describe image features, and used SVM to build a classification model. 2.2.2 CNN-based methods In recent years, deep learning has been widely used in many computer vision tasks [5, 12, 34, 35]. Zhou et al. [36] proposed a two-stream CNN for forged face detection. Afchar et al. [37] proposed the detection of videos with forged faces at a mesoscopic level of analysis. Rana et al. [38] fused multiple advanced deep learning models based on deep ensemble learning technology to create a composite classifier with excellent performance. Chang et al. [39] used the SRM filter layer to obtain the noise features of images and input them into the improved VGG16 network to detect fake content. Du et al. [40] constructed a novel detection method to improve the generalization accuracy by making predictions relying on correct forgery evidence. Nguyen et al. [8] trained a CNN model to detect both forged images and segment forged regions simultaneously, which belongs in a multi-task learning problem. Chen et al. [41] proposed a unified framework that takes into account the spatial features within a single frame and the temporal inconsistencies between frames. Similarly, Nguyen et al. [42] proposed a three-dimensional CNN model that can learn spatio-temporal features from an adjacent frame sequence in the video. 2.2.3 Integration of traditional methods and CNN Admittedly, CNN-based methods do have many advantages over traditional approaches. They can extract features of different scales from the input image. Their feature expression ability is stronger, so the abstract features can be automatically obtained. However, CNN-based methods are more costly in resource consumption, more difficult, and time-consuming during training. Specifically, in order to obtain high accuracy, they require a large number of training samples, various optimization techniques, and powerful hardware support. In addition, CNN is poor in both interpretability and feature extraction. Fortunately, traditional methods are superior in the above aspects. Therefore, CNN is treated as a feature extractor and fused with traditional methods in this paper. That is to say, the deep features extracted from CNN are used as supplementary features and fused with multiple hand-crafted features to enhance the detection capabilities of the established model. These features describe specific tampered artefacts and modification traces in the forged images, which can help effectively capture the inconsistent biological signals and unreal detailed information. Therefore, classification models such as Random Forest (RF) or other small-scale neural networks with small parameter spaces can fully meet the needs of classification task. Compared with deep learning models, these small-scale classifiers have lower requirements on the scale of training samples and have more obvious advantages in terms of computational complexity. 3 METHODOLOGY Here, the main concern is the detection of manipulated faces. The flowchart of the proposed method is illustrated in Figure 6. Specifically, for a given input face video, it is first converted into a series of image sequences, and then face detection is performed on each individual frame to crop out the face region. It is worth mentioning that the face is detected with Face Parts Detection tool based on CascadeObjectDetector of FrontalFaceCART, LeftEye, RightEye, Mouth, and Nose. Next, the feature vectors of the face are extracted, which is the key step of our method. Finally, the extracted feature vectors are fed into the RF to train the binary classification model. FIGURE 6Open in figure viewerPowerPoint Flowchart of the proposed face forgery detection scheme 3.1 Face features extraction The key idea of this step is to obtain representative discriminant features that can be distinguished from real faces by analyzing the characteristics of forged faces. From the perspective of image quality analysis, the extraction is conducted concerning texture features, gradient-based sharpness indicators, and frequency domain features related to image quality. Then they are fused with deep features to ensure effective capture of the inconsistent information generated by fake faces and realize the accurate distinction between these two types of facial images. The entire process of face features extraction is shown in Figure 7. In the preprocess, the cropped face is turned into a grey-scale image, and its size is then normalized to 256 × 256. Note that, different from face recognition task, the cropped face does not need to be aligned here. FIGURE 7Open in figure viewerPowerPoint Flowchart of face features extraction 3.1.1 Texture features extraction based on structural weighting As mentioned in the introduction, the fake face region generated by the deep generation network will produce obvious edge artefacts during the process of fusion with the surrounding scene. After blurring, the facial skin and organs (such as teeth, eyes) become too smooth, resulting in loss of texture details and making them no longer consistent with the structural distribution of the background. The texture features of the image are represented by the grey distribution of the central pixel and its surrounding spatial neighbourhood, reflecting the regular and periodic changes of the structural organization and arrangement properties of the image surface. Therefore, the image texture is one of the important features to expose the forgery phenomenon. The classic LBP, a simple and effective local texture descriptor, can capture the rich structural information of the image and has grey-level invariance. It is currently widely used in the fields such as face detection and texture classification. Traditionally, LBP features are extracted from the original image, but in our method, the mean subtracted contrast normalized (MSCN) coefficients [43] calculated by formula (1) is used to extract LBP features. I ̂ ( i , j ) = I ( i , j ) − μ ( i , j ) σ ( i , j ) + C , (1) μ ( i , j ) = ∑ k = − K K ∑ l = − L L w k , l I k , l ( i , j ) , (2) σ ( i , j ) = ∑ k = − K K ∑ l = − L L w k , l [ I k , l ( i , j ) − μ ( i , j ) ] 2 , (3)where, I is a grey-scale image with a size of M × N, I ̂ represents the MSCN coefficient matrix, i ∈ { 1 , 2 , … , M } , j ∈ { 1 , 2 , … , N } , C (C = 1) is used to prevent the denominator of the fraction from being zero, w = { w ( k , 1 ) | k = − K , … , K ; = − L , … , L } symbolizes the Gaussian weighted filter window with central symmetry, K = L = 3. Generalized Gaussian distribution (GGD) is often used in the statistical analysis of image signals, and its shape parameter α and variance parameter σ2 can be regarded as image features for classification or regression. Here, the zero-mean GGD is used to fit the distribution of the MSCN coefficient matrix, and its parameter estimates [α, σ2] are extracted as a set of features, characterizing a two-dimensional vector. The circular LBP operator serves to compare the central pixel value with the pixel value of the remaining points in the circular neighbourhood with a radius of R. If the surrounding pixel value is greater than the central pixel value, the pixel point is marked as 1, otherwise 0. LBP calculation process is expressed as formulas (4) and (5): L B P P , R ( X , Y ) = ∑ n = 0 P − 1 2 n S ( x ) , (4) S ( x ) = 1 , x ≥ 0 0 , x < 0 , (5)where, R = 1, P = 8, (X,Y) represents the coordinates of the central pixel, x = xn − xc is the grey-scale difference between the nth pixel xn and the central pixel xc in the neighbourhood. In this case, a total of 28 = 256 binary patterns can be generated. However, as P increases, the number of generated patterns will increase exponentially. Moreover, this calculation method does not have rotation invariance. By improving the original LBP operator, the rotation-invariant uniform pattern L B P P , R r i u 2 is defined as: L B P P , R r i u 2 ( X , Y ) = ∑ n = 0 P − 1 S ( x ) , U ≤ 2 P + 1 , others , (6)where, U is the number of spatial transitions (that is, the number of 0–1 changes). In this case, the number of rotation-invariant uniform patterns is P + 1 = 9, and all non-uniform patterns are grouped into one. In this way, the texture histogram vector of each image is only 10-dimensional, greatly reducing the calculation and storage space. Therefore, this type of LBP operator is adopted in our method to extract texture features under the MSCN coefficients. In order to make full use of the structure information of the forged image, the local structure map is constructed and used as a weighting factor to highlight the structural loss in the forged image when counting the normalized histogram of LBP features, as shown in formula (7). The construction method of the local structure map MP is used to extract the phase congruency (PC) [44] features from the facial grey-scale image, and then select the larger one of the PC and the MSCN coefficient at each pixel point, as shown in formula (8). H ( k ) = ∑ i = 1 M ∑ j = 1 N w i , j δ ( x ) , δ ( x ) = 1 , x = 0 0 , o t h e r s , (7) M P ( i , j ) = max ( P C ( i , j ) , I ̂ ( i , j ) ) , (8)where, x = L B P R , P r i u 2 − k , k ( k ∈ [ 0 , 9 ] ) represents the number of generated LBP patterns, wi,j is the weighting factor, the amplitude value of the MP. The advantages of the local structure map constructed by the above method are shown in Figure 8. Figures 8a and 8e are a real face and fake face from the DFD dataset, respectively. Figures 8b, 8c and 8d, respectively, show the MSCN coefficients, PC map and MP of Figure 8a, and Figures 8f, 8g and 8h, respectively, represent the MSCN coefficients, PC map and MP of Figure 8e. It can be clearly observed that the MSCN coefficients focus on describing the fine details of the image, while the PC map serves well in capturing large-scale feature information such as the edges and lines. The combined local structure map MP can fully describe the rich edge structure and texture details of the image. Since PC is immune to the brightness and contrast of the image, it is very suitable for facial images taken under different lighting environments. Compared with the real image, this change can be clearly observed on the MSCN, PC and MP maps due to the fact that the forged image is blurred in the face-changing area. Especially for Figure 8h, not only are there artefacts visible to the naked eye at the boundary of the face-changing area, but at the same time, the contours of facial attributes (such as eyes, nose, and mouth) are no longer sharp and clear. Therefore, using the local structure map as the weighting factor of the LBP features is beneficial to highlight the structural loss of the forged image. FIGURE 8Open in figure viewerPowerPoint Examples of local structure map. Figures (a) and (e) are a real face and fake face from the DFD dataset, respectively. Figures (b), (c) and (d), respectively, show the MSCN coefficients, PC map and MP of Figure (a), and Figures (f), (g) and (h), respectively, represent the MSCN coefficients, PC map and MP of Figure (e) It is worth mentioning that since in observing objects, the human visual system follows coarse-to-fine pattern, multi-scale feature extraction can simulate this process. Therefore, low-pass filtering and 2 × 2 downsampling are used in this paper to extract the LBP features of facial images at five scales. In this way, a 60-dimensional texture feature vector is generated for each input image. 3.1.2 Gradient-based sharpness measurement When the texture distribution of the image is quite different, it is very effective to extract texture features. However, when the distinction between the texture information of the image is not clear, such as when the degree of density is similar, and when the thickness is similar, the texture features are usually difficult to accurately reflect the human visual perception. Different from texture, the sharpness indicator reflects the contrast between light and dark in the fine layers of the image, as well as the degree of virtuality and realness of the image contour boundary, which is considered one of the important indicators for evaluating image quality. Since the sharpness of masking-based manipulated images in the face-changing area has decreased to varying degrees, it is necessary to extract the sharpness indicator from the pixel level to jointly describe the characteristic of the forged images. The grey value of the pixel at the edge of the blurred image changes greatly, so their gradient value is greater. In this paper, two gradient-based sharpness indicators are extracted as one of the features of detecting DeepFake. The first sharpness indicator is the variance function, which reflects the dispersion degree of the image grey distribution. The smaller the variance, the less the range of grey value changes, and the lower the dispersion degree of the grey distribution would be, and vice versa. For an M × N grey-scale image I, the variance function can be expressed as: F = ∑ i = 1 M ∑ j = 1 N [ I ( i , j ) − μ ] 2 , (9)where, μ ( μ = 1 M