# FD2Foremer: Thinking Face Forgery Detection in Midfrequency Geometry Details

Authors: Wentao Li; Zhidong Shen
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Face forgery by DeepFake has caused widespread concern in community because of the synthesized media’s risks to the society. However, advances in recent years have been able to produce synthetic images indistinguishable from real images in the RGB space. Extracting midfrequency facial geometry details, including person-specific details and dynamic expression-dependent ones on facial geometry surfaces, is a promising way to highlight forgery clues during face forgery detection. In this paper, we use 3D face reconstruction to generate the displacement map from a single input face image, which is able to represent middle and fine scale details by indicating signed distance from the point in UV space. The cropped face images can also provide eyes and mouse information, so we use face image and its displacement map to extract the image features. Besides, we save the computation cost and maintain competitive performance using a universal transformer architecture and introduce a manifold distillation strategy to train our model from a more complex transformer backbone. Extensive experiments on various public DeepFake datasets indicate the effectiveness of the extracted facial geometry details, and proposed method achieves competitive performance.


---

## Body

## 1. Introduction

The development of deep learning and the availability of large-scale datasets have led to powerful deep generative models, which enables the facial manipulation and images generation. Deep generative models can promote the emerging development of entertainment and cultural industry but can also be used for malicious purposes. One such application of the generative models is DeepFake [1]. DeepFake (e.g., fake images, audios, and videos) has become a real threat to our society due to its realism and impact scopes. It can be used to mislead public opinion, disrupt social order, and even threaten face recognition system [2], intervene in government elections [3], and subvert state power [4]. It has become the most advanced new form of cyberattacks.In this work, we are committed to detecting the facial manipulation, related to the dataset of FaceForensics++ [5], Celeb-DF [6], and DFDC [7], which is difficult to discover the complex artifacts from facial appearance only. At the same time, lots of researchers have been driven to focus on extracting manipulation artifacts from other perspectives besides the RGB space.Some researches [8, 9] extract forgery artifacts using both different low-level local parts and high-level semantic features. The assumption is that the object details should include rapidly changing motion part and subject identities. For example, Feichtenhofer et al. [8] used a two-pathway SlowFast model for video recognition. One pathway is designed to capture semantic information that can be given by images or a few sparse frames. The other pathway is responsible for capturing rapidly changing motion. Zhao et al. [9] believe the difference between the real and fake images in face forgery detection is often subtle and local. So, they use multiple spatial attention heads to make the network attend to different local parts and use textural feature enhancement block to zoom in the subtle artifacts in shallow features. They further use the attention maps to aggregate the low-level textural feature and high-level semantic features. However, these details need to be extracted in multiframes, and we do not know how these details effect the faces from which extracted.In our work, for face forgery detection, we try to extract facial geometry details that are often connected with high frequency including person-specific details and dynamic expression-dependent ones (e.g. deeper skin wrinkles and creases) through 3D faces reconstruction [10]. First, a tentative 3D face reconstruction is conducted with the help of FLAME [11]. 2D image is encoded into a latent code, which consist of albedo parameters α, geometry parameter θ, ψ, β lighting coefficient l, and camera parameters c. A coarse geometry reconstruction can be achieved with the help of above latent code. For the supplement of identity shape details, we then embed another coefficient δ with dimension of 128, representing identity-specific information, to extract geometry details. With the help of a neural network, a displacement map is produced using expression ψ and subject-specific details δ, to augment FLAME mesh with facial geometry information, which refines the low-frequency geometry with higher frequency information. Though the detail reconstruction image lost some high-frequency clues, differing it a little bit from the original face. Displacement map still captures person-specific and dynamic expression-dependent details that often lost in DeepFakes, so we call it “midfrequency” geometry details to help us with the detection.When detecting forgery details with neural networks, we consider using transformer [12] architecture with low computation cost and high efficiency. Good balance of computation cost and high efficiency is often tough to achieve. Some architectures require extremely high computational resources, such as the popular ViTs are heavy-weight, harder to optimize [13], and need L2 regularization to prevent overfitting [14, 15]. So, we use the “MetaFormer” [16] structure with pooling operation for transformer architecture and propose a Forgery-Detection-with-Facial-Detail Transformer (FD2Foremer). To bring the light FD2Foremer with good performance, we introduce the manifold learning during training. We first pretrained swin transformer architecture with forgery details and face images as inputs. Then, the simplified patch embedding manifold loss [17] is used to provide the appropriate constraints on params of the light FD2Foremer with pretrained netw4ork as teacher network.(i)
In summary, our contributions can be summarized in three-fold:(ii)
We start with 3D face reconstruction during forgery detection and output facial geometry details for subtle artifacts capture.(iii)
We introduce the “MetaFormer” architecture into our network and propose a light transformer FD2Foremer with face displacement maps for DeepFake detection.(iv)
We introduce the manifold learning during the light network training. The experimental results on three different public datasets show that our method achieves competitive performance.


## 2. Related Work

Forgery creation, of particular interest in faces, has recently received a lot of attention given its widespread use. To eliminate the risks of misleading forged faces, face forgery detection becomes an increasingly emerging field of research. In this section, we provide a brief overview of several studies have been proposed relevant to our work.


### 2.1. Conventional Image Forgery Detection

Though several techniques have been proposed in the past decades to detect forgery in digital images, those conventional techniques cannot handle the detection of artifacts produced by neural networks well [18–23]. First, forgery is assumed to be done using linear or cubic interpolation by conventional techniques in most cases. Besides, recent advanced forgery techniques leave almost no visible artifacts on tampered faces, which can easily fool sensitive conventional detectors. Furthermore, face forgeries are much smaller and have typical shape, which requires specialized treatments.


### 2.2. Image Forgery Detection with Neural Networks

Nataraj et al. [24] used cooccurrence matrices to exhibit discriminative features of manipulated regions in boundaries shared with neighboring non-manipulated pixels. Their method passed the cooccurrence matrices through a CNN-LSTM model, allowing the network to learn important cooccurrence matrices essential features. Qian et al. found the awareness of frequency, especially under the compression condition, could be a cure. So, they applied frequency-aware decomposition and local frequency statistics on DeepFake detections, finally achieved outstanding performance on low quality media. Zhao et al. [9] used a multiattentional DeepFake detection network to treat face forgery detection as a fine-grained classification problem, mainly focused on different local parts and the subtle artifacts in shallow features. Zhou et al. [25] extracted tampering artifacts and local noise residual features by exploring steganalysis features. However, these methods extract artifacts from pixel levels or image features, i.e., only considering exploring the synthesis clues from the facial appearance.There is also research considering exploring forgery clues among the shape, pose, and the lighting condition of the head. Yang et al. [26] confirmed the significant difference in the estimated head poses in DeepFakes, by comparing head poses estimated from 2D landmarks in the real and faked parts of the face. De Carvalho et al. [27] proposed to explore forgery clues from the 2D illuminant maps of the image segments considering the inconsistency. Zhu et al. [28] disentangled the face image into common texture, identity texture, 3D shape, ambient light, and direct light roughly. And the identity texture and direct light are combined as the facial detail to be fed into a neural network. However, these methods are difficult in recovering small facial details from the input image due to the limited representation power and the extensive calculation.


### 2.3. Transformer Architecture

Transformers are first proposed to learn long-range sequential dependence for translation tasks [12] and then get widely used in numerous natural NLP tasks. With large-scale unlabeled text corpus, transformers achieve amazing performance in language pretraining tasks [29, 30]. Motivated by the success of transformers in NLP, the attention mechanism and transformers are applied to deal with vision tasks, such as image classification [31], object detection [32], image segmentation [14], and image captioning [33]. Notably, Chen et al. introduced iGPT [34], where a transformer is applied to image pixels after reducing image resolution and color space for self-supervised learning. Google proposed a visual transformer (ViT) that achieved state-of-the-art performance on ImageNet classification [31]. They show that ViT need pretraining on large datasets, such as ImageNet-22k and JFT-300M, and huge computation resources to achieve excellent performance in supervised image classification tasks. In 2020, Touvron et al. [14] proposed DeiT with adjusted network architecture, trying to tackle the data-inefficiency problem through data augmentation and knowledge distillation. However, the good performance comes at a high computational cost. To save the computational cost, we shift our attention to the architecture of transformers and what is responsible for the success of the transformers.


## 2.1. Conventional Image Forgery Detection

Though several techniques have been proposed in the past decades to detect forgery in digital images, those conventional techniques cannot handle the detection of artifacts produced by neural networks well [18–23]. First, forgery is assumed to be done using linear or cubic interpolation by conventional techniques in most cases. Besides, recent advanced forgery techniques leave almost no visible artifacts on tampered faces, which can easily fool sensitive conventional detectors. Furthermore, face forgeries are much smaller and have typical shape, which requires specialized treatments.


## 2.2. Image Forgery Detection with Neural Networks

Nataraj et al. [24] used cooccurrence matrices to exhibit discriminative features of manipulated regions in boundaries shared with neighboring non-manipulated pixels. Their method passed the cooccurrence matrices through a CNN-LSTM model, allowing the network to learn important cooccurrence matrices essential features. Qian et al. found the awareness of frequency, especially under the compression condition, could be a cure. So, they applied frequency-aware decomposition and local frequency statistics on DeepFake detections, finally achieved outstanding performance on low quality media. Zhao et al. [9] used a multiattentional DeepFake detection network to treat face forgery detection as a fine-grained classification problem, mainly focused on different local parts and the subtle artifacts in shallow features. Zhou et al. [25] extracted tampering artifacts and local noise residual features by exploring steganalysis features. However, these methods extract artifacts from pixel levels or image features, i.e., only considering exploring the synthesis clues from the facial appearance.There is also research considering exploring forgery clues among the shape, pose, and the lighting condition of the head. Yang et al. [26] confirmed the significant difference in the estimated head poses in DeepFakes, by comparing head poses estimated from 2D landmarks in the real and faked parts of the face. De Carvalho et al. [27] proposed to explore forgery clues from the 2D illuminant maps of the image segments considering the inconsistency. Zhu et al. [28] disentangled the face image into common texture, identity texture, 3D shape, ambient light, and direct light roughly. And the identity texture and direct light are combined as the facial detail to be fed into a neural network. However, these methods are difficult in recovering small facial details from the input image due to the limited representation power and the extensive calculation.


## 2.3. Transformer Architecture

Transformers are first proposed to learn long-range sequential dependence for translation tasks [12] and then get widely used in numerous natural NLP tasks. With large-scale unlabeled text corpus, transformers achieve amazing performance in language pretraining tasks [29, 30]. Motivated by the success of transformers in NLP, the attention mechanism and transformers are applied to deal with vision tasks, such as image classification [31], object detection [32], image segmentation [14], and image captioning [33]. Notably, Chen et al. introduced iGPT [34], where a transformer is applied to image pixels after reducing image resolution and color space for self-supervised learning. Google proposed a visual transformer (ViT) that achieved state-of-the-art performance on ImageNet classification [31]. They show that ViT need pretraining on large datasets, such as ImageNet-22k and JFT-300M, and huge computation resources to achieve excellent performance in supervised image classification tasks. In 2020, Touvron et al. [14] proposed DeiT with adjusted network architecture, trying to tackle the data-inefficiency problem through data augmentation and knowledge distillation. However, the good performance comes at a high computational cost. To save the computational cost, we shift our attention to the architecture of transformers and what is responsible for the success of the transformers.


## 3. Extract Facial Geometry Detail though 3D Reconstruction

We regard the face forgery detection problem as a binary end-to-end classification task about extracted features. The motivation of our work lies in the fact that geometry details (i.e. wrinkles) of individual faces are related with some unchanged identity details, whereas individual expressions affect details either, which contributes to the face forgery detection. Consequently, we extract geometry details in a trial-and-error way during 3D face reconstruction, which consist of expression-related dynamic information, such as wrinkles [35] and identity-specific static information. Dynamic geometry information is often influenced by all kinds of expressions that different for the same individuals, whereas static identity information varies cross different humans. By reconstructing 3D faces, a kind of facial geometry details called displacement map is inferred from both dynamic expressional domain and static identity-specified domain for forgery detection (see Figure 1).Figure 1
Midfrequency geometry details on facial surfaces during face reconstruction. The uniformity of gray scale in green line area is greater than the same area in real images, which indicates that displacement map extracts less geometry details from fake images. And the unnatural concentration of gray scale in red line circles different themselves from the displacement map of real images. (a) Real images and their midfrequency facial geometry details, (b) fake images and their midfrequency facial geometry details.
(a)(b)


### 3.1. 3D Faces Reconstruction

With the help of FLAME, we use an analysis-by-synthesis method to reconstruct 3D face of the input image: a latent code is regressed using an input imageI. Then another image Ir is synthesized by encoding the latent code. As shown in Figure 2, the fully connected layer is connected with a ResNet50 [36] network as the encoder Ec for the regression of the latent code. To synthesize images later, the latent code is divided into 50 albedo parameters α, 100 shape coefficient β, 15 pose parameters θ, 50 expression parameters ψ, 21 lighting coefficients l, and camera parameters c. In total, Ec predicts a 236-dimensional latent code.Figure 2
Extracting facial geometry details during 3D faces reconstruction. During coarse reconstruction, FLAME is used to reconstruct coarse geometry, whose representational power is limited by the low mesh resolution. Then, the convolutional networks are used to produce the displacement map with dynamic expression-related information and static identity-specific information as inputs. The displacement map augments 3D faces reconstruction, which refines the low-frequency geometry with higher frequency information, regarded as clues for face forgery detection.Then, we use FLAME to reconstruct 3D mesh coarsely. FLAME [11] is a 3D statistical face model that can construct the mesh with number of 5023 vertices. Given the pose parameters θ∈ℝ15, shape coefficient β∈ℝβ, and expression parameters ψ∈ℝψ, we can express the mesh as(1)Mβ,θ,ψ=WJβ,TPβ,θ,ψ,W,θ,where WJ,T,W,θ indicates the blend skinning function, J∈ℝ3k are joints and T∈ℝ3n presents vertices that need rotated, W∈ℝk×n indicates the weights used for linear smoothing. Then,(2)TPθ,β,ψ=BPθ;P+Bsβ;S+BEψ;Ε+T,indicates that we add pose correctives BPθ;P: ℝ3k+3⟶ℝ3n, shape blendshapes Bsβ;S:ℝβ⟶ℝ3andexpressionblendshapesBEψ;E:ℝψ⟶ℝ3n, to the no-poses template T, controlled by the pose P, learned identity S, and expression bases E. More details can be found in [11].To extract face texture, the Basel Face Model [37] is adopted and converted into FLAME UV space for the consistence with the FLAME mesh. Given the albedo coefficient α∈ℝα, the albedo int UV space Aα∈ℝd×d×3 are output using Basel Face Model. As for the camera settings, orthographic model is employed for the projection of FLAME meshes based on the assumption that individual faces are shot at a distance. The projection of meshes can be expressed as:(3)v=sΠMi+t,where the vertex Mi⟶ℝ3 is among M, Π∈ℝ2×3 are weights that project 3D vertexes into 2D image space orthographically, t∈ℝ2 indicates 2D translation and s∈ℝ denotes isotropic scale.We use the most frequently-employed Spherical Harmonics (SH) [38] as illumination model. Assuming the Lambertian facial reflectance and distant light source, we shade the face images as the equation below:(4)Bα,Ι,Nuvi,j=Aαi,j⊙∑k=19ΙkHkNi,j,where shaded texture Bi,j∈ℝ3, albedo Ai,j∈ℝ3, and surface normal Ni,j∈ℝ3 corresponds to a specific pixel i,j of UV images. The SH coefficients are defined as Ι=Ι1T,…,Ι9TT, with ΙkT∈ℝ3, and Hk: ℝ3⟶ℝ means basis .⊙ denotes the Hadamard product. The blue box in Figure 2 shows the coarse reconstruction branch for input face image.


### 3.2. Facial Geometry Details Generation

The analysis-by-synthesis method utilizes the latent code of numerous expressions, poses, and shapes to reconstruct the face with the help of FLAME model. But the small number of FLAM vertices and faces limits the representational power of the model, and therefore, FLAME mesh surface almost ignores the middle and fine scale details (see Figure2). Managing to present dynamic expression-related information and static identity-specific person-specific details and expression-dependent details for manipulation detection, we conduct the detail reconstruction [39] to produce displacement maps augmenting low-frequency geometry FLAME with higher frequency details. The pixel values of the displacement map control the signed distance from the point on base mesh to its corresponding point. We locate the surface points of FLAME mesh corresponding to the pixels in the displacement map, then inverse-project the points to the raw mesh along normal direction to find its corresponding points. The detail shape in Figures 2 and 3 shows the effect of our generated displacement maps. Though not perfect and lacking some high frequency information, displacement map representing person-specific and dynamic expression-dependent details is still helpful. So, we regard the displacement maps as “midfrequency” geometry details to help with DeepFake detections.Figure 3
The midfrequency geometry details on FLAME’s surfaces with help of displacement maps.The displacement map in UV coordinate augments the coarse FLAME mesh with facial geometry details by shifting shape points of FLAME mesh. Similar to the coarse reconstruction, another 128-dimension detail codeδ is regressed from image I using an encoder Ed the same architecture as Ec. We then concatenate the jaw pose parameters θjaw, FLAME’s expression parameters ψ with latent code δ, and use detail decoder Fd to generate D:(5)D=Fdθjaw,ψ,δ,where the jaw pose parameters θjaw∈ℝ3 and FLAME expression ψ∈ℝ50 indicate the dynamic expression-related information, meanwhile δ∈ℝ128 is the latent code representing the static identity-specific information. We then convert the D into a normal map for rendering.The geometry displacement map makes it possible to reconstruct 3D face with midfrequency information. Converted into UV space,Muv∈ℝd×d×3 and its surface normal Nuv∈ℝd×d×3 are combined with D to augment the mesh with geometry details as(6)Muv′=Muv+D⊙Nuv.Applying normal mapN′ from M′, the synthesis image Ir′ is rendered as(7)Ir′=ℛM,Bα,Ι,N′,c.Comparing the rendered detailed image with the real image, the decoderFd is forced to model detailed geometric information, with the help of the coarse reconstruction on VGGFace2 [40], BUPT-Balancedface [41], and Vox-Celeb2 [42]. As shown in Figure 2, midfrequency details in rendered images Ir′, including both dynamic expression-related information and static identity-specific information, are inferred mainly from the displacement map, which is exactly what we need for forged faces detection. We call the encoder Ec, encoder Ed and decoder Ed together as facial detail generator Gd to produce midfrequency facial geometry details.


## 3.1. 3D Faces Reconstruction

With the help of FLAME, we use an analysis-by-synthesis method to reconstruct 3D face of the input image: a latent code is regressed using an input imageI. Then another image Ir is synthesized by encoding the latent code. As shown in Figure 2, the fully connected layer is connected with a ResNet50 [36] network as the encoder Ec for the regression of the latent code. To synthesize images later, the latent code is divided into 50 albedo parameters α, 100 shape coefficient β, 15 pose parameters θ, 50 expression parameters ψ, 21 lighting coefficients l, and camera parameters c. In total, Ec predicts a 236-dimensional latent code.Figure 2
Extracting facial geometry details during 3D faces reconstruction. During coarse reconstruction, FLAME is used to reconstruct coarse geometry, whose representational power is limited by the low mesh resolution. Then, the convolutional networks are used to produce the displacement map with dynamic expression-related information and static identity-specific information as inputs. The displacement map augments 3D faces reconstruction, which refines the low-frequency geometry with higher frequency information, regarded as clues for face forgery detection.Then, we use FLAME to reconstruct 3D mesh coarsely. FLAME [11] is a 3D statistical face model that can construct the mesh with number of 5023 vertices. Given the pose parameters θ∈ℝ15, shape coefficient β∈ℝβ, and expression parameters ψ∈ℝψ, we can express the mesh as(1)Mβ,θ,ψ=WJβ,TPβ,θ,ψ,W,θ,where WJ,T,W,θ indicates the blend skinning function, J∈ℝ3k are joints and T∈ℝ3n presents vertices that need rotated, W∈ℝk×n indicates the weights used for linear smoothing. Then,(2)TPθ,β,ψ=BPθ;P+Bsβ;S+BEψ;Ε+T,indicates that we add pose correctives BPθ;P: ℝ3k+3⟶ℝ3n, shape blendshapes Bsβ;S:ℝβ⟶ℝ3andexpressionblendshapesBEψ;E:ℝψ⟶ℝ3n, to the no-poses template T, controlled by the pose P, learned identity S, and expression bases E. More details can be found in [11].To extract face texture, the Basel Face Model [37] is adopted and converted into FLAME UV space for the consistence with the FLAME mesh. Given the albedo coefficient α∈ℝα, the albedo int UV space Aα∈ℝd×d×3 are output using Basel Face Model. As for the camera settings, orthographic model is employed for the projection of FLAME meshes based on the assumption that individual faces are shot at a distance. The projection of meshes can be expressed as:(3)v=sΠMi+t,where the vertex Mi⟶ℝ3 is among M, Π∈ℝ2×3 are weights that project 3D vertexes into 2D image space orthographically, t∈ℝ2 indicates 2D translation and s∈ℝ denotes isotropic scale.We use the most frequently-employed Spherical Harmonics (SH) [38] as illumination model. Assuming the Lambertian facial reflectance and distant light source, we shade the face images as the equation below:(4)Bα,Ι,Nuvi,j=Aαi,j⊙∑k=19ΙkHkNi,j,where shaded texture Bi,j∈ℝ3, albedo Ai,j∈ℝ3, and surface normal Ni,j∈ℝ3 corresponds to a specific pixel i,j of UV images. The SH coefficients are defined as Ι=Ι1T,…,Ι9TT, with ΙkT∈ℝ3, and Hk: ℝ3⟶ℝ means basis .⊙ denotes the Hadamard product. The blue box in Figure 2 shows the coarse reconstruction branch for input face image.


## 3.2. Facial Geometry Details Generation

The analysis-by-synthesis method utilizes the latent code of numerous expressions, poses, and shapes to reconstruct the face with the help of FLAME model. But the small number of FLAM vertices and faces limits the representational power of the model, and therefore, FLAME mesh surface almost ignores the middle and fine scale details (see Figure2). Managing to present dynamic expression-related information and static identity-specific person-specific details and expression-dependent details for manipulation detection, we conduct the detail reconstruction [39] to produce displacement maps augmenting low-frequency geometry FLAME with higher frequency details. The pixel values of the displacement map control the signed distance from the point on base mesh to its corresponding point. We locate the surface points of FLAME mesh corresponding to the pixels in the displacement map, then inverse-project the points to the raw mesh along normal direction to find its corresponding points. The detail shape in Figures 2 and 3 shows the effect of our generated displacement maps. Though not perfect and lacking some high frequency information, displacement map representing person-specific and dynamic expression-dependent details is still helpful. So, we regard the displacement maps as “midfrequency” geometry details to help with DeepFake detections.Figure 3
The midfrequency geometry details on FLAME’s surfaces with help of displacement maps.The displacement map in UV coordinate augments the coarse FLAME mesh with facial geometry details by shifting shape points of FLAME mesh. Similar to the coarse reconstruction, another 128-dimension detail codeδ is regressed from image I using an encoder Ed the same architecture as Ec. We then concatenate the jaw pose parameters θjaw, FLAME’s expression parameters ψ with latent code δ, and use detail decoder Fd to generate D:(5)D=Fdθjaw,ψ,δ,where the jaw pose parameters θjaw∈ℝ3 and FLAME expression ψ∈ℝ50 indicate the dynamic expression-related information, meanwhile δ∈ℝ128 is the latent code representing the static identity-specific information. We then convert the D into a normal map for rendering.The geometry displacement map makes it possible to reconstruct 3D face with midfrequency information. Converted into UV space,Muv∈ℝd×d×3 and its surface normal Nuv∈ℝd×d×3 are combined with D to augment the mesh with geometry details as(6)Muv′=Muv+D⊙Nuv.Applying normal mapN′ from M′, the synthesis image Ir′ is rendered as(7)Ir′=ℛM,Bα,Ι,N′,c.Comparing the rendered detailed image with the real image, the decoderFd is forced to model detailed geometric information, with the help of the coarse reconstruction on VGGFace2 [40], BUPT-Balancedface [41], and Vox-Celeb2 [42]. As shown in Figure 2, midfrequency details in rendered images Ir′, including both dynamic expression-related information and static identity-specific information, are inferred mainly from the displacement map, which is exactly what we need for forged faces detection. We call the encoder Ec, encoder Ed and decoder Ed together as facial detail generator Gd to produce midfrequency facial geometry details.


## 4. Methodology

In the following sections, we propose the Forgery-Detection-with-Facial-Detail Transformer (FD2Former), including backbone to extract image features, the introduced transformer architecture and the fine-grained manifold distillation strategy.


### 4.1. Backbone

We employ face recognition [43] DL libraries to detect and crop faces frame by frame. And the facial geometry detail generator Gd is used to generate middle and fine scale details for face images. The existing methods using an alignment only centralizes the face without considering whether the face is frontolized, which easily leads to facial information loss. With the facial detail displacement map converted to UV space, the facial geometry details for all the faces can be located in the same spatial space. Since the pixels of UV displacement maps corresponding to full 3D face mesh, there is no information loss. Aligned face images can also provide pose, eyes blink and mouth movement information that cannot be perceived in the detail displacement map, so we use both face image and its detail displacement map to extract the face manipulation clues. In order to learn more facial movement information and facial geometry details, we choose convert inputs into much more informative high level image features rather than image patches directly. The aligned face images and the displacement map are concatenated and then fed into CNN backbone to extract high level image features. We employ a ResNet as the high-level image features extractor.


### 4.2. Forgery-Detection-with-Facial-Detail Transformer

Figure4 illustrates the architecture of the proposed Forgery-Detection-with-Facial-Detail Transformer for DeepFake detection. From the perspective of transformers introduction [12], many works have paid great attention to attention and focused on designing various attention-based token mixer components to achieve good performance. However, the good performance comes at a high computational cost, and these works pay little attention to the general architecture.Figure 4
The architecture of the proposed forgery-detection-with-facial-detail transformer, including the cropped face image and their corresponding displacement map as input, convolutional networks as backbone for image feature extraction and 24 transformer blocks for feature learning.Considering what makes it effect for the success of transformers, we use “MetaFormer” concept for our work. MetaFormer is a general architecture abstracted from numerous transformers [12], where the most components remain the same as transformers, but the token mixer is not specified. MetaFormer first apply input embedding to the input I, such as patch embedding for ViTs [31]:(8)X=InputEmbI,where X∈ℝN×C denotes the embedding tokens with sequence length N and embedding dimension C.Then, embedding tokens are fed into several MetaFormer blocks, each of which consists of two residual subblocks. Specifically, the first subblock with a token mixer is usually designed to mix token information and can be expressed as(9)Y=TokenMixerNormX+X,where Norm· means the normalization such as Layer Normalization [1]; TokenMixer· denotes a module mainly working for communicating information among tokens. In recently works, vision transformer models [31, 44, 45] and spatial MLP in MLP-like models [46, 47] have implemented various kinds of attention mechanism, which aims at mainly propagating token information and some mixing channels, like attention.The second subblock includes a two-layered MLP with nonlinear activation function,(10)Z=σNormYW1W2+Y,where W1∈ℝC×rC and W2∈ℝC×rC are learnable parameters with MLP expansion ratio r; σ· is a nonlinear activation function, such as GELU [48] or ReLU [49].We believe that such a general architecture contributes mostly to the success of the recent transformer and MLP-like models. To decrease the number of learnable parameters and save computational costs, we employ the simple operator, pooling, as the token mixer for Forgery-Detection-with-Facial-Detail Transformer.Assuming the inputT is in channel-first format, T∈ℝC×H×W, the pooling operator can be expressed as(11)T:,i,j′=1K×K∑p,q=1KT:,i+p−K+12,i+q−K+12−T:,i,j,where K means the pooling size. For the consistency with the residual connection in FD2Former block, subtraction of the input itself is added in (11). The PyTorch-like code of the pooling is shown in Algorithm 1.Unlike self-attention and spatial MLP that have computational complexity quadratic to the number of tokens, the pooling operation acquires a computational complexity linear to the sequence length without any learnable parameters. The overall framework of the FD2Former transformer part has 4 stages with H/4×W/4, H/8×W/8, H/16×W/16, and H/32×W/32 tokens, respectively, where H and W represent the height and width of the image features. The model has embedding dimensions of 64, 128, 320, and 512 responding to the four stages. And FD2Former has 24 blocks in total, where stages 1, 2, 3, and 4 contain 4, 4, 12, and 4 FD2Former blocks, respectively. The MLP expansion ratio is set as 4. Not surprisingly, the FD2Former of “MetaFormer” architecture with 24 blocks have fewer parameters (29M) than the same one of “Swin-S” architecture [50] with 24 blocks.import torch.nn as nnclass Pooling(nn.Module):def __init__(self, pool_size = 3):super().__init__()“““Padding size is set as half of pool size.”””self.pool = nn.AvgPool2d(pool_size, stride = 1padding = pool_size//2,count_include_pad = False)def forward(self,x):”””[B, C, H, W] =x.shapeSubtraction of the input feature is added, considering the residual connection of the transformer blocks.”””return self.pool(x) − x


### 4.3. Fine-Grained Manifold Distillation

To fully excavate the strong capacity, the fine-grained manifold distillation strategy is used to train the proposed FD2Former with the pretrained same FD2Former but of swin transformer block as the teacher. Since FD2Former of pooling operation can get better limitation during training, with a teacher network having more complex structure (see Figure 5).Figure 5
The schematic illustration of the distillation outline. The manifold distillation loss is plugged in the selected blocks of student and teacher DF2Former.For an appropriate constraint during model training, knowledge distillation should not only focus on distilling the output logic [14] but also consider the intermediate features images and their relationship. A natural thought about transferring feature maps may be a workable way, but its harsh conditions for the selection of teacher model cannot be ignored, which requires student and teacher models to have the same feature embedding dimension. Besides, it does not make use of interpatch information.We use the fine-grained manifold distillation to lift the limitation about the number of heads and dimension of embedding features during distillation. Specifically, given a batch of images that described asx1,y1, …, xn,yn, a single image can be divided into p patches. cS and cT indicates the patch embedding dimension for student and teacher DF2Former. And feature maps between student or teacher transformer blocks can be expressed as FT∈ℝn×p×cT and FS∈ℝn×p×cS. For the student DF2Former, the patch-level manifold of feature maps FS is produced through the calculation of relative distance as:(12)ℳFS=γFS′γFS′T,where γ is the conversion which reshape ℝn,p,c⟶ℝnp,c and FS′i,j,:=FSi,j,:/FSi,j,:2 means the embedding normalization. But should not be ignored is the unbearable load of computation for such a kind of calculation, if we compute the patch-level manifold gap between teacher and student as above. Since the batch size, number of patches and the embedding dimension of each patch lead to a good many calculations. Thus, the patch embedding manifold should be further simplified, which decomposed into two relations sample terms and one random sample error correction term. The manifold loss is simplified as:(13)Lmanifold−sp=αLcp+βLrs+Lci,where Lcp means cross-patch manifold loss, Lci indicates cross-image loss and Lrs means random-selected loss. α and β are weights that balance the contribution of these terms. Three terms are expressed as:(14)Lci=∑k=0pℳFT:,k,:−ℳFS:,k,:F2pLcp=∑s=0nℳFTs,:,:−ℳFSs,:,:F2nLrs=ℳFTr−ℳFSrF2,where FTr and FSr are random selection from FT and FS, whose dimensions are k,cS and k,cT. The number of randomly selected patches is controlled by k. Figure 6 illustrates the meaning of terms Lci, Lcp and Lrs.Figure 6
The diagram for computing simplified manifold loss. For the convenience of display, three feature maps are involved and each feature map is divided into four patches. (a) Cross-patch. (b) Cross-image. (c) Random-selected. (d) Patch embedding.
(a)(b)(c)(d)Considering the effectiveness of the hard-label distillation, the simplified patch embedding manifold loss is combined with the former to properly limit the student FD2Fomer during training:(15)Ltotal=∑lLmanifold−sp+12HψfsX,y+12HψfsX,yt,where l means the selected blocks for the insertion of simplified manifold loss.


## 4.1. Backbone

We employ face recognition [43] DL libraries to detect and crop faces frame by frame. And the facial geometry detail generator Gd is used to generate middle and fine scale details for face images. The existing methods using an alignment only centralizes the face without considering whether the face is frontolized, which easily leads to facial information loss. With the facial detail displacement map converted to UV space, the facial geometry details for all the faces can be located in the same spatial space. Since the pixels of UV displacement maps corresponding to full 3D face mesh, there is no information loss. Aligned face images can also provide pose, eyes blink and mouth movement information that cannot be perceived in the detail displacement map, so we use both face image and its detail displacement map to extract the face manipulation clues. In order to learn more facial movement information and facial geometry details, we choose convert inputs into much more informative high level image features rather than image patches directly. The aligned face images and the displacement map are concatenated and then fed into CNN backbone to extract high level image features. We employ a ResNet as the high-level image features extractor.


## 4.2. Forgery-Detection-with-Facial-Detail Transformer

Figure4 illustrates the architecture of the proposed Forgery-Detection-with-Facial-Detail Transformer for DeepFake detection. From the perspective of transformers introduction [12], many works have paid great attention to attention and focused on designing various attention-based token mixer components to achieve good performance. However, the good performance comes at a high computational cost, and these works pay little attention to the general architecture.Figure 4
The architecture of the proposed forgery-detection-with-facial-detail transformer, including the cropped face image and their corresponding displacement map as input, convolutional networks as backbone for image feature extraction and 24 transformer blocks for feature learning.Considering what makes it effect for the success of transformers, we use “MetaFormer” concept for our work. MetaFormer is a general architecture abstracted from numerous transformers [12], where the most components remain the same as transformers, but the token mixer is not specified. MetaFormer first apply input embedding to the input I, such as patch embedding for ViTs [31]:(8)X=InputEmbI,where X∈ℝN×C denotes the embedding tokens with sequence length N and embedding dimension C.Then, embedding tokens are fed into several MetaFormer blocks, each of which consists of two residual subblocks. Specifically, the first subblock with a token mixer is usually designed to mix token information and can be expressed as(9)Y=TokenMixerNormX+X,where Norm· means the normalization such as Layer Normalization [1]; TokenMixer· denotes a module mainly working for communicating information among tokens. In recently works, vision transformer models [31, 44, 45] and spatial MLP in MLP-like models [46, 47] have implemented various kinds of attention mechanism, which aims at mainly propagating token information and some mixing channels, like attention.The second subblock includes a two-layered MLP with nonlinear activation function,(10)Z=σNormYW1W2+Y,where W1∈ℝC×rC and W2∈ℝC×rC are learnable parameters with MLP expansion ratio r; σ· is a nonlinear activation function, such as GELU [48] or ReLU [49].We believe that such a general architecture contributes mostly to the success of the recent transformer and MLP-like models. To decrease the number of learnable parameters and save computational costs, we employ the simple operator, pooling, as the token mixer for Forgery-Detection-with-Facial-Detail Transformer.Assuming the inputT is in channel-first format, T∈ℝC×H×W, the pooling operator can be expressed as(11)T:,i,j′=1K×K∑p,q=1KT:,i+p−K+12,i+q−K+12−T:,i,j,where K means the pooling size. For the consistency with the residual connection in FD2Former block, subtraction of the input itself is added in (11). The PyTorch-like code of the pooling is shown in Algorithm 1.Unlike self-attention and spatial MLP that have computational complexity quadratic to the number of tokens, the pooling operation acquires a computational complexity linear to the sequence length without any learnable parameters. The overall framework of the FD2Former transformer part has 4 stages with H/4×W/4, H/8×W/8, H/16×W/16, and H/32×W/32 tokens, respectively, where H and W represent the height and width of the image features. The model has embedding dimensions of 64, 128, 320, and 512 responding to the four stages. And FD2Former has 24 blocks in total, where stages 1, 2, 3, and 4 contain 4, 4, 12, and 4 FD2Former blocks, respectively. The MLP expansion ratio is set as 4. Not surprisingly, the FD2Former of “MetaFormer” architecture with 24 blocks have fewer parameters (29M) than the same one of “Swin-S” architecture [50] with 24 blocks.import torch.nn as nnclass Pooling(nn.Module):def __init__(self, pool_size = 3):super().__init__()“““Padding size is set as half of pool size.”””self.pool = nn.AvgPool2d(pool_size, stride = 1padding = pool_size//2,count_include_pad = False)def forward(self,x):”””[B, C, H, W] =x.shapeSubtraction of the input feature is added, considering the residual connection of the transformer blocks.”””return self.pool(x) − x


## 4.3. Fine-Grained Manifold Distillation

To fully excavate the strong capacity, the fine-grained manifold distillation strategy is used to train the proposed FD2Former with the pretrained same FD2Former but of swin transformer block as the teacher. Since FD2Former of pooling operation can get better limitation during training, with a teacher network having more complex structure (see Figure 5).Figure 5
The schematic illustration of the distillation outline. The manifold distillation loss is plugged in the selected blocks of student and teacher DF2Former.For an appropriate constraint during model training, knowledge distillation should not only focus on distilling the output logic [14] but also consider the intermediate features images and their relationship. A natural thought about transferring feature maps may be a workable way, but its harsh conditions for the selection of teacher model cannot be ignored, which requires student and teacher models to have the same feature embedding dimension. Besides, it does not make use of interpatch information.We use the fine-grained manifold distillation to lift the limitation about the number of heads and dimension of embedding features during distillation. Specifically, given a batch of images that described asx1,y1, …, xn,yn, a single image can be divided into p patches. cS and cT indicates the patch embedding dimension for student and teacher DF2Former. And feature maps between student or teacher transformer blocks can be expressed as FT∈ℝn×p×cT and FS∈ℝn×p×cS. For the student DF2Former, the patch-level manifold of feature maps FS is produced through the calculation of relative distance as:(12)ℳFS=γFS′γFS′T,where γ is the conversion which reshape ℝn,p,c⟶ℝnp,c and FS′i,j,:=FSi,j,:/FSi,j,:2 means the embedding normalization. But should not be ignored is the unbearable load of computation for such a kind of calculation, if we compute the patch-level manifold gap between teacher and student as above. Since the batch size, number of patches and the embedding dimension of each patch lead to a good many calculations. Thus, the patch embedding manifold should be further simplified, which decomposed into two relations sample terms and one random sample error correction term. The manifold loss is simplified as:(13)Lmanifold−sp=αLcp+βLrs+Lci,where Lcp means cross-patch manifold loss, Lci indicates cross-image loss and Lrs means random-selected loss. α and β are weights that balance the contribution of these terms. Three terms are expressed as:(14)Lci=∑k=0pℳFT:,k,:−ℳFS:,k,:F2pLcp=∑s=0nℳFTs,:,:−ℳFSs,:,:F2nLrs=ℳFTr−ℳFSrF2,where FTr and FSr are random selection from FT and FS, whose dimensions are k,cS and k,cT. The number of randomly selected patches is controlled by k. Figure 6 illustrates the meaning of terms Lci, Lcp and Lrs.Figure 6
The diagram for computing simplified manifold loss. For the convenience of display, three feature maps are involved and each feature map is divided into four patches. (a) Cross-patch. (b) Cross-image. (c) Random-selected. (d) Patch embedding.
(a)(b)(c)(d)Considering the effectiveness of the hard-label distillation, the simplified patch embedding manifold loss is combined with the former to properly limit the student FD2Fomer during training:(15)Ltotal=∑lLmanifold−sp+12HψfsX,y+12HψfsX,yt,where l means the selected blocks for the insertion of simplified manifold loss.


## 5. Experiments

In this section, we first introduce benchmark datasets and details about implementations. Also, we conduct a set of ablation studies, and compare our method with previous works. We will describe the datasets and implementation details in Section5.1. The ablation studies and the analysis of the manifold learning strategy are described in Section 5.2 and Section 5.3. We will analyze the experimental results compared with the previous work in Section 5.4.


### 5.1. Datasets and Implementation Details

#### 5.1.1. Training Dataset

Faceforensics++ (FF++) [5] is a dataset released to standardize the evaluation of face forgery detection methods, which includes 1000 original videos and other 4000 manipulated videos. The manipulated videos are generated using four typical face swapping and reenactment methods, i.e., DeepFakes (DF) [51], FaceSwap (FS) [52], Face2Face (F2F) [20], and NeuralTextures (NT) [53]. Besides, there are three versions of FF++ in terms of compression level, i.e., raw, lightly compressed (HQ), and heavily compressed (LQ). Higher the compression level, the harder it is to distinguish the forgery traces. Since the uploaded manipulated videos always have a limited quality, we use the LQ versions in most experiments. We sample 270 frames for each train video and 100 frames for each test one.


#### 5.1.2. Test Datasets

The following datasets are adopted for evaluation. (1) the testing set of FF++ as described. (2) DeepFakes Detection Challenge (DFDC) dataset [7] containing a total of 123,546 videos with the help of paid actors. Each video lasts about 10 seconds and consists of 300 frames. (3) The Celeb-DF dataset [6] containing 408 real videos and 795 synthesized video sequences with reduced visual artifacts, released for the advance of research on manipulated face detection. The examples of data are shown in Figure 7.Figure 7
Examples of data from FaceForensics++, DFDC, and Celeb-DF datasets.


#### 5.1.3. Implementation Details

For the facial geometry detail generation, we acquire the detail displacement map using facial geometry detail generatorGd. For the neural network, we train the FD2Former of swin transformer block rather than pooling operation as the teacher. Then, we use the simplified patch embedding manifold loss to teach FD2Foremer of “MetaFormer” architecture with pretrained network. The Adamw optimizer is utilized for training with the initial learning rate of 5×10−4 and the warm-up learning rate is 5×10−7. The batch size is set to 64 and weight decay equals to 0.05, β1 = 0.9, β2 = 0.999. We use a cosine decay learning rate scheduler and 5 epochs of linear warm-up. In our implementation, the total epoch is 30.


### 5.2. Ablation Studies

For the analysis of the Forgery-Detection-with-Facial-Detail Transformer, facial geometry details and face image are regarded as complementary inputs and a transformer is utilized for the forged face detection. To evaluate each input, we quantitatively evaluate FD2Former with different inputs: a transformer with facial geometry details as input only, a transformer with cropped face image only, and a transformer with both original images and facial geometry details as input. The results are listed in Table 1.Table 1
Test results (%) of the FD2Former and its variants on FF++(LQ), DFDC and Celeb-DF. The “image” indicates the model with cropped face images as input only. The “detail” indicates the model with facial geometry details as input only. The “swin” indicates the FD2Former of the swin transformer backbone. The “meta” is the FD2Former of the MetaFormer backbone. The “manifold” and the “hard” mean the manifold distillation and the hard-label distillation respectively. The metric on FF++(LQ), DFDC and Celeb-DF dataset is ACC.
StructureFF++(LQ)DFDCCeleb-DFXception80.3285.6061.25Image(swin)81.1486.3278.13Detail(swin)78.0680.6876.94Img + detail(swin)83.2387.9783.51Img + detail + manifold(meta)82.7386.7281.36Img + detail + hard(meta)81.6786.0379.91The metric on FF++(LQ), DFDC and Celeb-DF dataset is ACC. Best results are shown in bold.First, the transformer only detects geometry details or cropped face achieves similar results. However, the model performs better with both face image and facial geometry details as input. Besides, the “MetaFormer” architecture achieves the similar and competitive results compared with the swin transformer architecture, but it has less learnable parameters, which saves training resources. Shown as Figure8, FD2Former(swin) almost saves half learnable params of CViT, while the accuracy only decreases by 0.53%.Figure 8
Comparison of the detection accuracy (%) and learnable params (M) with previous methods, Xception and CViT. The “FD2Former(swin)” is the FD2Former of the swin transformer backbone. The “FD2Former(meta)” is the FD2Former of the MetaFormer backbone. The experiment is conducted on DFDC dataset.We are also interested in the performance of the manifold learning strategy, compared with the hard-label distillation method. As shown in Table1, the manifold learning strategy is more efficient than the hard-label distillation strategy. The distilled FD2Former using our method outperform the model using hard-label distillation by 1.45% on Celeb-DF. The potential negative impact of the manifold learning strategy may be the increased consumption of computation resources and energy.


### 5.3. Analysis of the Fine-Grained Manifold Distillation

Aimed at patch-level knowledge distillation, we can insert the simplified manifold loss into any blocks under the condition of same patch numbers at corresponding blocks. As shown in Table2, the experiments are conducted to find the better insert location of the manifold loss. Results show that applying manifold loss at both the last stage and first stage improves the performance of student network better. We think it is because such a kind of insertion constrains the student network properly, while the student’s capacity could be limited if losses inserted in middle of the model.Table 2
Results (%) of distillation for different manifold computing location. All trained with the image and the displacement map as inputs on low-quality FF++.
TeacherStudentTeacher blocks extractedStudent blocks extractedACCDF2Former(swin)DF2Former(meta){11, 12}{23, 24}82.31DF2Former(swin)DF2Former(meta){1, 2}{1, 2}82.15DF2Former(swin)DF2Former(meta){1, 2, 11, 12}{1, 2, 23, 24}82.73DF2Former(swin)DF2Former(meta){1, 2, 3, 4, 11, 12}{1, 2, 5, 6, 23, 24}81.39All trained with the image and the displacement map as inputs on low-quality FF++. Best results are shown in bold.We test different values of hyper-parametersα and β, and presents results in Tables 3 and 4. The experiments show that assigning 1 and 0.2 to α and β improves the performance relatively. As Figures 9 and 10 shown, a small β could be more efficient because Lrs may partially coincide with Lcp, Lci terms, meanwhile the uncertainty of sample mechanism should be controlled.Table 3
Results (%) of distillation for a sequence of selectedα. All trained with the image and the displacement map as inputs on low quality FF++. The metric is ACC.
TeacherStudentαTesting data (ACC)FF++(LQ)DFDCDF2Former(swin)DF2Former(meta)0.581.7686.17DF2Former(swin)DF2Former(meta)182.7386.72DF2Former(swin)DF2Former(meta)1.582.4286.47DF2Former(swin)DF2Former(meta)282.1985.74DF2Former(swin)DF2Former(meta)2.581.7185.86The metric is ACC. Best results are shown in bold.Table 4
Results (%) of distillation for a sequence of selectedβ. All trained with the image and the displacement map as inputs on low-quality FF++. The metric is ACC.
TeacherStudentβTesting data (ACC)FF++(LQ)DFDCDF2Former(swin)DF2Former(meta)081.8585.43DF2Former(swin)DF2Former(meta)0.282.7386.72DF2Former(swin)DF2Former(meta)0.482.8685.77DF2Former(swin)DF2Former(meta)0.682.8386.58DF2Former(swin)DF2Former(meta)0.882.5186.27DF2Former(swin)DF2Former(meta)182.3786.08The metric is ACC. Best results are shown in bold.Figure 9
Accuracy (%) of the DF2Former(meta) when the hyper-parameter α varies. “(LQ)” refers to the heavily compressed FaceForensics++ dataset.Figure 10
Accuracy (%) of the DF2Former(meta) when the hyper-parameter β varies. “(LQ)” refers to the heavily compressed FaceForensics++ dataset.We also test the hyper-parameterk, which means the number of randomly selected patches for calculation of the sampling correction, random sampler loss Lrs. As Table 5 shows, five experiments are launched to search for an appropriate value of k, and we finally assign 256 to k for a better performance. As indicated in Figure 11, a bigger k usually leads to a better performance.Table 5
Results (%) of distillation for a sequence of selectedk. All trained with the image and the displacement map as inputs on low quality FF++. The metric is ACC.
TeacherStudentkTesting data (ACC)FF++(LQ)DFDCDF2Former(swin)DF2Former(meta)081.8585.43DF2Former(swin)DF2Former(meta)6482.4185.71DF2Former(swin)DF2Former(meta)12882.2786.54DF2Former(swin)DF2Former(meta)19282.6886.37DF2Former(swin)DF2Former(meta)25682.7386.72The metric is ACC. Best results are shown in bold.Figure 11
Accuracy (%) of the DF2Former(meta) when the hyper-parameter k varies. “(LQ)” refers to the heavily compressed FaceForensics++ dataset.


### 5.4. Comparison with Other Methods

Some previous works have shown the generalization performance problem if confronted with the unseen manipulation methods. In this section, the proposed method is compared with previous ones, Xception [54] and Face X-ray [55], to explore the performance during detecting unseen manipulation methods or datasets.Following Luo et al. [56], the evaluations on different unseen manipulation methods are conducted on the FF++(HQ) [5] database, and we compare the performance with previous methods. The proposed method is trained on F2F, DF, FS, and NT separately and tested on the remaining methods, taking the AUC as the evaluation metric. Table 6 presents the experimental results. Compared to the classic method XceptionNet, the proposed FD2Former achieves a significant improvement most of time, since the former overly relies on the texture patterns in RGB space without thinking unseen details. Face X-ray [55], achieves a better generalization performance benefitting from the blending evidence detection. The FD2Former leverages both textures and highlighted clues extracted from the facial geometry details, which probably why it generalizes better from one method to another.Table 6
Generalization evaluation (%) with previous methods on HQ (c23) FF++. AUC is used as metric of performance on the unseen manipulation technique. The highlighted are best results.
Training dataModelTesting data (AUC)DFF2FFSNTDFXception [54]99.373.649.073.6Face X-ray [55]98.763.360.069.8FD2Former98.9469.1759.5877.39F2FXception [54]80.399.476.269.6Face X-ray [55]63.098.493.894.5FD2Former81.7898.1277.3189.52FSXception [54]66.488.899.471.3Face X-ray [55]45.896.198.195.7FD2Former70.2197.8399.2793.12NTXception [54]79.981.373.199.1Face X-ray [55]70.591.791.092.5FD2Former80.4390.8292.5394.62AUC is used as metric of performance on the unseen manipulation technique. Best results are shown in bold.Following [57] Khodabakhsh et al. [58], the generalization performance is analyzed quantitatively on unseen data and compared with other methods, including the classic Xception [54], the ensemble of EfficientNet’s variants [59], and the Face X-ray [55]. The models are trained on FF++ (HQ) [5] and evaluated on Celeb-DF [6], DFDC [7], respectively. Such a generalization problem may confront more challenges than the experiments within FF++. We can see from Table 7 that our method achieves apparent improvements over EfficientNetB4 Ensemble [59] and Xception [54], indicating that our model learns more robust representations than the previous methods. Figure 12 displays the obtained result of some tested face images.Table 7
Generalization evaluation (%) on the unseen dataset, Celeb-DF, and DFDC. The highlighted are best results.
ModelTrainingTesting AUCCeleb-DFDFDCEfficientNetB4Ensemble [59]FF++55.863.0Xception [54]59.467.9Face X-ray [55]74.270.0FD2Former79.273.6Generalization evaluation (%) on the unseen dataset, Celeb-DF, and DFDC. Best results are shown in bold.Figure 12
Obtained result of some tested face images for DeepFake detection with the estimated category and the probabilities of each one. (a) FaceForensics++. (b) DFDC. (c) Celeb-DF.
(a)(b)(c)


## 5.1. Datasets and Implementation Details

### 5.1.1. Training Dataset

Faceforensics++ (FF++) [5] is a dataset released to standardize the evaluation of face forgery detection methods, which includes 1000 original videos and other 4000 manipulated videos. The manipulated videos are generated using four typical face swapping and reenactment methods, i.e., DeepFakes (DF) [51], FaceSwap (FS) [52], Face2Face (F2F) [20], and NeuralTextures (NT) [53]. Besides, there are three versions of FF++ in terms of compression level, i.e., raw, lightly compressed (HQ), and heavily compressed (LQ). Higher the compression level, the harder it is to distinguish the forgery traces. Since the uploaded manipulated videos always have a limited quality, we use the LQ versions in most experiments. We sample 270 frames for each train video and 100 frames for each test one.


### 5.1.2. Test Datasets

The following datasets are adopted for evaluation. (1) the testing set of FF++ as described. (2) DeepFakes Detection Challenge (DFDC) dataset [7] containing a total of 123,546 videos with the help of paid actors. Each video lasts about 10 seconds and consists of 300 frames. (3) The Celeb-DF dataset [6] containing 408 real videos and 795 synthesized video sequences with reduced visual artifacts, released for the advance of research on manipulated face detection. The examples of data are shown in Figure 7.Figure 7
Examples of data from FaceForensics++, DFDC, and Celeb-DF datasets.


### 5.1.3. Implementation Details

For the facial geometry detail generation, we acquire the detail displacement map using facial geometry detail generatorGd. For the neural network, we train the FD2Former of swin transformer block rather than pooling operation as the teacher. Then, we use the simplified patch embedding manifold loss to teach FD2Foremer of “MetaFormer” architecture with pretrained network. The Adamw optimizer is utilized for training with the initial learning rate of 5×10−4 and the warm-up learning rate is 5×10−7. The batch size is set to 64 and weight decay equals to 0.05, β1 = 0.9, β2 = 0.999. We use a cosine decay learning rate scheduler and 5 epochs of linear warm-up. In our implementation, the total epoch is 30.


## 5.1.1. Training Dataset

Faceforensics++ (FF++) [5] is a dataset released to standardize the evaluation of face forgery detection methods, which includes 1000 original videos and other 4000 manipulated videos. The manipulated videos are generated using four typical face swapping and reenactment methods, i.e., DeepFakes (DF) [51], FaceSwap (FS) [52], Face2Face (F2F) [20], and NeuralTextures (NT) [53]. Besides, there are three versions of FF++ in terms of compression level, i.e., raw, lightly compressed (HQ), and heavily compressed (LQ). Higher the compression level, the harder it is to distinguish the forgery traces. Since the uploaded manipulated videos always have a limited quality, we use the LQ versions in most experiments. We sample 270 frames for each train video and 100 frames for each test one.


## 5.1.2. Test Datasets

The following datasets are adopted for evaluation. (1) the testing set of FF++ as described. (2) DeepFakes Detection Challenge (DFDC) dataset [7] containing a total of 123,546 videos with the help of paid actors. Each video lasts about 10 seconds and consists of 300 frames. (3) The Celeb-DF dataset [6] containing 408 real videos and 795 synthesized video sequences with reduced visual artifacts, released for the advance of research on manipulated face detection. The examples of data are shown in Figure 7.Figure 7
Examples of data from FaceForensics++, DFDC, and Celeb-DF datasets.


## 5.1.3. Implementation Details

For the facial geometry detail generation, we acquire the detail displacement map using facial geometry detail generatorGd. For the neural network, we train the FD2Former of swin transformer block rather than pooling operation as the teacher. Then, we use the simplified patch embedding manifold loss to teach FD2Foremer of “MetaFormer” architecture with pretrained network. The Adamw optimizer is utilized for training with the initial learning rate of 5×10−4 and the warm-up learning rate is 5×10−7. The batch size is set to 64 and weight decay equals to 0.05, β1 = 0.9, β2 = 0.999. We use a cosine decay learning rate scheduler and 5 epochs of linear warm-up. In our implementation, the total epoch is 30.


## 5.2. Ablation Studies

For the analysis of the Forgery-Detection-with-Facial-Detail Transformer, facial geometry details and face image are regarded as complementary inputs and a transformer is utilized for the forged face detection. To evaluate each input, we quantitatively evaluate FD2Former with different inputs: a transformer with facial geometry details as input only, a transformer with cropped face image only, and a transformer with both original images and facial geometry details as input. The results are listed in Table 1.Table 1
Test results (%) of the FD2Former and its variants on FF++(LQ), DFDC and Celeb-DF. The “image” indicates the model with cropped face images as input only. The “detail” indicates the model with facial geometry details as input only. The “swin” indicates the FD2Former of the swin transformer backbone. The “meta” is the FD2Former of the MetaFormer backbone. The “manifold” and the “hard” mean the manifold distillation and the hard-label distillation respectively. The metric on FF++(LQ), DFDC and Celeb-DF dataset is ACC.
StructureFF++(LQ)DFDCCeleb-DFXception80.3285.6061.25Image(swin)81.1486.3278.13Detail(swin)78.0680.6876.94Img + detail(swin)83.2387.9783.51Img + detail + manifold(meta)82.7386.7281.36Img + detail + hard(meta)81.6786.0379.91The metric on FF++(LQ), DFDC and Celeb-DF dataset is ACC. Best results are shown in bold.First, the transformer only detects geometry details or cropped face achieves similar results. However, the model performs better with both face image and facial geometry details as input. Besides, the “MetaFormer” architecture achieves the similar and competitive results compared with the swin transformer architecture, but it has less learnable parameters, which saves training resources. Shown as Figure8, FD2Former(swin) almost saves half learnable params of CViT, while the accuracy only decreases by 0.53%.Figure 8
Comparison of the detection accuracy (%) and learnable params (M) with previous methods, Xception and CViT. The “FD2Former(swin)” is the FD2Former of the swin transformer backbone. The “FD2Former(meta)” is the FD2Former of the MetaFormer backbone. The experiment is conducted on DFDC dataset.We are also interested in the performance of the manifold learning strategy, compared with the hard-label distillation method. As shown in Table1, the manifold learning strategy is more efficient than the hard-label distillation strategy. The distilled FD2Former using our method outperform the model using hard-label distillation by 1.45% on Celeb-DF. The potential negative impact of the manifold learning strategy may be the increased consumption of computation resources and energy.


## 5.3. Analysis of the Fine-Grained Manifold Distillation

Aimed at patch-level knowledge distillation, we can insert the simplified manifold loss into any blocks under the condition of same patch numbers at corresponding blocks. As shown in Table2, the experiments are conducted to find the better insert location of the manifold loss. Results show that applying manifold loss at both the last stage and first stage improves the performance of student network better. We think it is because such a kind of insertion constrains the student network properly, while the student’s capacity could be limited if losses inserted in middle of the model.Table 2
Results (%) of distillation for different manifold computing location. All trained with the image and the displacement map as inputs on low-quality FF++.
TeacherStudentTeacher blocks extractedStudent blocks extractedACCDF2Former(swin)DF2Former(meta){11, 12}{23, 24}82.31DF2Former(swin)DF2Former(meta){1, 2}{1, 2}82.15DF2Former(swin)DF2Former(meta){1, 2, 11, 12}{1, 2, 23, 24}82.73DF2Former(swin)DF2Former(meta){1, 2, 3, 4, 11, 12}{1, 2, 5, 6, 23, 24}81.39All trained with the image and the displacement map as inputs on low-quality FF++. Best results are shown in bold.We test different values of hyper-parametersα and β, and presents results in Tables 3 and 4. The experiments show that assigning 1 and 0.2 to α and β improves the performance relatively. As Figures 9 and 10 shown, a small β could be more efficient because Lrs may partially coincide with Lcp, Lci terms, meanwhile the uncertainty of sample mechanism should be controlled.Table 3
Results (%) of distillation for a sequence of selectedα. All trained with the image and the displacement map as inputs on low quality FF++. The metric is ACC.
TeacherStudentαTesting data (ACC)FF++(LQ)DFDCDF2Former(swin)DF2Former(meta)0.581.7686.17DF2Former(swin)DF2Former(meta)182.7386.72DF2Former(swin)DF2Former(meta)1.582.4286.47DF2Former(swin)DF2Former(meta)282.1985.74DF2Former(swin)DF2Former(meta)2.581.7185.86The metric is ACC. Best results are shown in bold.Table 4
Results (%) of distillation for a sequence of selectedβ. All trained with the image and the displacement map as inputs on low-quality FF++. The metric is ACC.
TeacherStudentβTesting data (ACC)FF++(LQ)DFDCDF2Former(swin)DF2Former(meta)081.8585.43DF2Former(swin)DF2Former(meta)0.282.7386.72DF2Former(swin)DF2Former(meta)0.482.8685.77DF2Former(swin)DF2Former(meta)0.682.8386.58DF2Former(swin)DF2Former(meta)0.882.5186.27DF2Former(swin)DF2Former(meta)182.3786.08The metric is ACC. Best results are shown in bold.Figure 9
Accuracy (%) of the DF2Former(meta) when the hyper-parameter α varies. “(LQ)” refers to the heavily compressed FaceForensics++ dataset.Figure 10
Accuracy (%) of the DF2Former(meta) when the hyper-parameter β varies. “(LQ)” refers to the heavily compressed FaceForensics++ dataset.We also test the hyper-parameterk, which means the number of randomly selected patches for calculation of the sampling correction, random sampler loss Lrs. As Table 5 shows, five experiments are launched to search for an appropriate value of k, and we finally assign 256 to k for a better performance. As indicated in Figure 11, a bigger k usually leads to a better performance.Table 5
Results (%) of distillation for a sequence of selectedk. All trained with the image and the displacement map as inputs on low quality FF++. The metric is ACC.
TeacherStudentkTesting data (ACC)FF++(LQ)DFDCDF2Former(swin)DF2Former(meta)081.8585.43DF2Former(swin)DF2Former(meta)6482.4185.71DF2Former(swin)DF2Former(meta)12882.2786.54DF2Former(swin)DF2Former(meta)19282.6886.37DF2Former(swin)DF2Former(meta)25682.7386.72The metric is ACC. Best results are shown in bold.Figure 11
Accuracy (%) of the DF2Former(meta) when the hyper-parameter k varies. “(LQ)” refers to the heavily compressed FaceForensics++ dataset.


## 5.4. Comparison with Other Methods

Some previous works have shown the generalization performance problem if confronted with the unseen manipulation methods. In this section, the proposed method is compared with previous ones, Xception [54] and Face X-ray [55], to explore the performance during detecting unseen manipulation methods or datasets.Following Luo et al. [56], the evaluations on different unseen manipulation methods are conducted on the FF++(HQ) [5] database, and we compare the performance with previous methods. The proposed method is trained on F2F, DF, FS, and NT separately and tested on the remaining methods, taking the AUC as the evaluation metric. Table 6 presents the experimental results. Compared to the classic method XceptionNet, the proposed FD2Former achieves a significant improvement most of time, since the former overly relies on the texture patterns in RGB space without thinking unseen details. Face X-ray [55], achieves a better generalization performance benefitting from the blending evidence detection. The FD2Former leverages both textures and highlighted clues extracted from the facial geometry details, which probably why it generalizes better from one method to another.Table 6
Generalization evaluation (%) with previous methods on HQ (c23) FF++. AUC is used as metric of performance on the unseen manipulation technique. The highlighted are best results.
Training dataModelTesting data (AUC)DFF2FFSNTDFXception [54]99.373.649.073.6Face X-ray [55]98.763.360.069.8FD2Former98.9469.1759.5877.39F2FXception [54]80.399.476.269.6Face X-ray [55]63.098.493.894.5FD2Former81.7898.1277.3189.52FSXception [54]66.488.899.471.3Face X-ray [55]45.896.198.195.7FD2Former70.2197.8399.2793.12NTXception [54]79.981.373.199.1Face X-ray [55]70.591.791.092.5FD2Former80.4390.8292.5394.62AUC is used as metric of performance on the unseen manipulation technique. Best results are shown in bold.Following [57] Khodabakhsh et al. [58], the generalization performance is analyzed quantitatively on unseen data and compared with other methods, including the classic Xception [54], the ensemble of EfficientNet’s variants [59], and the Face X-ray [55]. The models are trained on FF++ (HQ) [5] and evaluated on Celeb-DF [6], DFDC [7], respectively. Such a generalization problem may confront more challenges than the experiments within FF++. We can see from Table 7 that our method achieves apparent improvements over EfficientNetB4 Ensemble [59] and Xception [54], indicating that our model learns more robust representations than the previous methods. Figure 12 displays the obtained result of some tested face images.Table 7
Generalization evaluation (%) on the unseen dataset, Celeb-DF, and DFDC. The highlighted are best results.
ModelTrainingTesting AUCCeleb-DFDFDCEfficientNetB4Ensemble [59]FF++55.863.0Xception [54]59.467.9Face X-ray [55]74.270.0FD2Former79.273.6Generalization evaluation (%) on the unseen dataset, Celeb-DF, and DFDC. Best results are shown in bold.Figure 12
Obtained result of some tested face images for DeepFake detection with the estimated category and the probabilities of each one. (a) FaceForensics++. (b) DFDC. (c) Celeb-DF.
(a)(b)(c)


## 6. Conclusion

In this paper, we propose a novel approach for the detection of face manipulation by reconstructing 3D face. We find the facial geometry details through the 3D coarse reconstruction and detail reconstruction. We use the displacement map to amplify the complex artifact patterns. The clues in the geometry details and the cropped face images are fed into FD2Former to classify whether the input face is real or not. Meanwhile, for the capabilities of many edge devices such as smartphones and IoTs, we use MetaFormer architecture to build a light neural network and introduce a manifold learning strategy to improve the performance of our method. The comprehensive experiments on FaceForencis++, Celeb-DF, and DFDC exhibit the effectiveness and generalization of our FD2Former. On the whole, our work presents a novel direction to detect the face manipulation clues through 3D face reconstruction.



---

*Source: 9278715-2022-09-29.xml*