Video Forgery Detection Using Multiple Cues on Fusion of EfficientNet and Swin Transformer
Authors: Chenyu Liu
Jia Li
Junxian Duan
Huaibo Huang
Venue: ICLR 2022 Submitted (2022)
Keywords: deepfakes
video forgery detection
high-frequency
texture
optical flow
EfficientNet
Swin Transformer

Abstract: The rapid development of video processing technology makes it easy for people to forge videos without leaving visual artifacts. The spread of forged videos may lead to moral and legal consequences and pose a potential threat to people's lives and social stability. So it is significant to identify deepfake video information. Although the previous detection methods have achieved high accuracy, the generalization is poor when facing unprecedented data in the real scene. There are three fundamental reasons. The first is that capturing the general clue of artifacts is difficult. The second is that selecting the appropriate model is challenging in specific feature extraction. The third is that exploiting fully and effectively the extracted features is hard. We find that the high-frequency information in the image and the texture in the shallow layer of the model expose the subtle artifacts. The optical flow of the real video has variations while the optical flow of the deepfake video has rarely variations. Furthermore, consecutive frames in the real video have temporal consistency. In this paper, we propose a dual-branch video forgery detection model named ENST, which integrates parallelly and interactively EfficientNet-B5 and Swin Transformer. Specifically, EfficientNet-B5 extracts the artifacts information of high frequency and texture in the shallow layer of the model. Swin Transformer captures the subtle discrepancies between optical flows. To extract more robust face features, we design a new loss function for EfficientNet-B5. In addition, we also introduce the attention mechanism into EfficientNet-B5 to enhance the extracted features. We conduct test experiments on FaceForensics++ and Celeb-DF (v2) datasets, and comprehensive results show that ENST has higher accuracy and generalization, which is superior to the most advanced methods.

Under review as a conference paper at ICLR 2022
VIDEO FORGERY DETECTION USING MULTIPLE CUES ON FUSION OF EFFICIENTNET AND SWIN TRANS- FORMER
Anonymous authors Paper under double-blind review
ABSTRACT
The rapid development video processing technology makes it easy for people forge videos without leaving visual artifacts. The spread of forged videos may lead to moral and legal consequences and pose a potential threat to people's lives and social stability. So it is significant to identify deepfake video information. Although the previous detection methods have achieved high accuracy, the gen- eralization is poor when facing unprecedented data in the real scene. There are three fundamental reasons. The first is that capturing the general clue of artifacts is difficult. The second is that selecting the appropriate model is challenging specific feature extraction. The third is that exploiting fully and effectively the the
extracted features hard. We find that high-frequency information image and the texture in the shallow layer of the model expose the subtle arti- facts. The optical flow of the real video has variations while the optical flow the deepfake video has rarely variations. Furthermore, consecutive frames in the real video have temporal consistency. In this paper, we propose a dual-branch video forgery detection model named ENST, which integrates parallelly and in- teractively EfficientNet-B5 and Swin Transformer. Specifically, EfficientNet-BS extracts the artifacts information of high frequency and texture in the shallow layer of the model. Swin Transformer captures the subtle discrepancies between opti- cal flows. To extract more robust face features, for EfficientNet-BB B5. In addition, we also introduce the attention mechanism into EfficientNet-B5 to enhance the extracted features. We conduct test experiments FaceForensics** and Celeb-DF (v2) datasets, and comprehensive results show that ENST has higher accuracy and generalization, which is superior to the most advanced methods.
design new loss function
Index Terms: deepfakes, video forgery detection, high frequency, texture, optical flow, EfficientNet, Swin Transformer
INTRODUCTION
Video synthesis techniques based on deep learning Karras et (2017 2018): Thies al. (2019) have reduced the threshold for people to manipulate videos, and the high-quality deepfake videos that are indistinguishable to the human eye are increasingly available Deeptakes (2020); Faceswap 2019); Fakeapp (2020). Although this technology has been applied to Cineflex, Crytek, and driver- less technology, which has had positive effect on society. Inevitably, is illegally abused by some malicious users to forge indistinguishable false information, which seriously affects the level of public trust in the government and society, causes irreparable social problems, and can even pose serious threat to politics.
Many deepfake detection methods Wang et al. (2019) Kim al. 2021 Jeon et al. (202 have been proposed to address this problem and significant progress has been made. Previous works explore forgery patterns based on manual features Ferrara et al. (002 Pan et al. (2012): Cozzolino et al. (2015) and then classify them by zooming images. While most of the recent learning- bearning-based approaches Cozzolino define the problem as binary classification of image video with process of first extracting
the subtle differences between the real and forged
CogroaachssCzzzolinocc et al. (2017) (2017a)

---

Under review as a conference paper at ICLR 2022
the global features of the image using a neural network and then feeding them to classifier to distinguish between genuine and faked images.
However, with the increasing refinement of deep generative methods, the generated forged videos are becoming increasingly difficult to discriminate, especially some low-quality ones with only a few subtle visual artifacts. Most existing detection methods still focus on specific artifacts, leading to significant challenges in the generalization and effectiveness of detection systems in realistic scenarios. This creates an urgent need to discover subtle cues that are more critical for detection, and then to extract and incorporate appropriate modules with efficient backbone networks to enhance these artifact features and combine them effectively for forgery detection.
After systematic analysis, we find three fundamental reasons for the low generalization of the model. The first is the difficulty of capturing general artifact cues and the limitations of the dataset in terms of quantity and quality. The second is the limitations of selecting suitable network model for specific feature extraction. The third is the limitations of fully and effectively utilizing the extracted features. In terms of feature cues, we find that although the generated videos are realistic with few artifacts in the texture, the frequency domain, especially at high frequency, still reveals significant differences, even for low-quality videos, and many forgery methods are designed to generate videos that are slightly defective to human vision, with few methods targeting the frequency domain. In addition, it is found that texture information located at the shallow level of the detection network rather than the higher-level semantic information is particularly critical for detection. To eliminate convert the images from RGB
the effect of luminance in RGB texture feature extraction, to YCbCr and remove the luminance (Y) channel, and combine Cb and Cr channels with high frequencies. In addition, we find characteristic that the optical flow of the real video changes significantly while the optical flow of the deepfake video has little change, and coupled with the temporal consistency of the continuous frames of the real video, we choose the high frequency, texture, and optical flow as the feature cues for video forgery detection.
For features such as high frequency and texture, which require attention local details, we choose EfficientNet-B5 Tan & Le| (2019). a CNN class network that is superior in local feature extrac- tion, as one of the backbone networks, which has higher accuracy and performance compared with other CNNs. For features such as optical flow, which reflect the inter-frame temporality, we choose Swin Transformer Liu et al. (021)) a Transformer class network that is superior in global repre- sentation, another backbone network. which applies to all scales of images, simple and flexible in computation, and shows excellent performance compared with other Transformers. In addition, we introduce the attention mechanism and loss function for EfficientNet-B5 to extract more robust face features.
In summary, in this paper, we propose novel multiple-cue video forgery detection model ENST using a combination of high frequency, low-level texture, and optical flow cues, based on the fusion of EfficiintNet-BB and Swin Transformer, which fully integrates multiple key features and uses an advanced backbone network for extraction. To demonstrate the effectiveness of ENST, we conduct extensive experiments on FaceForensicst+ Rssler et al. (2019) and Celeb-DF (v2) Li et al. (019). as well as a comparison of various settings in an ablation study. Comprehensive experiments indicate that ENST has excellent competitive properties.
The contributions of this paper are as follows:
1. We investigate the current video forgery detection methods in-depth and systematically, and find three reasons for the poor generalization ability of these methods, as well as discover several artifact cues in forgery detection, and use these cues to design an efficient network structure for the above reasons.
We propose novel network structure ENST, which can utilize the local feature extraction ability of EfficientNet-B5 and the global relationship sensing ability of Swin Transformer, and integrate them to complement their abilities during data processing. In addition, we introduce the attention module and a new loss function for EfficientNet-B5 to enhance the ability to extract features.
3. Our experiments on FaceFoorensics*+ and Celeb-DF (v2) demonstrate that ENST achieves supe- rior classification performance and generalization compared to other state-of- ooffthe-att art methods.
The rest of this paper is organized as follows. Section 2 presents related work in the field of video forgery detection and describes the motivation for the proposed ENST. Section 3 details the specifics
2

---

Under review as a conference paper at ICLR 2022
of the proposed model. Section 4 describes the experimental results and analysis. Section 5 con- cludes this paper.
RELATED WORK
2.1 VIDEO FAKE CUES
In recent years, various deep learning forgery methods have produced synthetic images or videos of faces Suwajanakorn al. (2017) 2015) that have caused some degree of harm society. The problem of deep forgery detection has received extensive attention in the vision field, and many forgery detection methods have been proposed detect such forged products. Some methods focus on mining specific artifacts generated by the forgery process, such color space Li et al. (2018); McCloskey & Albright (2018) and shape cues. Many deep learning methods Marra et al. (2018); Yu al. (2018); et al. (2018) use deep neural networks to extract high-level semantic information from the spatial domain and then classify a given image or video. Some methods convert the image from the spatial domain to the frequency domain Stuchi et al. (2017); Franzen (2018), capturing some information that is valuable for forgery detection. Stuchi et al. (017)) employ a fixed set of filters to extract different ranges of frequency information and then adopt fully connected layer to obtain classification results. Durall et al. (2019) extract the frequency domain information using the DFT transform and average the amplitudes of the different frequency bands. Other methods extract statistical features, such as capturing features from spatial textures Pan et al. (2.09 | (201 and transform domain coefficient distributions Chen al. (2009); Lyu & Farid (2005). Video detec- tion methods can discover differences in the temporal dimension of video frames Sabir et al. (2019);
Conotter
Guera & Delp 2018), such as Waag & Dantcheva (2020) using spatial and motion information, and CChntha et al.j (2020); Amerini et al. (2020); Caldelli et al. (2021) using optical flow for video forgery detection. While the above methods are limited to specific cues, we take full advantage of
these cues.
2.2 DEEPFAKE DETECTION STRUCTURES
Many deep learning models have been proposed to extract features and discover forgery patterns directly without targeting a specific forgery cue or artifact, classify real and fake images or videos, and implement deep forgery detection created by random methods. EfficientNet (Tan & Le (2019) achieves higher accuracy with larger network by combining a composite model expansion method with neural structure search techniques to optimize the dimensions of network depth, network width, and input image resolution, which has higher accuracy and performance compared to other advanced CNNs. Each layer of Swin Transformer computes self-attention within the window of the transform, enabling cross-window connectivity for all scales of the image. Swin Transformer is flexible and efficient, with only linear complexity for image size. Convolution module in CNN is good at extract- ing local details, and as the layers get deeper, the attention field will be wider and more suitable for image processing. But capture global information, detection model often requires stacking many convolutional layers. While attention in Transformer is good at grasping the whole, but requires a large amount of data for training. Conformer (Peng al. (202T) uses parallel and complementary approach, combining both CNN and Transformer branches to preserve the features extracted by both branches, to achieve performance beyond CNN and ViT Dosovitskiy al, (2020) with comparable parameter complexity.
PROPOSED METHOD
In this section, outline the principles and details of the different parts of the proposed method according to the data processing process.
It is challenging for most methods to extract sufficiently comprehensive features with suitable net- work models and then integrate them effectively for forgery detection. Inspired by Conformer Peng et al. (2021), we choose two models EfficientNet-B5 and Swin Transformer with more efficient performance among CNNs and Transformers, as baselines and combine them for video forgery de- tection.
3

---

Under review as a conference paper at ICLR 2022
EfficientNet-B5 extracts local features of frames, such high frequency features and texture fea- tures by using the characteristic of focusing well on local features. Swin Transformer extracts arti- fact features from the optical flow map between frames obtain the global representation of frames, where the optical flow features responding to the temporal consistency between frames are used.
We design parallel interactive deep forgery detection architecture ENST to learn potential repre- sentation that can distinguish between real and forged faces, and the pipeline ENST is shown in Figure
Lustrosr
EfficientNet 85
High ww..freguencyyyy
Swin Trarffomer--
Feature Maps
Softmax
Real/Fake
Video
Estracted Faces
14h
CaCr
frames feetures
Patch Embeddings
Optical
Feesture Interaction Module
Figure 1: The video forgery detection pipeline of proposed ENST, where Swin Transformer-A rep- resents the first three stages in Swin Transformer, Swin Transformer-B represents the last stage in Swin Transformer, and D represents DCT operation.
3.1 PRE-PROCESSING
We first extract the frames of the input video, and then use the Multitask Cascaded CNNs (MTCNN) Zhang al. (2016) detect and extract the faces present in each frame, crop the faces of each frame, then resize to 224x224 pixels. Moreover, we normalize them to zero mean and unit variance, and finally obtain the extracted faces. After that, first operate single frame in the video, and the face in frame is input to two branches after the basic feature extraction process.
3.2 EFFICIENTNET-BR B5 BRANCH
We exploit the structurally improved EfficientNet-B5 branch to extract the deepfake information in the high frequency and texture, and enhance it to finally obtain the features of fully connected linear layer. We introduce the specific details the following subsections.
3.2.1 FEATURE
The motion process of faces complex, and most forgery methods do not consider sufficiently precise global constraints during forgery, which can easily lead to missing facial texture details of the generated faces and inconsistencies among features diverse domains.
As the skin tones in RGB images faces from RGB to YCrCb by separating and removing the channel which denotes luminance.
influenced by the luminance, convert the extracted
In addition, considering the important effect of high frequency forgery detection, convert the image from the spatial domain to the frequency domain by DCT operation, and then use high-pass filter to extract the high frequency component, which was merged with the two channels of Cr and Ch to form a dimeesiinaiionall 224x224x3 feature tensor. Then it was fed into EfficientNet-BB to extract the detail components in high frequency and the fine artifacts of the shallow texture.

---

Under review as a conference paper at ICLR 2022
3.2.2 LOSS
We assume that real and fake faces in videos have distinguishable feature distributions, and samples of different classes are clustered together. To extract robust face features and to distinguish the video distributions real and fake faces, instead of using the more common softmax and cross- cross-entropy, we combine softmax loss, additive angular margin loss (ArcFace) Deng et al. (2018) and single-center loss (SCL) Li et al. (202T) as the loss function of EfficientNet-B3 for extracting features. ArcFace and SCI. have some similarities, both are used to compress intra-class tightness and enhance inter- class diversity, And we combine these objective functions to improve the accuracy of the extracted features.
AreFace
ArcFace is an improvement on Spherefacce Liu et al. (0117) in terms of feature vector normalization and additive angular spacing, forcing a boundary between the distance a sample to its class center and the distance of sample other class centers in corner space. It improves interclass separa- bility, and also enhancing intraclass tightness and interclass diversity, allowing the model to learn features that are highly distinguishable between real and fake faces features, resulting in more robust classification for forgery detection. ArcFace loss is defined as:
1 N
e (coss +m²)
Arefacee
log (c-(coo +m])
(1)
Lj=1j es-cos
where N is the batch size, is the class number, is an additive angular margin penalty between I; and W fj is the angle between the weight W, and the feature IG Wj is the j-th column of the weight W, and is the radius of hypersphere where the learned embedding features are thus distributed on.
Single-center Loss (SCL)
The purpose of SCL is to minimize the distance from the real face to the center point while maxi- mizing the distance from the fake face to the center point, that the network can learn subtle forgery information and reduce the optimization difficulty. SCL is defined as:
Lsc Mnot + maxt Mman mv D, D),
where Mnat is the average Euclidean distance of the true face representation from the center point C, and Mman is the average Euclidean distance of the false face representation from the center point C. The Euclidean distance m is related to the arithmetic square root of the feature dimension D. To facilitate the setting of the hyper-parameter m, the boundary is designed here as mvD.
In addition, considering that SCL is based on mini-batch samples and focuses directly on feature representation, while softmax loss focus the global and how map the feature represen- tation to the discrete label space. So, we use the global information retained by softmax loss to guide the update of center points SCL and increase the robustness of training.
In general, to reconcile the local feature representation and the global update, we use a combination of three loss functions, The total loss function is defined as:
Taotal Togoftmaz + a Areface + BTsees
where a and B are hyper-parameters weights that adjust the balance between foot Arefacee and Lee- They are used to provide a relatively efficient and flexible total loss function.
3.2.3 ATTENTION
The attention mechanism allows models to focus on and fully learn crucial information, and is widely adopted module in models in natural language processing and computer vision et al. Huang et al, Vaswani et al. (2017).
2017
We take EfficientNet-BB B5 as the extraction model for artifact features in high frequency and low layer textures. To focus more on the artifacts in the feature maps, inspired by Bonettini et al. (2020), we insert the attention module between the MBConv layers of EfficientNet-B5, shown Figure 2
5

---

Under review as a conference paper at ICLR 2022
x5
x7
x7
724 x222333
Conv 3 3
1128112/48
MBConv1
112x112224
MBConv6 3
55-56
MBConv6 5
28-22 26
MBConv6 3 3
14*14x1228
MBConv6 5
14x14x177
MBConv6 5
7 304
MBConv6 3
7
7x7x532
Pooling &
2048
1
Texture
EfficientNetB85
132x 12-x112222 22
Conv x
112x0124
112 112 X24
Sigmoid
Attention
Figure 2: Attention module is introduced into MBConv interlayers in EfficientNet-B5.
Figure Souureeshen shows the effect of adding only one attention module. In the practical experiment, we add the attention module sequentially from front to back between layers to compare the effect of the attention mechanism on the overall detection performance of the model. First, the high frequency and texture information of the frames are combined and fed to EfficientNet-BS. Forging the images will result in overlapping areas of high frequency and texture information, which can be guide to the extraction of texture information. In addition, the attention module notice these subtle artifacts in the shallow layers. The final output is an enhanced fusion feature high frequencies and textures.
3.3 SWIN TRANSFORMER BRANCH
Considering that the optical flow of the real video changes, while the optical flow of the faked video rarely changes. In addition, the distance between parts of the face and the head motion vary continuously and regularly between frames of the real video, while most forgery methods seldom concern to ensure the continuity between frames of the deep forgery video by effective modeling, which leads to the temporal inconsistency of the consecutive frames.
To exploit this temporal variation difference, we first divide the video into O~ consecutive frames, and extract the optical flow of frame and frame i 4 1 as the optical flow of frame by using PWC- Net|Sun et al. (2018). Then we input them into Swin Transformer-A to obtain the patch embeddings of the middle layer, where Swin Transformer-A represents the first three stages in Swin Transformer.
3.4 FEATURE INTERACTION MODULE
We introduce the feature of complementary fusion by adding feature interaction module. In order solve the problem of mismatching the size of the feature map in the Efficient-BS branch and the patch embeddings in the Swin Transformer branch, we use a special conversion operation, in which a 1xl convolution used align the dimensionality of the feature map with the number of channels of the patch embeddings. Then, the spatial dimensions are aligned by the downsampled module. Finally, the feature map are added to the patch embedding.
In addition, the global context is gradually exported from the Swin Transformer branch to the feature map in Efficient-B5 enhance the global perception capability of the Efficient- -B5 branch. We adopt similar operation by first upsampling the patch embedding align the spatial scales. Then lxl convolution used to align the number channels of the patch embeddings with the dimensionality of the feature map.
Moreover, considering that the forgery operation causes changes in various aspects of the forgery region, the same regions of different features extracted have definite similarities, so this parallel and complementary fusion of the two branches also enables the two branches to guide each other and focus the local artifact that is instrumental for forgery detection.

---

Under review as a conference paper at ICLR 2022
3.5 FEATURE CONCATENATION
This part is the combination of all face region features of frame extracted from two branches connected to form feature concatenation of frame which includes the extracted high frequency, texture features, and patch embeddings.
The above operations are performed for each frame in sequence to obtain the feature concatenations of the O~N frames of the videos. After resizing, they are transformed into individual patches, which are combined into patches, and then transformed into a new patch embedding, and then input it to Swin Transformer-B, then followed by the linear layer and softmax layer, finally output the classification prediction of the whole video, where Swin Transformer- B represents the last stage in Swin Transformer. It is worth mentioning that, after "i-th frame" and before "All frames features" in Figure represent the operation process of the i-th frame alone, and the rest of Figure reppesents the operation process of all frames.
EXPERIMENTS
4.1 DATASETS
In this paper, we use Faceforensics** to train and verify the performance of our proposed model ENST, and Celeb-DF (v2) to verify the generalization of the model.
4.1.1 FACEFORENSICS+#
FaceForensics++ facial forgery video dataset, including original videos and videos generated by the four most representative face manipulation methods (Face2Face, FaceSwap, DeepFakes and NeuralTextures), each category has 1000 forgery videos, and the extracted frames contain more than 500,000 forged images in each category. According different compression levels, the com- pression factors can be classified as cO(raw), (23(mi)) and c40(high). As with the setup in FaceForensics++ we split the dataset each category into fixed train set, validation set and test set containing 720, 140 and 140 videos, respectively. In the following experiments, we use two compression levels of (meddim) and c40(high) for the data. By default, the metric are Acc and AUC.
4.1.2 CELEB-DF (v2)
Celeb-DF (v2) contains both real and fake videos from Youtube. Unlike the FaceForensics**, the fake videos in Celeb-DF (v2) further segmented handle issues such as color inconsistency and low-frequency smoothing, and have online playback-level high-resolution pictures quality, Celeb- DF (v1) contains 795 fake videos. Celeb-DF (v2) is an extension of Celeb-DF (yl) and includes 590 original videos collected from YouTube with different age, race, and gender themes, and 5639 corresponding fake videos. Celeb-DF (v2) has more diverse and challenging data, which is beneficial to verify the generalization of the proposed method.
4.2 SETTINGS
Based on the parameters of Conformer by default, combine the parameters in both branches of EfficiemtNet-B5. and Swin Transformer. ENST uses an optimizer of AdamW. The momentum is 1.0. The weight decay factor is 0.05. The batch size is 256. The epoch is 300. The initial learning rate is 0.001. The LR scheduler is decaying with a cosine schedule.
4.3 COMPARISON METHOOSS
We compare the proposed method ENST with previous high-performance video forgery detection models on FaceForensicst+, and the results are shown in Table [ Comparing with some advanced methods, find that although c40 reduces the AUC by 6.50 and the accuracy by 10.57 compared to c23, still outperforms other methods. We analyze that although the compression loses the details of image texture, the extracted high-frequency information and the optical flow information between
7

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of the results of different models the two quality types of FaceForensics*
c40
c23
Model
AUC
Ace
AUC
Acc
MesoNet Afchar et al. (2018)
70.47
83.10
Face X-ray Li et al. (2020)
61.60
87.40
Xception Choller (201
89.30
86.86
96.30
95.73
Two-branch Masi et al. (2020)
86.59
98.70
Efficient-B4 B4 Zhao et al. 2021
90.40
88.69
97.60
99.29
ENST(Ours)
91.12
88.73
97.62
99.30
video frames have excellent robustness to the compressed information, which can well resist the compression interference and maintain the robustness of the model.
Table 2: Comparison of generalization results of different models, which train on the FaceForen-
sics++ train set and test on Celeb-DF (v2) (AUC(%)).
Model
FF++
Celeb-DF (v2)
Two-stream Zhou et al. (2018)
70.10
53.80
Mesolnception Afchar et (2018)
83.00
53.60
Two Branch Masi et al. (2020)
93.18
73.41
Efficient-Be B4 Zhao et al.
(021
99.8
67.44 44
ENST(Ours)
99.8
67.51
In order verify whether ENST has superior generalization in realistic scenarios that are in- cluded in the train set, we put the model trained on FaceForensicst+ on Celeb-DF (v2) for experi- ments on transfer learning, we use the fine-tuning model, i.e., freeze parts of the convolutional layers of the pre-trained model, train the remaining convolutional layers, and after the model converges, reduce its learning rate and fine-tune the layers. We randomly select 40 samples in Celeb-DF (v2) with FaceForensics++ to train the model to prevent catastrophic forgetting, and the results are shown in Table The experimental results show that ENST has a better transfer capability and the ability to learn types of samples. It shows superior generalization new types of data.
4.4 ABLATION STUDY
To verify the effectiveness of each part of ENST, we conduct an ablation study and evaluate the network FaceForensicst+.
4.4.1
COMPARISON OF LOSS
Table 3: Comparison of results using different loss functions in Efficiem[Net-B5 on c23 the Face- Forensics++.
Loss Function
AUC
Acc
Arcface
88.15
84.48
SCL
92.27
90.64
Softmax Areface
91.52
87,86
Softmax + SCL
95.46
96,93
Softmax+ Areface SCL
98.67
97.65
On FaceFForensics*+, we compare the experimental results of using different loss functions when extracting features in EfficienNNet-B5. as shown in Table 3. First, we compare the classification results of Arcface and SCL individually, after which softmax is added to guide the improvement of the global classification. Finally the three losses are combined. The experimental results indicate that the highest accuracy is achieved when softmx, Areface, and SCL are integratively used.
8

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of results the FaceForensics** with the addition of the attention module different stages of EfficientNet-B. where v 'indicates the addition of the attention module in stage of EfficientNettBS. Stage indicates after the first MBConv. Only the first five of the seven listed here.
V
Stage
3
5
AUC
96.35
97.82
98.53
98.67
98.55
95.63
Acc
94.65
96.07
96.53
97.65
94.49
89.48
4.4.2 COMPARSION OF ATTENTION
On Faceesornsics we start from the previous of the EfficientNet-B5 structure and add attention modules at different stages, and the experimental results are shown in Table which shows that the accuracy increases gradually with the addition of attention modules. However, when continuing to add at the 4th stage, the accuracy has slightly decreased, and when continuing to join in the 5th stage, the accuracy rate shows a significant decrease, which is consistent with our speculation. Our anal- ysis suggests that the artifacts are mainly concentrated in the shallow network of EfficientNet-BS. and the shallow feature extraction cation network to classify, while the deep feature extraction is little significance. Furthermore, the increase of the attention module will increase the computational volume and affect the overall performance improvement.
provide enough critical information for the whole classifi-
CONCLUSION
In this paper, we systematically analyze the reasons for the low generalization of the current forgery video detection system and discover some subtle but critical cue features in forgery detection. In or- der combine the discovered cues and improve the generalization problem, we propose network structure ENST with parallel interaction fusion of two branches, EfficientNet-B5 and Swin Trans- former. We also introduce the attention mechanism and add new loss function to EfficientNet-B5 to further focus and enhance the artifact features and extract more robust face features. Com- prehensive experiments indicate that ENST has better performance and generalization than other advanced methods.
REFERENCES
Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: a compact facial video forgery detection network. 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1-7, 2018.
I. Amerini, L. Galteri, R. Caldelli, and A. D. Bimbo. Deepfake video detection through optical flow based cnn. In 2019 JEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2020.
N. Bonettini, E. D. Cannas, S. Mandelli, L. Bondi, P. Bestagini, and S. Tubaro. Video face manipu- lation detection through ensemble of cnns. 2020.
R. Caldelli, L. Galteri, I. Amerini, and A. D. Bimbo. Optical flow based enn for detection of unlearnt deepfake manipulations. Pattern Recognition Letters, (10), 2021.
D. Chen, J. Li, S. Wang, and S. Li. Identifying computer generated and digital camera images using fractional lower order moments. In IEEE Conference on Industrial Electronics Applications, 2009.
Akash Chintha, Aishwarya Rao, Saniat Sohrawardi, Kartavya Bhatt, and Raymond Ptucha. Lever- aging edges and optical flow on faces for deepfake detection. 2020 IEEE International Joint Conference Biometrics (IJB), 2020.

---

Under review as a conference paper at ICLR 2022
F. Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2017a.
F. Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2017b.
V. Conotter. Detecting photographic and computer generated composites. In International Confer- ence on Electronic Mechanical Engineering Information Technology. 2011.
D. Cozzolino, D. Gragnaniello, and L. Verdoliva. Image forgery localization through the fusion of camera-based, feature-based and pixel-based techniques. In 2014 IEEE International Conference on Image Processing (ICIP), 2015.
D. Cozzolino, G. Poggi, and L. Verdoliva. Recasting residual-based local descriptors as convolu- tional neural networks: an application image forgery detection. In the 5th ACM Workshop, 2017.
Deepfakes. http://ghhhb..cmme/eephh/hhhaanp 2020.
Deng, J. Guo, and S. Zafeiriou. Areface: Additive angular margin loss for deep face recognition. 2018.
A. Dosovitskiy, Beyer, A. Kolesnikov, D. Weissenborn, and N. Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. 2020.
R. Durall, M. Keuper, F. J. Pfreundt, and J. Keuper. Unmasking deepfakes with simple features. 2019.
Faceswap. Fakeapp. http::/wwww.aaaavvddacccceeeettt 2020.
http://ghhhb.ooo//cccccowww//s/h 2019.
Ferrara, Bianchi, De Rosa, and A. Piva. Image forgery localization via fine-grained analysis of cfa artifacts. IEEE Transactions on Information Forensics Security, 765):1666-1577, 2012.
Franzen. Image Classification in the Frequency Domain with Neural Networks and Absolute Value DCT. Image and Signal Processing, 2018.
D. Guera and E. J. Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2018.
Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, Cenet: Criss-cross attention for semantic segmentation. International Conference on Computer Vision.
H. Jeon, Y. Bang, and S. S. Woo. Fdftnet: Facing off fake images using fake detection fine-tuning network. 2020.
H. Jie, S. Li, S. Gang, and S. Albanie. Squceze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99), 2017.
Karras, T. Aila, S. Laine, and Lehtinen. Progressive growing of gans for improved quality, stability, and variation. 2017.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
Minha Kim, Shahroz Tariq, and Simon S. Woo. Cored: Generalizing fake media detection with continual representation using distillation, 2021.
H. Li. B. Li. S. Tan, and J. Huang. Detection of deep network generated images using disparities in color components. 2018.
Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, and Yongdong Zhang. Frequency-avee aware discriminative feature learning supervised by single-center loss for face forgery detection, 2021.
10

---

Under review as a conference paper at ICLR 2022
L. Li, J. Bao, T. Zhang, H. Yang, and B. Guo. Face x-ray for more general face forgery detection. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu. Celeb-df: A new dataset for deepfake forensics. 2019.
W. Liu, Wen, 7. Yu, M. Li, B. Raj, and I.. Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. 2021.
Lyu and H. Farid. How realistic is photorealistic? IEEE Transactions on Signal Processing, 53 (2)8845-850, 2005.
F. Marra, D. Gragnaniello, D. Cozzolino, and L. Verdoliva. Detection of gan-generated fake im- ages over social networks. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2018.
I. Masi, A. Killekar, R. M. Mascarenhas, S. P. Gurudatt, and W. Abdalmageed. Two-Branch Recur- rent Network for Isolating Deepfakes in Videos. Computer Vision - ECCV 2020, 2020.
Scott McCloskey and Michael Albright. Detecting gan-generated imagery using color cues, 2018.
F Pan, J. B. Chen, and J. W. Huang. Discriminating between photorealistic computer graphics and natural images using fractal geometry. Science China Series F: Information Sciences, 52(002): 329-337, 2009.
X. Pan, Z. Xing, and S. Lyu. Exposing image splicing with inconsistent local noise variances. In IEEE International Conference on Computational Photography, 2012.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye.
Conformer: Local features coupling global representations for visual recognition, 2021.
Andreas Rssler, D. Cozzolino, L.. Verdoliva, C. Riess, Thies, and Matthias Niener. Faceforen-
sics++: Learning to detect manipulated facial images. 2019.
E. Sabir, J. Cheng, A. Jaiswal, W. Abdalmageed, and P. Natarajan. Recurrent convolutional strategies for face manipulation detection in videos. 2019.
A. Stuchi, M. A. Angeloni, R. F. Pereira, Boccato, and Rrf Attux. Improving image classi- fication with frequency domain layers for feature extraction. In 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), 2017.
D. Sun, X. Yang, M. Liu, and J. Kautz. Pwe-net: Cuns for optical flow using pyramid, warping, and cost volume. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
S. Suwajanakorn, S. M. Seitz, and I. Kemellaccer--Shiizemee What makes tom hanks look like tom hanks. In IEEE International Conference on Computer Vision, 2015.
Supasorn Suwajanakorn, Steven M. Seitz, and Ira Keneemmacher&SShizzerma. Synthesizing obama: Learning lip sync from audio. ACM Transactions on Graphics, 36(4CD).95.1-95.33 2017.
M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. 2019.
Justus Thies, Michael Zollhfer, and Matthias Niener. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions Graphics, 2019.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv, 2017.
Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros. Cnn-generated images are surprisingly easy spot... for now. 2019.
11

---

Under review as a conference paper at ICLR 2022
Y. Wang and A. Dantcheva. A video is worth more than 1000 lies. comparing 3denn approaches for detecting deepfakes. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), 2020.
N. Yu, L. Davis, and M. Fritz. Attributing fake images to gans: Learning and analyzing gan finger- prints. 2018.
K. Zhang, Z. Zhang. Z. Li, and Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10))1499-155 2016.
Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi- attentional deepfake detection, 2021.
Zhou, X. Han, V. I. Morariu, and L. S. Davis. Two-stream neural networks for tampered face detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.
12


Reviews:
[{"review_id": "LZe2IkKWTL", "reviewer": "Reviewer_8erJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper is well structured and very easy to follow. The authors analyzed why generalization is hard for video forgery detection and give a clear motivation/description of proposed parallel network.\n2. The idea of fusing multiple artificial clues and introducing interactive between two branches to guide each other is novel and makes sense to me.\n3. The proposed new loss function and attention mechanism and the ablation study for them make the proposed method more convincing.\n\nCons:\nI think the experiment part can be further improved.\n1. The compared methods in Table1 and Table2 are not the same, which is confusing. To make the comparison more clear, I think the authors can give one sentence summary for each of methods just to give readers a sense of the difference between proposed method and compared method.\n2. The authors claim that proposed method has better generalization but the experiment conducted for Table2 is not convincing to me.\n    1. Details of transfer learning/fine-tuning are missing. What’s effect of fine-tuning process on generalization?\n    2. The numbers reported in Table2 is AUC or Acc?\n3. I like the idea of two-branch network structure, but to back up this design, it will be stronger to conduct ablation study to investigate the contribution of each branch and the effect of feature interaction between two branches.", "summary_of_the_paper": "This paper proposes a new video forgery detection method (ENST) by combining multiple forgery clues including high frequency, low-level texture, and optical flow. ENST employs a two-branch network, an EfficientNet-B5 branch for high frequent and texture info and an Swin transformer branch for optical flow info. The motivation behind this design is that forgery traces are more likely to present in texture regions where human eyes cannot easily catch, and the optical flow of forgery videos has rare variations compared to real videos.", "summary_of_the_review": "I would recommend accepting this paper. The technique part looks sound to me. The two-branch network structure, the feature interaction between EfficientNet-B5 and Swin transformer, as well as the use of attention and new design of loss function for EfficientNet-B5 make the proposed method solid and convincing. My major concern is from experimental results and ablation study.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IZFHw-DpzX_", "reviewer": "Reviewer_3xwe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n1) Important topic\n2) Identifies important issues in previous deepfake detectors.\n\n### Weakness:\n1) The weakness that author mentions are the main contribution of this paper are already identified by [a].\n2) The paper lacks novelty as the proposed method is using previously available methods and new additions are not significant.\n3) There are many places in the paper, where the author makes some claims but they are not supported by any reference.\n4) The paper is not well written and need rewriting in many sections. For example the use of word “inability” in page 2 paragraph 3 is not correct.\n5) The literature reviews lacks many recent methods [a], [b], [c], [d] and many more. I will suggest the author to look at these papers are find more related papers from them.\n6) The empirical analysis lacks comparison with recent SOTA methods such as CLRNet [a], FReTAL [b], TAR [c], CoReD [d] etc. \n\n[a] Tariq, S., Lee, S., & Woo, S. (2021, April). One detector to rule them all: Towards a general deepfake attack detection framework. In Proceedings of the Web Conference 2021 (pp. 3625-3637).\n\n[b] Kim, Minha, Shahroz Tariq, and Simon S. Woo. \"FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[c] Lee, Sangyup, et al. \"TAR: Generalized Forensic Framework to Detect Deepfakes Using Weakly Supervised Learning.\" IFIP International Conference on ICT Systems Security and Privacy Protection. Springer, Cham, 2021.\n\n[d] Kim, M., Tariq, S., & Woo, S. S. (2021, October). Cored: Generalizing fake media detection with continual representation using distillation. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 337-346).\n\n\n\n\n\n", "summary_of_the_paper": "The author proposes a method for detecting deepfakes that makes use of EfficientNet B5 and Swin Transformer. Additionally, the authors propose a loss function and an attention mechanism for EfficientNet B5. The proposed methods demonstrate increased accuracy and generalizability when compared to the baselines.", "summary_of_the_review": "The paper lacks novelty and writing need improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6eJli_bTgZf", "reviewer": "Reviewer_c2vw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n1. The proposed model integrates the information of frequency domain, spatial domain and optical flow to detect video forgeries.\n2. The paper is well written and easy to follow.\n\nWeaknesses\n1. The proposed multi-branch framework has already been used in previous works; the information of frequency domain, spatial domain and optical flow used in video/image forgery detection have been explored in previous works; the loss functions, i.e., ArcFace and SCL, and the attention module are all borrowed from existing methods. The novelty of this paper seems limited.\n2. This proposed framework utilizes two of the most advanced model structures, i.e., EfficientNet and Swin Transformer, but the results in Tables 1 and 2 only show very marginal improvement.\n3. The authors claimed three clues that are helpful for the video forgery detection, i.e., 1, high-frequency information, 2, texture in the shallow layer of the model, 3, the optical flow of the real video has variations while the optical flow of the deepfake video has rarely variations. While the first has already been explored while the second one is not surprised as well, it would be interesting to see the evidence (examples) of the clues, especially for the third one.\n\n\n", "summary_of_the_paper": "The proposed multi-branch framework that integrates the information of frequency domain, spatial domain and optical flow to detect video forgeries. It also applies two of the most advanced model structures, i.e., EfficientNet and Swin Transformer.", "summary_of_the_review": "My majorpr concern is the novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0GTZaBttbL", "reviewer": "Reviewer_eCnG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern about this submission is related to the contribution, that in my opinion is limited with respect to the current literature. In fact, many of the observations/findings made by the authors are well known in the forensics field and already exploited in published papers. More specifically:\n\n- The importance of high-frequency features has been exploited in many papers that often work on the so called noise residuals obtained suppressing the scene content, see basic works on constrained CNNs (Cozzolino et al. 2017, Bayar et al. 2018). Also frequency domain analysis has been applied to better enhance such frequency domain features (Frank et al. 2020).\n\n- Temporal inconsistencies have been largely exploited especially for deepfake detection, using different types of analyses, even considering optical flow like in this work, see (Amerini et al. 2020, Caldelli et al. 2021).\n\n- Attention mechanisms have been included in several solutions, also similar to what has been done in this work, see (Dang et al. 2020, Bonettini et al. 2020)\n\n- Dual-branch structures that also include a fusion step have been used in several papers, see (Zhou et al. 2018, Masi et al. 2020).\n\n- Working in the YCbCr domain has been done in (Li et al. 2018).\n\nIt is also worth observing that EfficientNet-B7 has been used by the winner of the Kaggle competition in 2020, while transformers have been already considered for deepfake detection in some recent publications (Coccomini et al. 2021, Zheng et al. 2021).\n\nFor the experimental analysis one can observe a very limited improvement in terms of AUC and Accuracy with respect to Zhao et al. 2021, see Table 1 and Table 2. Note that in Zheng et al. results on Celeb-DF are higher than those reported in Table 2 of this work (do these numbers refer to AUC or Accuracy?). Maybe it is worth to check this misalignment (see Table 4 in Zheng et al.). In addition, for what I can understand the experiment carried out in Table 2 uses some videos of Celeb-DF for fine-tuning. I think that if you want to show generalization, you should not include those videos in training. \n\nGeneralization is tested only considering two datasets, FaceForensics++ and Celeb-DF. In my opinion a more extensive analysis should be carried out (see for example the Kaggle dataset proposed by Facebook) and more competitive methods should be included for comparison.\n\nMinor comment: lack of information for some references in the bibliography (venue of publication is missing) and many typos\n(e.g. Rrf Attux, Niener, Rssler). The reference by Zhou et al. is repeated twice.\n\nReferences\n- Bayar et al. Constrained Convolutional Neural Networks: A New Approach Towards General Purpose Image Manipulation Detection, IEEE TIFS 2018\n- Dang et al. On the detection of digital face manipulation, CVPR 2020\n- Frank et al. Leveraging Frequency Analysis for Deep Fake Image Recognition, ICML 2020\n- Coccomini et al. Combining EfficientNet and Vision Transformers for video deepfake detection, arXiv 2021\n- Zheng et al. Exploring Temporal Coherence for More General Video Face Forgery Detection, CVPR 2021", "summary_of_the_paper": "This paper faces the problem of video deepfake detection. The idea is to rely on high frequency, texture and optical flow cues to gain generalization. To this end it is proposed a dual-branch detection approach: low-level features are extracted by EfficientNet-B5 that also includes an attention mechanism, while temporal inconsistencies are captured by Swin Transformer. Experiments are carried out on FaceForensics++ and Celeb-DF (v2) and show better generalization ability with respect to state-of-the-art.\n", "summary_of_the_review": "In this paper it is proposed a deep learning based approach for video deepfake detection. In my opinion the technical novelty is not sufficient to warrant publication in ICLR and the performance improvement is marginal with respect to state-of-the-art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new method for deepfake detection (ENST) which relies on high-frequency information, low-level/shallow features, and optical flow. In particular, EfficientNet-B5 is used to extract the high frequency info and shallow features, and a Swin Transformer to capture discrepancies between optical flows. Empirical validation on FaceForensics++ and Celeb-DF shows some improvements over the baselines.

The reviewers found this to be a relevant and timely topic. The reviewers also found that integrating information from the frequency domain, the spatial domain, and optical flow is a promising approach. There were three reviewers suggesting rejection, and one suggesting acceptance. After the rebuttal and discussion phase, the following remaining issues were highlighted: 
- **Limited technical novelty** (nearly all components used in this work were already expired in other work).
- Underwhelming empirical improvements given the fact that the model uses EfficientNet-B5 and the SwinTransformer. 
- Many claims are still not supported by empirical evidence. For instance, to claim generalisation, an extensive analysis, including more datasets as well as competing methods should be carried out.
Decision: Reject