Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The proposed multi-branch framework that integrates the information of frequency domain, spatial domain and optical flow to detect video forgeries. It also applies two of the most advanced model structures, i.e., EfficientNet and Swin Transformer.
main_review: Strengths 
1. The proposed model integrates the information of frequency domain, spatial domain and optical flow to detect video forgeries.
2. The paper is well written and easy to follow.

Weaknesses
1. The proposed multi-branch framework has already been used in previous works; the information of frequency domain, spatial domain and optical flow used in video/image forgery detection have been explored in previous works; the loss functions, i.e., ArcFace and SCL, and the attention module are all borrowed from existing methods. The novelty of this paper seems limited.
2. This proposed framework utilizes two of the most advanced model structures, i.e., EfficientNet and Swin Transformer, but the results in Tables 1 and 2 only show very marginal improvement.
3. The authors claimed three clues that are helpful for the video forgery detection, i.e., 1, high-frequency information, 2, texture in the shallow layer of the model, 3, the optical flow of the real video has variations while the optical flow of the deepfake video has rarely variations. While the first has already been explored while the second one is not surprised as well, it would be interesting to see the evidence (examples) of the clues, especially for the third one.



summary_of_the_review: My majorpr concern is the novelty.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The author proposes a method for detecting deepfakes that makes use of EfficientNet B5 and Swin Transformer. Additionally, the authors propose a loss function and an attention mechanism for EfficientNet B5. The proposed methods demonstrate increased accuracy and generalizability when compared to the baselines.
main_review: ### Strengths:
1) Important topic
2) Identifies important issues in previous deepfake detectors.

### Weakness:
1) The weakness that author mentions are the main contribution of this paper are already identified by [a].
2) The paper lacks novelty as the proposed method is using previously available methods and new additions are not significant.
3) There are many places in the paper, where the author makes some claims but they are not supported by any reference.
4) The paper is not well written and need rewriting in many sections. For example the use of word “inability” in page 2 paragraph 3 is not correct.
5) The literature reviews lacks many recent methods [a], [b], [c], [d] and many more. I will suggest the author to look at these papers are find more related papers from them.
6) The empirical analysis lacks comparison with recent SOTA methods such as CLRNet [a], FReTAL [b], TAR [c], CoReD [d] etc. 

[a] Tariq, S., Lee, S., & Woo, S. (2021, April). One detector to rule them all: Towards a general deepfake attack detection framework. In Proceedings of the Web Conference 2021 (pp. 3625-3637).

[b] Kim, Minha, Shahroz Tariq, and Simon S. Woo. "FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[c] Lee, Sangyup, et al. "TAR: Generalized Forensic Framework to Detect Deepfakes Using Weakly Supervised Learning." IFIP International Conference on ICT Systems Security and Privacy Protection. Springer, Cham, 2021.

[d] Kim, M., Tariq, S., & Woo, S. S. (2021, October). Cored: Generalizing fake media detection with continual representation using distillation. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 337-346).






summary_of_the_review: The paper lacks novelty and writing need improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper faces the problem of video deepfake detection. The idea is to rely on high frequency, texture and optical flow cues to gain generalization. To this end it is proposed a dual-branch detection approach: low-level features are extracted by EfficientNet-B5 that also includes an attention mechanism, while temporal inconsistencies are captured by Swin Transformer. Experiments are carried out on FaceForensics++ and Celeb-DF (v2) and show better generalization ability with respect to state-of-the-art.

main_review: My main concern about this submission is related to the contribution, that in my opinion is limited with respect to the current literature. In fact, many of the observations/findings made by the authors are well known in the forensics field and already exploited in published papers. More specifically:

- The importance of high-frequency features has been exploited in many papers that often work on the so called noise residuals obtained suppressing the scene content, see basic works on constrained CNNs (Cozzolino et al. 2017, Bayar et al. 2018). Also frequency domain analysis has been applied to better enhance such frequency domain features (Frank et al. 2020).

- Temporal inconsistencies have been largely exploited especially for deepfake detection, using different types of analyses, even considering optical flow like in this work, see (Amerini et al. 2020, Caldelli et al. 2021).

- Attention mechanisms have been included in several solutions, also similar to what has been done in this work, see (Dang et al. 2020, Bonettini et al. 2020)

- Dual-branch structures that also include a fusion step have been used in several papers, see (Zhou et al. 2018, Masi et al. 2020).

- Working in the YCbCr domain has been done in (Li et al. 2018).

It is also worth observing that EfficientNet-B7 has been used by the winner of the Kaggle competition in 2020, while transformers have been already considered for deepfake detection in some recent publications (Coccomini et al. 2021, Zheng et al. 2021).

For the experimental analysis one can observe a very limited improvement in terms of AUC and Accuracy with respect to Zhao et al. 2021, see Table 1 and Table 2. Note that in Zheng et al. results on Celeb-DF are higher than those reported in Table 2 of this work (do these numbers refer to AUC or Accuracy?). Maybe it is worth to check this misalignment (see Table 4 in Zheng et al.). In addition, for what I can understand the experiment carried out in Table 2 uses some videos of Celeb-DF for fine-tuning. I think that if you want to show generalization, you should not include those videos in training. 

Generalization is tested only considering two datasets, FaceForensics++ and Celeb-DF. In my opinion a more extensive analysis should be carried out (see for example the Kaggle dataset proposed by Facebook) and more competitive methods should be included for comparison.

Minor comment: lack of information for some references in the bibliography (venue of publication is missing) and many typos
(e.g. Rrf Attux, Niener, Rssler). The reference by Zhou et al. is repeated twice.

References
- Bayar et al. Constrained Convolutional Neural Networks: A New Approach Towards General Purpose Image Manipulation Detection, IEEE TIFS 2018
- Dang et al. On the detection of digital face manipulation, CVPR 2020
- Frank et al. Leveraging Frequency Analysis for Deep Fake Image Recognition, ICML 2020
- Coccomini et al. Combining EfficientNet and Vision Transformers for video deepfake detection, arXiv 2021
- Zheng et al. Exploring Temporal Coherence for More General Video Face Forgery Detection, CVPR 2021
summary_of_the_review: In this paper it is proposed a deep learning based approach for video deepfake detection. In my opinion the technical novelty is not sufficient to warrant publication in ICLR and the performance improvement is marginal with respect to state-of-the-art.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new video forgery detection method (ENST) by combining multiple forgery clues including high frequency, low-level texture, and optical flow. ENST employs a two-branch network, an EfficientNet-B5 branch for high frequent and texture info and an Swin transformer branch for optical flow info. The motivation behind this design is that forgery traces are more likely to present in texture regions where human eyes cannot easily catch, and the optical flow of forgery videos has rare variations compared to real videos.
main_review: Pros:
1. This paper is well structured and very easy to follow. The authors analyzed why generalization is hard for video forgery detection and give a clear motivation/description of proposed parallel network.
2. The idea of fusing multiple artificial clues and introducing interactive between two branches to guide each other is novel and makes sense to me.
3. The proposed new loss function and attention mechanism and the ablation study for them make the proposed method more convincing.

Cons:
I think the experiment part can be further improved.
1. The compared methods in Table1 and Table2 are not the same, which is confusing. To make the comparison more clear, I think the authors can give one sentence summary for each of methods just to give readers a sense of the difference between proposed method and compared method.
2. The authors claim that proposed method has better generalization but the experiment conducted for Table2 is not convincing to me.
    1. Details of transfer learning/fine-tuning are missing. What’s effect of fine-tuning process on generalization?
    2. The numbers reported in Table2 is AUC or Acc?
3. I like the idea of two-branch network structure, but to back up this design, it will be stronger to conduct ablation study to investigate the contribution of each branch and the effect of feature interaction between two branches.
summary_of_the_review: I would recommend accepting this paper. The technique part looks sound to me. The two-branch network structure, the feature interaction between EfficientNet-B5 and Swin transformer, as well as the use of attention and new design of loss function for EfficientNet-B5 make the proposed method solid and convincing. My major concern is from experimental results and ablation study.
review_title: null

Meta-review: The authors propose a new method for deepfake detection (ENST) which relies on high-frequency information, low-level/shallow features, and optical flow. In particular, EfficientNet-B5 is used to extract the high frequency info and shallow features, and a Swin Transformer to capture discrepancies between optical flows. Empirical validation on FaceForensics++ and Celeb-DF shows some improvements over the baselines.

The reviewers found this to be a relevant and timely topic. The reviewers also found that integrating information from the frequency domain, the spatial domain, and optical flow is a promising approach. There were three reviewers suggesting rejection, and one suggesting acceptance. After the rebuttal and discussion phase, the following remaining issues were highlighted: 
- **Limited technical novelty** (nearly all components used in this work were already expired in other work).
- Underwhelming empirical improvements given the fact that the model uses EfficientNet-B5 and the SwinTransformer. 
- Many claims are still not supported by empirical evidence. For instance, to claim generalisation, an extensive analysis, including more datasets as well as competing methods should be carried out.
Decision: Reject