Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for deepfake detection against adversarial attacks. It proposes to use an ensemble of models whose inputs are subsets of frequency spectrum. It also theoretically proves that the ensemble of models reduce the dimension of the adversarial subspace, which could increase the robustness.
strength_and_weaknesses: Strength

1. The idea of training ensemble of models using disjoint frequency spectrum rather than pixels conceptually makes sense. As transforming images into frequency domain can be seen as a decomposition of input features, and dispatching these features to different models can further reduce the complexity of single models (which could make them more robust).
2. The authors also provide a theoretical proof on how their method could reduce the dimension of the adversarial subspace.

Weakness

I didn’t see any major weaknesses in this paper.
clarity,_quality,_novelty_and_reproducibility: The paper seems clear and novel to me.
summary_of_the_review: Overall, the paper is equipped with interesting ideas and strong experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose an ensemble approach to deepfake detection based on a frequency-domain decomposition of input images, such that each model in the ensemble is likely to see fewer non-robust features. The authors offer a theoretical justification in terms of a reduction in the dimensionality of the adversarial subspace, building upon the work of Tramèr et al. (2017). Evaluation using GAN-generated images in both the white- and black-box settings, leveraging the AutoAttack benchmark, shows significant improvements compared to SOTA.
strength_and_weaknesses: - Strengths
  - The problem is of interest and current understanding is still lacking. The proposed approach and theoretical analysis are very valuable.
  - The experiments do a good job establishing the advantage against using the full spectrum, and justify the saliency-based partitioning.

- Weaknesses
  - As pointed out by other reviewers, important discussion is missing regarding both related work and experiment setup.
  - It is not clear what is the limit of this redundancy as far as classification accuracy is concerned. At which point does the model reduce to random guessing, where real and fake appear too similar? I would like to see an experiment exploring this.
clarity,_quality,_novelty_and_reproducibility: - Clarity and quality of the presentation are good, subject to filling in the missing discussion.
- Novelty: The theoretical analysis closely follows the work of Tramèr et al. (2017).
  - I understand that the authors were mainly concerned with the problem of deepfake detection, but the frequency decomposition ensemble is fairly interesting in itself and deservers a more dedicated study, e.g., in the context of mainstream adversarial attacks.
 - I encourage the authors to continue this investigation.

- Nitpicking
  - In the notation part in Section 2, please fix $\in$ to $\subseteq$.
  - It may help to rewrite equations 2-6 in terms of the common inner expressions. Perhaps those can be called something like $\kappa_2(g_i)$ and $\kappa_\infty(g_i)$.
summary_of_the_review: The paper is missing important discussion of related works, acknowledging prior solutions to the same problem, as well as critical details in the experimental setup and evaluation.

Update: reviewer discussions indicate those concern are still not resolved
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an adversarially robust deepfake detection approach. The approaches divided the frequency information into multiple blocks based on random bisection or saliency information. This divided information is provided to the different models and later combined for robust detection accuracy. The proposed approach was found robust in handling a few attacks and acts as a baseline for future studies. 
strength_and_weaknesses: The proposed approach was found robust in handling few adversarial attacks which make the existing detection algorithms ineffective. 

The claims made especially being the first work are somewhat wrong. In the literature, several works have explored the vulnerability and present robust deepfake detection approaches. 

[1] EnsembleDet: ensembling against adversarial attack on deepfake detection
[2] Defending against GAN-based DeepFake Attacks via Transformation-aware Adversarial Faces

The proposed algorithm is found vulnerable against one of the adversarial attacks and hence demands extensive evaluation against state-of-the-art attacks including frequency-based attacks. 

[3] Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples
[4] MD-CSDNetwork: Multi-Domain  Cross  Stitched  Network for Deepfake Detection, IEEE International  Conference on Automatic Face and Gesture Recognition, 2021

The proposed algorithm should also be compared with ensemble-based approaches the baseline must also contain the results with ensemble algorithms: (i) train different networks on frequency information and (ii) utilize multiple types of input images and training of a detection algorithm.

The majority of the content in the paper refers to the appendix only, which makes the paper hard to read and follow. The authors need to make sure that all the important information is presented in the main paper. 

The comparison with SOTA work is also shallow. 

How the adversarial training has been performed? The results reported by AT seem wrong and need independent verification. 
clarity,_quality,_novelty_and_reproducibility: The paper is hard to follow due to extreme reference to the appendix and the unavailability of the important results in the main paper. 

The novelty is limited as an ensemble is an active area of research and has been extensively explored utilizing frequency information. 

The paper might be easy to reproduce; however, the authors have not mentioned their intention of releasing the source codes. 
summary_of_the_review: The proposed research needs attention towards the evaluation strategies including novel attacks belonging to the black box, frequency-based, and adaptive attacks not gradient-based. The comparison in the paper is shallow. 

On top of that, the generalizability of the proposed algorithm against unseen adversaries also needs to be addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on learning a robust deepfake detector against adversarial attacks. The authors propose a method named Disjoint Deepfake Detection (D3). The method is motivated by the observation that redundancy exists in the frequency space, which means fewer signals are enough to make correct predictions. Based on this, the authors divide the DCT spectrum of an image into disjoint partitions and adopt an ensemble method to adversarially train each individual model. The authors show clear performance improvement, especially on the dataset of FFHQ with the deepfake generation by StyleGAN.
strength_and_weaknesses: **Strength**

[1] This paper studies a less explored task, to robustify the deepfake detector. It should have practical needs and is quite important.

[2] The method is motivated by the experimental observation that redundancy exists in the frequency space, especially for deepfake images.

[3] Based on the observation, the ensemble model is natural to be considered.

[4] The authors shared some theoretical analysis of the proposed ensemble method, though I didn't check the proof exactly and am not sure if it is solid.

**Weakness**

[1] Incomplete literature survey. Missing important related work. The authors claimed "Defending against adversarial examples, in general, has been shown to be a difficult task (Athalye et al., 2018), and is unexplored in the deepfake detection setting." However, there are published papers aiming at the adversarial defense of deepfake detection. For example:

*Gandhi, Apurva, and Shomik Jain. "Adversarial perturbations fool deepfake detectors." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.*

In this IJCNN paper, the authors explored two defensive mechanisms to deepfake detectors: (i) Lipschitz regularization, and (ii) Deep Image
Prior (DIP). Thus, the statement "Defending against adversarial examples is unexplored in the deepfake detection setting" is inaccurate.

[2] Overclaim. "We propose D3, the first deep fake detection framework designed to be adversarially robust." This is an overclaim, in consideration of my last comment.

[3] Missing details in experimental settings. The descriptions on the dataset are vague. The authors claimed, "We use 50,000 images for training, 10,000 for validation, and 10,000 for testing sets". Among these, how many are real images, and how many are deepfake images? I assume it is possible that the authors first collected 70000 images from the FFHQ dataset, and then generated a deepfake image for each by StyleGAN. Also, it is not clear how to generate deepfake images by StyleGAN in this paper. The goal of deepfake detection is not clear either. Is it going to output a 0/1 binary value to verify if an image is fake or not, or output a detection region to indicate the deepfake? This relates to the understanding of the evaluation metric AUC-ROC used in this paper, but with few descriptions of its details. This brings significant difficulty in understanding the experiments and justifying the performance of the proposed method. 

[4] Worse performance on FaceForensics++ dataset. In Table 8, I assume the authors are reporting accuracies (unfortunately no descriptions, and this table is even unreferred in the main text). The performance of the proposed D3 is significantly worse than the adversarial training method AT(1). Again, it seems not mentioned which types of AT the authors are using, by Madry's, or else? In a word, does this experiment indicate the proposed method only works on specific datasets, like FFHQ, but not working on video dataset FaceForensics++?

[5] This paper aims to solve a deepfake detection problem, in the adversarial setting, with datasets of human faces. But unfortunately,  almost no figures are shown in this paper to show the task. Only an image of Obama is given in Figure 1 to show the method pipeline. I am still confused after reading the whole paper, that what does the deepfake image look like? And which part of the deepfake images are detected as a clue for final judgment? Only some tables and curves are shown, but some of them lack detailed descriptions as I mentioned above.

[6] In Tables 1,2,3, plenty of 100%s are shown for the proposed method. This is very different in general image classification datasets, like CIFAR-10. It is hard for state-of-the-art adversarial defense methods to reach 60%, even with very large networks like WideResNet. I am not saying the proposed results are too good to be true, but I am curious if the authors could give some insights behind these perfect results, especially on FFHQ images. In comparison, the baselines are extremely worse, even worse than a random guess I assume. Does this indicate a stronger baseline should be considered?

clarity,_quality,_novelty_and_reproducibility: **Clarity** 

The authors basically clearly present their problem and methods. The proposed method is easy to follow. But there are many ambiguous places in the experiment part, which makes the audience hard to understand.

**Quality**

The quality is overall below the line of acceptance by ICLR, with insufficient literature survey, and experiment details.

**Novelty**

The task is less explored (other than the wording "unexplored" by the authors). The proposed method is enlightened by the ensemble adversarial training but on frequency space. I suppose the technical novelty is marginal.

**Reproducibility**

Considering the inadequate details in the experiments, I suppose reproducibility is nearly impossible, without the release of code.
summary_of_the_review: In summary, I vote for the rejection of this manuscript, due to the insufficient literature survey, ambiguous experiment details, and unclear result analysis.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose an ensemble approach to deepfake detection based on a frequency-domain decomposition of input images, such that each model in the ensemble is likely to see fewer non-robust features. The authors also offer a theoretical justification to show that the ensemble of models could reduce the dimension of the adversarial subspace and increase the robustness. 

Strengths:  Training an ensemble of models using a disjoint frequency spectrum is novel and makes sense. Results show that the proposed method achieves superior results than SOTA methods. Theoretical analysis extending prior work is valuable.

Weakness: The idea of ensemble models, frequency domain, and redundancy space for adversarial robustness is not novel. Lack some experimental evidence to support the claims. The disjoint frequency spectrum plays major importance in D3, and if an adversarial attack only focuses on the original input without considering the frequency partitions, it may be unfair for pixel-based methods. Missing more details and visualisations about frequency components, e.g., How to define the d frequency components, is 64 for dct-2d? The generalization importance can also be seen from the values reported in table 6 on page 17. The evaluation is on BigGAN (type of GAN only), still, the performance is quite low as compared to the performance on other forms of GANs used in the main paper. Similar to StarGAN. That's why the evaluation under unseen GAN evaluation makes sense to showcase the effectiveness of this work. Another concern is a possible bias toward one form of adversarial attack.
justification_for_why_not_higher_score: Some missing experimental evidences to support the claims. 
justification_for_why_not_lower_score:  Training an ensemble of models using a disjoint frequency spectrum is novel and makes sense.
Decision: Reject