Towards Adversarially Robust Deepfake Detection: An Ensemble Approach
Authors: Ashish Hooda
Neal Mangaokar
Ryan Feng
Kassem Fawaz
Somesh Jha
Atul Prakash
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deepfakes
Ensembles
Adversarial Subspace
Frequency
Defense

Abstract: Detecting deepfakes remains an open problem. Current detection methods fail against an adversary who adds imperceptible adversarial perturbations to the deepfake to evade detection. We propose Disjoint Deepfake Detection (D3), a deepfake detector designed to improve adversarial robustness beyond de facto solutions such as adversarial training. D3 uses an ensemble of models over disjoint subsets of the frequency spectrum to significantly improve robustness. Our key insight is to leverage a redundancy in the frequency domain and apply a saliency partitioning technique to disjointly distribute frequency components across multiple models. We formally prove that these disjoint ensembles lead to a reduction in the dimensionality of the input subspace where adversarial deepfakes lie. We then empirically validate the D3 method against white-box attacks and black-box attacks and find that D3 significantly outperforms existing state-of-the-art defenses applied to deepfake detection.

Under review as a conference paper at ICLR 2023
ROBUST DEEPFAKE DETECTION USING DISJOINT ENSEMBLES ON REDUNDANT FEATURES
Anonymous authors Paper under double-bling review
ABSTRACT
Detecting GAN- -AA-geeneatedd deepfake images remains open problem. Current detection methods fail against an adversary who adds imperceptible adversarial perturbations to the deepfake to evade detection. We propose Disjoint Deepfake Detection (D3), a detector designed to improve adversarial robustness beyond de facto solutions such as adversarial training. D3 uses an ensemble of models over disjoint subsets of the frequency spectrum to significantly improve robustness. Our key insight to leverage a redundancy in the frequency domain and apply a saliency partitioning technique to disjointly distribute frequency components across multiple models. We formally prove that these disjoint ensembles lead to a reduction in the dimensionality of the input subspace where adversarial deepfakes lie. We then empirically validate the D3 method against white-box attacks and black-box attacks and find that D3 significantly outperforms existing state-of-thee art defenses applied to GAN- -eneraeed deepfake detection.
INTRODUCTION
Significant advances in deep learning are responsible for the advent of "deepfakes", which can be misused by bad actors for malicious purposes. Deepfakes broadly refer to digital media that has been synthetically generated or modified by deep neural networks (DNNs). Modern DNNs such as generative adversarial networks (GANs) (Goodfellow et al. 2014) are now capable of synthesizing hyper-realistic deepfakes, which can then be used to craft fake social media profiles (Martineau, 2019), generate pomography (Cole 2017), spread political propaganda, and manipulate elections.
While recent work has made respectable efforts towards solving the deepfake detection problem, many of these detectors rendered ineffective by adversarial examples. The deepfake detection problem asks the defender to classify a given image as deepfake or real (we focus on GAN-generated detectors often leverage DNNs the frequency space of the in- put image (Frank et al. 2020). However, CCarlini & Farid (2020) (amongst others) have shown that such defenses fail the adversary can simply use adversarial perturbation techniques to evade de- tection (Szegedy et al. 2014; Goodfellow et al. 2015 Biggio et al.) [013]] Carlini & Wagner (017)) Defending against adversarial examples, in general, has been shown be difficult task (Athalye) et al. 2018), and is critical problem in the deepfake detection setting.
deepfake images).
Our key intuition to mitigate this problem is to utilize redundant information the frequency feature space of deepfakes to generate disjoint ensembles for deepfake detection. Specifically, we show in Section 3.1 that we can achieve good detection performance with only a subset of the features, particularly in the frequency domain. This enables us to build ensemble of performant classifiers, each using a disjoint set of frequencies. In contrast to traditional ensembles (where each model shares the same set of frequencies), key advantage of this design is that non-robust frequencies partitioned across all the models in the ensemble. Thus, an adversary is no longer able to perturb single non-robust frequency evade all models rather, they must find perturbations to evade multiple sets of disjoint frequencies, which raises the attack cost.
Leveraging the above intuition, key contributions
follows:
We propose D3. a GAN- -AN-geeerated deepfake detection framework designed to be adversar- ially robust. D3 builds robust ensemble of models that use disjoint partitions of the input

---

Under review as a conference paper at ICLR 2023
features. This is achieved by leveraging redundancy in the feature space. D3 achieves bustness while still exhibiting natural deepfake detection AUC-ROC scores as high as 99% (sce Section 4.4 for details).
2. Extending the theoretical results by Tramer et al. (b01 on dimensionality of adversarial subspaces, we prove new bounds on the maximum number of adversarial directions that can be found under an ensemble with disjoint inputs. Our bounds are tight for both the Eg and Lo perturbation norms (Lemmas 3.1 and 3.2 in Section 3.3) and indicate that D3 reduces the dimension of the adversarial subspace.
3. We evaluate D3 against white-box and black-box attacks on variety of GAN- GAN-ggnnrated deepfake images and find that D3 significantly outperforms state-of-the-an defenses such as ADP (Pang et al. 2019). GAL (Kariyappa & Qureshi 2019) and DVERGE (Yang et al.) 2020), suggesting a reduction in dimension of the adversarial subspace. For example, as indicated by our evaluation in Section|4. 4.2) D3 maintains 100% adversarial accuracy against AutoAttack (Croce& Hein| 2020b) where baselines drop below 20%. Increasing the num- ber of attack steps to 1000 reduces D3's robustness to 61% % in comparison to 0% for all baselines.
2 BACKGROUND AND RELATED WORK
Notation. We consider distribution D over X' X,, where X' C IR is the input space and y C 2 is the finite class-label space. We denote vectors in boldface (e.g., x). We denote a trained classifier as a function F X' + Y' (the classifier is usually parameterized by its weights w, omitted for brevity). We denote the loss function as [(x, y). An ensemble classifier is a function M(Ji Fa...a X + y that combines the logit outputs 1 12, ...,, In of multiple classifiers F1, F2,.... Fn with voting aggregation function A Raxe + )
We now provide a definition for adversarial examples against classifier. For a classifier F and input-label pair (x, y). an adversarial example is a perturbed input x' such that (1) x' is misclassified, i.e., F(x') * y and (2) ||x x'I is within small ball, where ||-I| a given norm. The value of c is chosen to be small so that the perturbation is imperceptible to humans.
Deepfake Detection. The research community has made rapid progress towards detecting GAN- generated deepfake images. Examples of proposed detection schemes include DNN classi- fiers (Wang et al., 2020) Yu et al.) 2019), color-space anomaly detection (MeCloskey Albright 2018), and co-occurrence matrix analysis (Nataraj al. 2019). amongst others (Tariq et al. 2019
Mi et al. 2020; Marra et al. 2018 Guarnera et al. 2020 Marra et al. 2019).
However, recent line of work has significantly advanced the state-of-the-art for deepfake detection by leveraging frequency-spact analysis. For example, Frank et al.] (2020) proposed the idea of detecting deepfakes with the Discrete Cosine Transform (DCT) a peepprepooe preeprrcessing transform before a binary DNN-based deepfake classifier. Similar work has also achieved remarkable performance
Zhang et al. (2019) GAN simulators to extract similar frequency artifacts, and Durall et al. 2019) successfully train DNNs in the frequency domain to detect deepfakes.
Unfortunately, the aforementioned detectors have been rendered ineffective in adversarial settings. Specifically, CCarlni & Farid (2020) showed that frequency-baser detectors are vulnerable to adver- sarial examples - an adversary can add imperceptible adversarial perturbations
deepfake that evade such detectors, rendering them ineffective. Others have corroborated this observation (Hus-
Isain et al. 2021 2022b: Neekhara et al. |021 Gandhi & Jain
2020 Vo et al. 2022: Shahriyar & Wright 2022 Fernandes & Jha 2020 Liao et al. 202 Hussain et al. 2022a).
Detecting Adversarial Deepfakes. widely accepted solution to countering adversarial exam- ples is to train the model on adversarial examples, generated during training (Madry et al.) 2018). However, Carlini & Farid (2020) suggest that adversarial training alone is unlikely to achieve sig- nificant improvement in robustness in the difficult deepfake detection setting. Our experiments in deepfake detector, even when adver- sarially trained, cannot withstand deepfake examples crafted using variety of attacks (see Tables [ 2) While we also adversarially train each model in our ensembles, we find that our ensembles significantly improve robustness over standalone adversarially trained model (see Section 4.2
Section 4.2 confirm that Frank et al. (2020)* 's frequency-spar
2

---

Under review as a conference paper at ICLR 2023
1.0
Partitioning Scheme
Ensemble
0.9
Model #1
0.8
Deeplake: Standard
0.7
Deeptske: DCT Bask
GFAR-16: DCT fadds
Deepfake fake?
0.6
OFAR 10: Standard Basis
0.5
DCT
0.4
Moda #in Re
100%
50%
25% 12.5% 6.25%
Spectrum
% features
Figure 2: AUC-ROC scores of a single CNN classifier trained on fixed subset (randomly selected) of the input features. Redun- dancy in frequency spectra per- mits near-perfect deepfake detec- tion even when using ~ 3% of the components (see green line).
Figure The processing pipeline of D3. It partitions the DCT spectrum of an image into disjoint partitions using ei- ther randomized or saliency-based approaches. Each fre- quency partition is fed to separate model that is adversari- ally trained. voting mechanism over the ensemble decides the output.
Another class of defenses focuses removing the effect of the adversarial perturbation without any changes made to the underlying detection technique unfortunately. these approaches are computationally intensive, e-g., upto 30 minutes per image (Gandhi & Jain 2020 Jiang al. [02I)) in comparison to no additional overhead in D3. Nevertheless, both approaches are complementary in that ppe--proeessigg could be combined with adversarially robust deepfake detectors.
Another approach is an ensemble of models (Pang et al., 2019 Kariyappa & Qureshi 2019 Yang al., 2021)
principle, the adversary then forced attack multiple models, instead just one. However, He et al. (2017) have shown that arbitrarily ensembling models does not necessarily lead more robust defenses. One reason for this (as per [[lyas et al. (2019) is that each model tends to learn the same non-robust features, i.e., an adversary is able to perturb the same feature to evade all models. Recent efforts at adversarially robust ensembles have attempted to mitigate this phe- nomenon by introducing additional regularizers to the training process (Pang et al. 2019 Kariyappal & Qureshi 2219;; Yang et al. 202I). In contrast, D3 bypasses this phenomenon by disjointly parti- tioning features amongst models, thereby outperforming these approaches (see Section 4.2)
More recently, Dutta et al. (202I) proposed EnsembleDet, which ensembles different DNN archi- tectures on the same set of features defense. We differ from EnsembleDet philosophically. in that we do not require different architectures, and instead leverage artifact redundancy to design frequency-partitee -arritioned ensembles (see Section3]oor more detailed explanation). Furthermore, D3's partitioning approach to achieve disjoint ensembles is novel, and it offers theoretical and empirical advantages where such feature redundancy is available, e.g., GAN-generated deepfakes.
3
OUR APPROACH
We now present D3 (Figure 1 an ensemble deepfake detection framework that leverages disjoint frequency models to achieve robust detection of GAN-generated deepfake images. Section 3.
presents our observation of redundant information in the frequency space of deepfakes. Section 3.2
details how redundancy allows frequencies to be partitioned between multiple models for robust ensemblinga explains our exact frequency partitioning schemes. Finally, Section 3.3 provides theoretical insights into why D3 improves adversarial robustness.
3.1 REDUNDANT INFORMATION IN DEEPFAKE IMAGE SPECTRA
As discussed in Section 2 ensembles are promising approach to adversarial robustness, so long as perturbing the same set of features does not simultancously evade the individual models. We propose designing an ensemble that avoids this shortcoming by disjointly partitioning the input feature space amongst individual models.

---

Under review as a conference paper at ICLR 2023
Unfortunately. for traditional image classifications tasks like CIFARIO (Krizhevsky et al. 2009 and ImageNet (Deng et al., 2009). a disjoint partitioning is likely to hurt natural classification perfor- mance, as partitioning results in information loss for each model. We visualize this phenomenon for CIFARIO via the red and teal lines in Figure 2 which plots the worsening performance of a simple convolutional classifier using increasingly small subsets of the input features (pixel frequeney).
Our first key insight is that, unlike traditional image classification tasks, disjoint partitioning is feasible for deepfake detection. Specifically, we observe "redundancy" in frequency-space artifacts signals relevant for deepfake detection Fl are distributed throughout the frequency spectrum. This observation best exemplified by the green line in Figure 2 which plots the deepfake detection performance using increasingly small subsets of the spectrum using the same classifier. Using as few as ~ 3% of the frequency components yields near-perfect deepfake detection performance. We emphasize that this does not hold for subsets of pixels (blue line), signals for detection are not dell-distributed in the RGB space. Thus, the frequency-space contains plenty of redundancy, which can potentially be leveraged to design robust ensemble.
3.2 LEVERAGING REDUNDANCY TO BUILD ROBUST ENSEMBLE
Using the observations in Section 3.1 one can craft a robust ensemble by partitioning the frequency components amongst multiple detector models, without hurting natural detection performance. Fig- ure visualizes this partitioning as part of our ensembling pipeline. Specifically, for each individual model we mask (i.e., zero out) the frequencies not used. For example, consider an ensemble of two such "disjoint" models FA and FB. where the full-spectrum frequency feature vector f = [fi fel- Then, the input feature vector to FA is [fi; 0) and to FB is [O; f2]. Since the input feature space (frequencies) is not shared amongst the individual models, the adversary cannot simply attack the ensemble by targeting common frequencies.
Furthermore, we note that the choice of partitioning scheme, i.e., how the frequencies are partitioned plays an important role in robustness of the ensemble. Specifically, the chosen scheme should aim to design all models as "equals" - if some models are less robust than others, then the adversary can target them to overturn the ensemble's decision. Below we propose a naive solution (random partitioning), followed by a scheme designed directly achieve this goal (saliency partitioning).
Random Partitioning. For an ensemble of size n, this naive partitioning scheme divides the set of d frequency components into n equally-sizer disjoint subsets, uniformly at random. Each model in the ensemble then receives one subset.
Saliency Partitioning. While signals for deepfake detection distributed throughout the spec- trum, there still exists a adversarial saliency ordering of these frequencies that determines their robustness for the deepfake detection task. For an ensemble of size n. our saliency partitioning technique is aimed at ensuring each model receives a fair proportion of salient frequencies. To this end, we follow Carlini & Wagner (2017) and Feng et al.) (2022) compute saliency values for all frequencies. This is achieved by adversarially perturbing deepfake x to x + 8², where 8" is the perturbation computed with an unbounded 1sc PGD attack until attack success or 1000 steps, and computing saliency for the ith frequency
s; = EZEx Vfxx+
(1)
where subscript denotes the 7th component. Intuitively, higher gradients and larger perturbation magnitudes imply larger saliencies. Frequencies are then ordered by their saliencies, and distributed in round-robin fashion amongst the models.
3.3 ADVERSARIAL SUBSPACE OF DISJOINT ENSEMBLES
now show that ensemble of such disjoint frequency models increases robustness against adversarial examples by reducing the dimension of the adversarial subspace. For single model F and input x. Tramer al. (2017b) approximate
Given the partitioning approach in Section 3.2
We use the same architecture for our primary defense implementation; details are provided in Section 4.1
2 Frank et al. 2020) show that these artifacts manifest in the form of grid-like spectral pattern, and attribute their presence the upsampling process in generative models. Existing work proposes detectors that leverage the entire frequency spectrum for deepfake detection.
4

---

Under review as a conference paper at ICLR 2023
the k- diddinnnioonal adversarial subspace the span of orthogonal perturbations r1:". .k R such that ri ri 2 A 1 1 i I where g = VxLx(xx,)) y). LF is the loss function used to train F, and y is the increase in loss sufficient to cause missclassification For perturbations satisfying the fn-norm, i.. ||rı|2 < A < S k, the adversarial dimension is bounded by llll (tight). In what follows, we extend this result and provide bounds for dimensionality of the shared adversarial subspace between n disjoint models.
provide tight bounds for both & and &o norms in Lemma [.1 and Lemma 3.2 respectively (with detailed proofs in Appendix [A.I) D. We consider these bounds for two voting mechanisms: (1) majority, where the ensemble outputs deepfake if at least [n/2] classifiers predict deepfake, and (2) at-least-one, where the classifier outputs deepfake at least one classifier predicts deepfake, otherwise it outputs real.
Lemma 3.1. Given nt disjoint models, Fi,..., having gradients gi,... .gn Rd for input- label pair (x,)) V) (where gj = (x, the maximum number k of orthogonal vectors rı, T2." .k E Rdd satisfying ||ril22 I and, rg %i for all S j I (at-least-one voting) or for at least [ models (majority voting), for all I S k given by:
k: min
d,
gi 12
(at-least-one voling)
7/s
2
k S
min
d,
max [41
Σ
|lsill
(majority voting)
KK
Σ
79)²
JEK
JEK
22
k 2 min
d,
min |K
Iggi
(majority voting)
(4)
12
(Σ 79) JEK
JEK
Lemma 3.2. Given disjoint models. Fi,.... Fr,, having gradients g1, .gn Rd for input- label pair (x. 8) (where gi VxCx, (x. (x.y)). the maximum number lo of orthogonal vectors
11,12: r2. Ik E gd satisfying ||rıllso I and Eggg Tri] 2 Y1 for all 1 S S n (at-least-one
voting) or for at least [2] models (majority voting), for all 1 1 kisk
(d. [d.1mm
(4|minn
( (211g1l1 1l n2n? (22
I)
k = min ==min
(at-least-one voting)
(5)
k = min (d, median
< <e n2
gn
I)
(majority voting)
(6)
n2
Implications. If all n disjoint models in the ensemble are "near-identical" (as expected per our gg and 71 R3 R Tao then Lemma 3.1 | This implies that ensemble of
saliency partitioning scheme), i.e., ||g: dd, for at-least-one voting reduces to ie 2 min disjoint models offers potential reduction in dimensionality of the adversarial subspace by a factor of n compared any individual constituent disjoint model. Similar interpretation holds for Lemma 3.2, where reductions are now by a factor of n.
Furthermore, using Tramer al. 2017 's GAAS technique for dimensionality estimation, can show that individual disjoint models exhibit lower dimensionality as compared to that of a single model trained on the full frequency spectrum Additional empirical analysis of individual models in Section 4.3 confirms this. This implies that ensembling such disjoint models leads to even further reduction dimensionality of the adversarial subspace.
'Intuitively, this can be attributed to each disjoint model receiving fewer non- non-robust features, but continuing perform well due to the redundancy observation. Additional details for GAAS estimation can be found in Appendix A.2
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTAL EVALUATION
Our experiments broadly aim to answer the following questions:
Q1. What is the performance of D3 ensembles against white-box and black-box adversaries?
Q2. How does choice of partitioning scheme and ratio impact robustness?
Q3. What is the natural deepfake detection performance of D3 ensembles vs. a full-spectrum detector on unperturbed deepfakes?
In the following sections, we address these questions by detailing our setup and experiments.
4.1 EXPERIMENTAL SETUP
We adopt the following settings to evaluate D3 for detecting deepfakes and adversarial deepfakes.
Datasets and Pre-processing. We primarily perform our experiments using dataset comprising real, and -eN-generteed deepfake images of individuals. The real images are obtained from Flickr- Faces HQ (FFHQ) dataset of celebrity headshots (Karras et al.) 2019). Deepfake images are gen- erated using the state-of-the-arr StyleGAN, trained with standard hyperparameters on the 128x120 FFHQ. Specifically, deepfake images generated using the de facto approach of sampling points uniformly on a 512-5122200 dineensonai sphere, and feeding them through the generator DNN to obtain images of "people that do not exist". From each of the classes (deepfake and real), use 50,000 images for training, 10,000 for validation, and 10,000 for testing sets. Finally, we transform all deepfake images to their frequency space representation using the 2D-DCT function (Ahmed al. 1974). We also evaluate D3 using other datasets, including images from BigGAN (Brock et al, 2018 (conditional GAN trained ImageNet (Deng al.. 2009)), StarGAN (Choi al. 2020) (image-to-imal conditional GAN trained on CelebA-HQ (Karaas et al. 2018)). and other types of non-GAN generated deepfakes. ie.. FaceSwap and NeuralTextures deepfakes from the Face- Forensics++ dataset (Rössler et al. 2019). Additional details for these datasets can be found in Appendix A.3.3
D3 Architecture and Training. All models in our D3 ensembles implement the same shallow four- layer CNN binary classifier employed by the baseline Frank al. (2020) frequency-space ssspue defense. All models are adversarially trained with PGD-10 attacks (Madry et al. 2018) deepfake images, by minimizing the binary cross-entropy loss with the Adam optimizer and an initial learning rate of 0.001. We ROC over the validation set). All models are implemented using the PoTorch v1.6.0 framework for Python. Additional details adversarial training, including hyperparameters, can be found in Appendix I
batch size of 512 for maximum of 20 epochs (with early stopping based on AUC-
D3 Configurations. We consider two configurations of our ensemble that partition frequency com- ponents into 2 or 4 disjoint subsets, i.e., ensembles of size n = and n = 4, respectively. We select these values of n based on analysis presented in Section 4.3 We refer the following configurations in rest of the paper: (a) D3-R(2) and D3-R(4) for n 2 or n = 4 that use the random partitioning strategy; and (b) D2-S(2) and D3-S(4) for
or 4 that use the saliency partitioning strategy. Ensemble decisions are obtained via a voting function that computes the mean logits of n models.
Baselines. We consider adversarially trained (AT) version of Frank al. 2020 full spectrum frequency-space classifier, as well as several ensemble- based defenses against adversarial examples: GAL (Karyyppa& & Qureshi 2019), ADP (Pang et al., 2019), DVERGE (Yang et al.) 2020) as our baselines. TRS (Yang et al. 2021) performed poorly for deepfake detection we weren's 't able to get good natural performance (likely due to the regularizer not converging over the frequency space). For all baselines, we denote the ensemble size in parentheses, e.g., ADP(4). Additional baseline details are provided in Appendix A.3.
Metrics. Since deepfake detection is binary classification task, we AUC-ROC score to measure the natural detection performance of D3. AUC-ROC measures the area under the receiver-operator curve, and compares the true positive rate to the false positive rate for different classifier thresh- olds. We employ adversarial accuracy (fraction of correctly classified adversarial examples) as our performance metric for robustness. A higher adversarial accuracy implies more robustness.

---

Under review as a conference paper at ICLR 2023
Table 1: Adversarial accuracies for loo perturbation attacks. We present accuracies for all D3 configurations, for the AT (adversarial training) baseline, and for ensemble-based baselines ADP, GAL, and DVERGE (DV). Parentheses denote the number of models used for defenses and the number steps used for attacks.
Attacks
c
AT(1)
ADP(4)
GAL(4)
DV(4)
D3-R(2)
D3-S(2)
D3-R(4)
D3-S(4)
1/255
7.6
0.2
98.2
100.0
99.8
100.0
APGD-
4/255
0.2
0.0
0.0
21.5
97.8
100.0
99.2
100.0
CE(50)
8/255
0.0
0.0
0.0
21.5
97.2
100.0
98.4
100.0
16/255
0.0
0.0
0.0
0.0
6.0
96.2
83.8
100.0
4/255
100.0
99.4
96.6
71.5
99.6
99.0
100.0
99.5
16/255
100.0
99.4
72.8
69.0
99.6
99.0
100.0
99.5
FAB
32/255
61.6
93.2
0.2
32.6
98.0
99.5
99.5
64/255
0.0
4.8
0.0
0.0
0.0
5.8
0.0
9.5
4/255
99.8
96.4
96.0
28.0
99.4
98.2
97.0
16/255
92.0
66.6
51.0
28.0
96.8
98.2
99.4
96.6
Square
32/255
25.8
13.6
6.6
21.0
50.8
94.8
88.0
96.2
64/255
0.0
0.4
0.0
0.5
0.4
45.2
4.0
95.0
Table 2: Adversarial accuracies for & perturbation attacks for same configurations as in Table
Attacks
AT(1)
ADP(4)
GAL(4)
DV(4)
((2
D3-S(2)
D3-R(4)
D3-S(4)
25.0
45.5
98.4
100.0
99.8
100.0
APGD-
16.0
1.8
1.8
24.5
97.2
100.0
98.9
100.0
CE(50)
5
0.0
0.0
0.0
21.5
96.0
100.0
98.2
100.0
10
0.0
0.0
0.0
0.0
1.2
76.2
63.6
100.0
10
99.4
99.6
39.6
67.5
97.6
99.0
100.0
100,0
20
20.4
81.0
0.0
64.5
0.8
99.0
54.5
98.5
FAB
0.0
0.0
80
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
99.0
94.4
94.0
13.0
99.0
100.0
99.4
20
83.8
79.8
65.6
10.0
98.2
99.0
99.8
99.6
Square
40
33.2
32.0
6.4
7.5
56.4
97.0
86.2
99.6
80
1.0
0.0
0.0
0.0
0.0
22.2
2.4
95.8
4.2 ROBUSTNESS AGAINST ADVERSARIAL EXAMPLES
We evaluate robustness D3 and baselines against attacks in both the white-box and black-box settings, and find that D3 is robust than baselines. All attacks crafted using the AutoAttack benchmark, which is widely accepted as an evaluation standard for adversarial examples (Croce & Hein 2020b). White-box attacks include the adaptive step-size PGD attacks using cross-entropy (APGD- CE) and Carlini-Wagner Wagner (APGD-CW) loss functions as well as the Fast Adaptive Boundary (FAB) attack. Black-box attacks include Square attack (Andriushchenko al. 2020).
Table presents adversarial accuracies under different foo perturbation budgets and find that D3 (specifically, the D3-S(4) configuration) significantly outperforms all baselines under white-box set- tings. For example, D3-S(4) achieves 99.5% for attack budgets of 8/255 vs. 21.5% for clos- est performing baseline DVERGE. This performance gap tends to widen under large perturbation budgets. Similar trends hold under the Square black-box attack; for example. D3 achieves 95.0% adversarial accuracy vs. 0.5% for the best baseline on a perturbation budget of e = 64/255.
Table 2 presents adversarial accuracies under different le perturbation budgets, and we find similar trend D3 outperforming baselines by significant margin (e.g., 100% accuracy vs. 21.5% for DVERGE 5).
Table present results for & and loo attacks where we increase the number of iteration steps in the white- box APGD-CE attack. While D3's accuracy does drop to 61% under 1000 attack steps, continues to outperform all baselines (which unanimously drop to 0%). Ieguontademossstttes that above trends continue hold under increased perturbation budgets.

---

Under review as a conference paper at ICLR 2023
ATC] ADPW)
GMlef (NV1)
D3422
B3HR2) D3--5/1
Sallency Partteming
Random Purtiorinng
Adversatial Accuracy
D3.92
--
Adversaria Accuracy
1000
100
80
60-
46-
20
20
1.0 Epsilon
20
40
Epsilon
Fraction spectrum
Fraction spectrum
Figure 3: Adversarial accuracy of D3 and base- lines perturbation budget increased (&2 on the left, foo on the right).
Figure Expected adversarial accuracy (lz on the left, lo on the right) of individual models different partitioning ratios E [0, 100].
Table 3: Adversarial acc. for APGD-CE attack at la Loo 8//255 under increasing steps.
8
Steps
AT(1)
ADP(4)
GAL(4)
DV(4)
D3-R(2)
D3-S(2)
D3-R(4)
D3-S(4)
50
16.6
2.5
1.5
24.5
100.0
100.0
100.0
100.0
500
0.0
0.0
0.0
21.5
63.0
98.6
99.0
100.0
1000
0.0
0.0
0.0
0.0
34.0
85.5
94.5
100.0
50
0.0
0.0
0.0
21.5
98.0
100.0
100.0
100.0
500
0.0
0.0
0.0
18.5
0.0
2.0
49.0
92.0
1000
0.0
0.0
0.0
0.0
0.0
0.5
8.0
61.0
We also evaluate D3 on other GAN-generated datasets (BigGAN and StarGAN) and find a similar trend: it significantly outperforms baseline deepfake detection methods. Additional details can be found in Appendix A.3.3
Finally, we find that naive attack strategies such as transfer attacks from a full full-spectrum model do poorly against D3. Additional details can be found in Appendix/A.3..
Summary. We find for Q1 that D3 ensembles offer significant robustness improvements against both white-box and black-box attacks crafted using the popular AutoAttack benchmark (Croce & Hein 2020b) that tries multiple state-of-the-art attacks on models. For both for and l perturbation norms, D3 continues to exhibit high adversarial accuracies (e.g., > 99.5% for foc, 32/255) where existing defenses fail. D3 continues to offer accuracies greater than 61% (Eo,, E = 8/255) and even 100% (l2, 1) when attack steps are increased to 1000, setting where existing state- of-the-art defenses are reduced to 0% accuracy.
4.3 IMPACT OF PARTITIONING SCHEME ON ROBUSTNESS
In Section observed that ensembles constructed using our saliency partitioning strategy (D3-S(2) and D3-S(4)) demonstrated higher adversarial accuracies than those using our random partitioning strategy (D3-R(2) and D3-R(4)) under all attacks. Both strategies outperformed the full- spectrum detector. We explain these results by examining the robustness of the individual constituent models under both strategies.
Figure f plots the expected adversarial accuracy over the individual models in the ensemble for both schemes (using the same APGD-50 attack on 1000 images for Eg = &oo 4/255). We observe that all the individual models (looking at 50% and 25%) are more robust than the full-spectrum (100%) detector, thus satisfying the conditions in Lemmas 3.1 and 3.2 For individual models looking at 12.5%, this is not the case (see high standard deviations). Additionally, individual models produced using saliency partitioning are more robust than those using random (due to proportionate distribution of robust/non-rooust frequencies), suggesting further reduction in adversarial subspace
8

---

Under review as a conference paper at ICLR 2023
dimensionality when ensembling. Finally, arbitrarily partitioning beyond 25% does not improve robustness of individual models, suggesting diminishing gains beyond an ensemble of = 4 models.
Summary. We find for Q2 that individual disjoint models obtained via saliency partitioning significantly more robust than a full spectrum model. These improvements still exist for random partitioning, but not large. Further, individual models with partitioning ratios of 50% and 25% outperform a full spectrum trained model, with ratios beyond 25% not improving robustness.
4.4 NATURAL DEEPFAKE DETECTION PERFORMANCE
As per Table we find that all D3 configurations (even D3-R(4) and D3-S(4) where individual models use 25% of frequency components) achieve AUC-ROC scores over 99%. In other words, there is no loss in natural performance when compared to the full spectrum frequency classifier al. 2020).
Type
AT(()
D3-R(2)
D3-S(2)
D3-R(4)
D3-S(4)
Frequency
99.99
99.96
99.98 96.45
99.93
99.95
Pixel
97.59
90.42
90.11
93.83
Table 4: Natural deepfake detection AUC-ROC scores of D3 full-spectrum detector AT(1).
We emphasize the importance of frequency-space redundancy in enabling our partitioning scheme, allowing D3 to achieve these scores. To this end, we also compute scores obtained from ensembles that do not apply DCT images, i.e., D3 ensembles in the pixel space. Each of these ensembles exhibit considerably lower performances as artifacts are not evenly distributed throughout the pixel space. We refer the reader to Figure 2 to further visualize this effect.
Summary. We find for Q3 that D3 ensembles preserve the natural accuracy of a full-spectrum model we observe reductions of less than 1% AUC-ROC score. This comparable performance can be attributed to the redundancy of relevant artifacts in the frequency space. In contrast, ensembles with partitioning in the pixel space suffer from drops of 8% scores.
5 LIMITATIONS AND FUTURE WORK
Societal impact and limitations. Modern deepfakes raise several societal and security threats; D3 step towards mitigating that. Nonetheless, adversarial deepfakes also have benign use cases, e.g., anonymization of an end-user on an online network; D3 could prevent such anonymization. Additionally, D3 is focused on GAN -AN-generated deepfake images - since relies upon redundancy in the frequency space, it may not be effective against against other types of deepfakes that avoid these artifacts. We believe that the benefits of D3 outweigh such potential concerns.
Applicability to domains other than deepfake detection. We presented D3 ensemble frame- work for adversarially robust deepfake detection. However, we hypothesize that this approach could be applied to any classification task that exhibits redundancy in feature space. While aware such a space for the popular CIFARIO and ImageNet classification tasks (see Table / in Appendix A.3.44 for preliminary results CIFARIO 10 using the DCT), there are several classification tasks in, say, the audio domain that exhibit redundancy in features, e.g., keyword spotting and fake speech detection. Exploring this hypothesis is interesting future research direction.
CONCLUSIONS
In this paper, we present D3, an ensemble approach to deepfake detection that exploits redundancy in frequency feature space by partitioning the frequencies across multiple models. We show theoretical advantages to such disjoint partitioning of input features, that it reduces the dimensionality of the adversarial subspace. We empirically validate that D3 offers significant gains in robustness under multiple adaptive white-box and black-box attacks, maintaining high adversarial accuracy where best-performing baselines drop below 20%.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nasir Ahmed, T- Natarajan, and Kamisetty R Rao. Discrete Cosine Transform. IEEE Transactions on Computers, 1974.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square At- tack: query-efficieet black-hox adversarial attack via random search.. In Proceedings of Euro- pean Conference on Computer Vision, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In Proceedings of International Conference Machine Learning, 2018.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smnić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion Attacks Against Machine Learning Test Time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1006, 2018.
Nicholas Carlini and Hany Farid. Evading deeppfake-imaee detectors with white-and black-box at- tacks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 658-659, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Samantha Cole.
Ai-assisted fake porn is here and we're all f****d. d. htts://www.iice. com/ en mt__us/artiille/gggeee
-aa-aggtttaaakeeee a fake-----oo 2017.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference Machine Learning, pp. 2196-2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter-free attacks. International conference on machine learning, pp. 2206- 2216. PMLR, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Ricard Durall, Margret Keuper, Franz-Josef Pfreundt, and Janis Keuper. Unmasking deepfakes with simple features. arXiv preprint arXiv: 1911110066, 2019.
Himanshu Dutta, Aditya Pandey, and Saurabh Bilgaiyan. Ensembledet: ensembling against adver- sarial attack deepfake detection. Journal of Electronic Imaging, 3066):063030, 2021.
Ryan Feng, Neal Mangaokar, Jiefeng Chen, Earlence Fernandes, Somesh Jha, and Atul Prakash. Graphite: Generating automatic physical examples for machine-learning attacks on computer vision systems. arXiv preprint vrXiv:2002.07088 2022.
Steven Lawrence Fernandes and Sumit Kumar Jha. Adversarial attack on deepfake detection using based texture patches. In European Conference on Computer Vision, pp. 220-235. Springer, 2020.
Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In International Confer- on Machine Learning, pp. 3247-3258. PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Apurva Gandhi and Shomik Jain. Adversarial perturbations fool deepfake detectors. In 2020 Inter- national joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015. URL http://
arxiv.org/ absg/abs/144/ 6572
Luca Guarnera, Oliver Giudice, and Sebastiano Battiato. Deepfake detection by analyzing con- volutional traces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 666-667, 2020.
Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In 11th {USENIX] workshop offensive technologies ([WOOT) 17), 2017.
Shehzeen Hussain, Paarth Neekhara, Malhar Jere, Farinaz Koushanfar, and Julian McAuley. Ad- versarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3348- $357,2021.
Shehzeen Hussain, Todd Huster, Chris Mesterharm, Paarth Neekhara, Kevin An, Malhar Jere, Harshvardhan Sikka, and Farinaz Koushanfar. Reface: Real-time adversarial attacks on face recognition systems. arXiv preprint arXiv:2206.0444 0478833 2022a.
Shehzeen Hussain, Paarth Neekhara, Brian Dolhansky, Joanna Bitton, Cristian Canton Ferrer, Julian McAuley, and Farinaz Koushanfar. Exposing vulnerabilities of deepfake detection systems with robust attacks. Digital Threats: Research and Practice (DTRAP), 3(3):1 1-2,, 2022b.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances neural information processing systems, 32, 2019.
Jianguo Jiang, Boquan Li, Shuai Yu, Chao Liu, Shaohua An, Mingqi Liu, and Min Yu. A residual fingerprint-based defense against adversarial deepfakes. In 2021 IEEE 23rd Int Conf on High Performance Computing & Communications: 7th Int Conf on Data Science & Systems: 19th Int Conf on Smart City; 7th Int Conf on Dependability in Sensor, Cloud Big Data Systems & Application HMPCC//S//SaarCCyyDDeeeaad
pp- 797-804. IEFE, 2021.
Sanjay Kariyappa and Moinuddin Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901. 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4401-4414 2019.
Davis E King. Dlib-ml: A Machine Learning Toolkit. JMLR, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Quanyu Liao, Yuezun Li, Xin Wang, Bin Kong, Bin Zhu, Siwei Lyu, Youbing Yin, Qi Song, and Xi Wu. Imperceptible adversarial examples for fake image detection. arXiv preprint 2021.
11

---

Under review as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Lear- ing Representations, 2018. URL URLhtttsat its / (tw.rmmmmddd..1fa....
net
Francesco Marra, Diego Gragnaniello, Davide Cozzolino, and Luisa Verdoliva. Detection of gan- generated fake images over social networks. In 2018 IEEE conference on multimedia information processing and retrieval (MIPR). pp. 384-389. IEEE, 2018.
Francesco Marra, Cristiano Saltori, Giulia Boato, and Luisa Verdoliva. Incremental learning for the detection and classification of gan-generated images. In 2019 IEEE international workshop information forensics and security (WIFS), pp. 1-6. IEEE, 2019.
Paris Martineau. Facebook removes accounts with ai-generated profile photos. https://www removes accounts al generated osotoss/
wi wired. wwt.//oory/aceeeooooooooo/o/
2019.
Scott McCloskey and Michael Albright. Detecting gan-generated imagery using color cues. arXiv preprint urXiv:/8/2.08247, 2018.
Zhongjie Mi, Xinghao Jiang, Tanfeng Sun, and Ke Xu. Gan-generated image detection with self- attention mechanism against gan generator defect. IEEE Journal of Selected Topics in Signal Processing, 14(5):969-9811 2020.
Lakshmanan Nataraj, Tajuddin Manhar Mohammed, BS Manjunath, Shivkumar Chandrasekaran, Arjuna Flenner, Jawadul H Bappy, and Amit Roy-Chowdhury. Detecting gan generated fake images using co-occurrence matrices. Electronic Imaging, 2019(5):532-1, 2019.
Paarth Neekhara, Brian Dolhansky, Joanna Bitton, and Cristian Canton Ferrer. Adversarial threats to deepfake detection: A practical perspective. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 923-932, 2021.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pp. 4970-4979. PMLR, 2019.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics+:: Learning to detect manipulated facial images. In International Con- ference Computer Vision (ICCV), 2019.
Shaikh Akib Shahriyar and Matthew Wright. Evaluating robustness of sequence-based deepfake detector models by adversarial perturbation. In Proceedings of the Workshop on Security Implications of Deepfakes and Cheapfakes, pp. 13-18, 2022.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http:::pww.....og/////o abs/aa/s/.11.. 6199
Shahroz Tariq, Sangyup Lee, Hoyoung Kim, Youjin Shin, and Simon S Woo. Gan a friend or foe? framework to detect various fake face images. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, pp. 1296-1303, 2019.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 1705.07244, 2017a.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint (0ti::/7000345555 2017b.
Ngan Hoang Vo, Khoa D Phan, Anh-Duy Tran, and Duc-Tien Dang-Nguyen. Adversarial attacks deepfake detectors: A practical analysis. In International Conference on Multimedia Modeling, pp. 318-330. Springer, 2022.
12

---

Under review as a conference paper at ICLR 2023
Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn Cun-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8695-8704 2020.
Huanrui Yang, Jingyang Zhang, Hongliang Dong. Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: diversifying vulnerabilities for en- hanced robust generation of ensembles. Advances in Neural Information Processing Systems, 33: 5505-5515, 2020.
Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. Tis: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 2021.
Ning Yu, Larry S Davis, and Mario Fritz. Attributing fake images to gans: Learning and analyzing gan fingerprints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7556-7566, 2019.
Xu Zhang, Svehor Karaman, and Shih-Fu Chang. Detecting and simulating artifacts in gan fake images. In 2019 IEEE International Workshop Information Forensics and Security (WIFS), pp. 1-6. IEEE, 2019.
A APPENDIX
In Appendix I. we provide proofs for our bounds on dimensionality of the adversarial subspace of estimates the dimension of the adversarial
disjoint ensembles, such D3. Dopendix/A.2emmiireee subspace using combinatorial construction of orthogonal perturbations. In Appendix [A.3] we provide details of our baseline configurations and attack settings for reproducibility purposes, as well as additional experimental results on other datasets. Finally, in Appendix A.5 we provide additional insight and motivation for our saliency frequency partitioning scheme.
A.1 PROOFS
A.1.1 ORTHOGONAL GRADIENTS
Prior to providing proofs for the adversarial dimensions, we demonstrate that gradients for disjoint classifiers always orthogonal to each other. We will this for later results. Given input space X' and class-label space y. we have n disjoint classifiers Fn. If is the DCT transfor- mation matrix, we can define to be the transformation matrix for the classifier F,. Each disjoint transformation T, has a lot of zeros. Only the rows corresponding to the unmasked frequencies of classifier J, have non-zero entries. Moreover, since no frequency is shared by any two classifiers, the jth row will have non-zero entries in exactly one of the disjoint transformation matrices, i.e. T.TJ O Vi j.
Next, the disjoint classifiers Fn. where F; Txx y. are trained using loss functions f.,, fr, respectively. Now, the dot product between the gradients of classifiers F, and J, is given by
(VxLr:) (VxLr,) = (Tx + (T) (VTxLF,) TTT (VT,xCz)
(T,x(o,)
A.1.2 PROOF LEMMA 3.1
From Tramèr et al. 2017b)) we know that for a classifier F X - y where X' E Rd is the input space and y E Z is the finite class label space, the dimension of the adversarial subspace around input-label pair (x, y) where x E X and E y, is approximated by the maximal number of orthogonal perturbations F1; r2, ... Fk such that ||ri||2 l S and grr; 2 ? A < k. Here,
13

---

Under review as a conference paper at ICLR 2023
= Vx (x) y) and is the increase in loss function L sufficient cause mis-classification. Tramer et al. (017b) provide a tight bound for k:
e Elg
min d.
(8)
722
Σ
We now extend this result for n disjoint classifiers. Let g' Now, for at-least-on voting,
Bj
rr
Σ
71
r
AisiSk
π
n
Applying the result from Tramer et al. (017b) (Equation 8 on the above inequality (Equation 9 we get:
2
[|'
min
d
2
Σ
71
(10)
2
Σ
gi
min
d,
==1
(since gi [g Vi + j. using Equation 7
%
Now, for majority voting, we again apply the results from Tramèr et al. (017b)) (Equation g) How- ever, the derivation now depends on the selection of models that the adversary chooses target. To obtain the lower and upper bounds, we can select n with the most and least adversarial dimen- sions respectively. Following similar derivation as before, get:
2
g
k 2 min
d,
min |k|=1
(11)
z2
Σ 7
8 2 Σ
gg
k < min
d,
max
1=1
(12)
|
Σ 7
A.1.3 PROOF OF LEMMA 3.2
Follow up work from Tramèr et al. (2017a) also provides a tight bound for the adversarial dimension in the loo case. They provide tight bound for the number of orthogonal perturbations Fi, rk Rd such that ||rilloo c, given by sign(g)Tr; VE gradient.
ed V1 < i < k where sign(g) is the signed
14

---

Under review as a conference paper at ICLR 2023
Ri since gj s are non-zero only on sign(g)) Vr e Rd. Applying the
We now extend this result for n disjoint classifiers. For g' non-overlapping dimensions, we can see that sign(g')*r= above results here, we get
n
ed
Σ (ig Tri
v1 sisk
(13)
Vk
Now, similar to Tramer et al. (2017a), we compute the perturbation magnitude along random permutation of the signed gradient. For each 1 S S and 1 S < k, we get
E rr
E
19
(p)
sign(o;;) (p)) -r(P))
193
(p)
E
( -11)))
(14)
p=1
gill
nvh
A.2 ESTIMATING DIMENSION OF ADVERSARIAL SUBSPACE
Tramer al. 2017al combinatorial construction find & orthogonal adver- sarial directions for a full spectrum (100%) model as well as individual models that use fraction of the frequency features. The frequency features selected using the saliency partitioning scheme (Section 3.2)
In this section,
We construct orthogonal perturbations by multiplying the vectors from a Regular Hadamard matrix component-wise with sign(g) where is the model gradient. We consider 3 settings, based on fraction of frequency features 100%, 50% and 25%. For each setting, correctly classified inputs, and compute loss gradients for each input. Using a regular Hadamard Matrix of order w, the above construction gives w orthogonal directions. Out of these w, the number of directions that are actually adversarial provide an estimate for k. We run the analysis for (4, 16, 36, 64, 100}, and select the maximum value.
then select 1000
Figure plotos the proportion of images (out of 1000) with at least & adversarial dimensions. We ob- serve that both the 50% and 25% settings have fewer images with high dimension the adversarial subspace. These results support the analysis done in Section 3.3 Further, the reduced adversarial dimension for the 50% and 25% settings consistent with their improved robustness over the full spectrum (100%) model (Section 4.3
0.40
100%
Proportion of Images
0.35
50%
0.30
25%
0.15
0.10
0.05
0.00
80
100
Figure Distribution adversarial saliency magnitudes for frequency components.
15

---

Under review as a conference paper at ICLR 2023
A.3 ADDITIONAL EVALUATION DETAILS
A.3.1 DETAILS FOR BASELINES
In this section, we provide additional details for all models/ensembles used for the baseline evalua- tion. We implement all the baselines using the four-layer CNN employed by Frank et al. (2020).
Adversarial-Trass Trainnig (Marry et al. 2018). We employ adversarial training, which involves train- ing the CNN adversarial examples for deepfake images only (not real images), Adversarial ex- amples are constructed using the PGD-10 10 attack (i.e., 10 steps of gradient descent), with 1/255 and step size of 0.001. We train for 20 epochs with batch size of 512.
GAL (Kariyappa & Qureshi 2019). This approach minimizes transferability via a regularizer. Specifically, GAL minimizes the cosine similarity between individual models' loss gradients. We set the coefficient for standard cross-entropy loss as 1, and for the cosine loss as 0.5. We train for 35 epochs with batch size of 128.
ADP (Pagg et al., 2019). This approach also minimizes transferability via regularizer. The primary term ADP's regularizer maximizes the volume spanned by individual model's non- prediction output vectors, i.e., the vectors obtained by removing the top-confidence class, so as to maximize diversity without affecting accuracy. We set coefficients for standard cross-entropy loss I, for the volume term in the regularizer as 0.5, and for the entropy term in the regularizer as We train for epoch with a batch size of 128.
DVERGE (Yang et al. 2020). DVERGE minimizes transferability by adversarially training indi- vidual models on "adversarial features" constructed against other models in the ensemble. Adversar- ial features constructed by minimizing novel ceaturedistillation objective over 10 steps, with e 0.07 and step size as 0.007. We train for epochs with a batch size of 128.
A.3.2 DETAILS FOR ATTACKS
We consider the following attacks for our evaluation. These attacks constitute the entire ensemble of attacks used in the AutoAttack Benchmark. We tune the attacks where necessary to get the strongest attack setting.
APGD-CE/CW (Croce & Hein 2020b) is a step-size free variant of the standard PGD attack. It adjusts the step size during the attack based on the convergence of the loss and the overall perturba- tion budget. We optimize the adaptive PGD-attack on the Cross Entropy (CE) and Carlini Wagner (CW) loss functions. We use the same set of parameters for the attack as mentioned in AutoAttack Benchmark other than the step size decay parameter which set to 0.1 instead 2.
FAB (Croce & Hein 2020a) is a iterative first order attack that utilizes geometry of the decision boundary to minimize the perturbation required to cause mis-classiffcation. We use the same set of parameters as AutoAttack.
Square AAdriushchenko et al., 2020 is an efficient black-box attack that uses random square shaped updates to approximate the decision boundary. We use the same set of parameters as Au- toAttack..
We also include Table results for a "baseline" attack, in which we evaluate D3 on adversarial examples computed against full-spectrum classifier. As expected, such adversarial examples do not evade D3.
f.o
0.5
1
5
10
20
1/255
4/255
8/255
16/255
32/255
Accuracy
100.0
100.0
100.0
100.0
1.4
100.0
100.0
100.0
100.0
49.1
Table Adversarial accuracy of D3-S(4) under adversarial examples crafted against a full-spectrum detector.
16

---

Under review as a conference paper at ICLR 2023
A.3.3 EVALUATION ON OTHER DATASETS
BigGAN. The BigGAN dataset comprises images GAN-generated images belonging to the 1000 classes in the ImageNet (Deng et al. 2009) dataset. Images generated using BigGAN (Brock al. 2018) trained on the 256x256 ImageNet dataset of 1000 object and animal classes. Images are resized to 128x128. We 35,000 images for training. 7,500 images for validation, and 7,500 for testing sets. Corresponding real images are obtained from the ImageNet dataset.
Table BigGAN adversarial accuracies for Loo and Ez perturbation attacks for same configurations as in 2
Attacks
AT(I)
ADP(4)
GAL(4)
DV(4)
D3-S(4)
200
1/255
16.2
9.1
41.5
5.5
91.0
APGD
4/255
14.4
9.1
9,4
0.0
90.0
CE(50)
8/255
14.4
9.1
9.4
0.0
78.3
16/255
14.4
9.1
9.4
0.0
26.6
1/255
15.9
9.1
2.3
5.4
91.0
APGD- CW(50)
4/255
14.4
90.5
8/255
14.4
9.1
0.0
79.6
16/255
14.4
9.1
9.4
0.0
26.7
82
0.1
13.6
91.0
APGD-
1
16.6
9.1
27.1
2.8
91.0
CE(50)
5
14.4
9.1
9,4
0.0
70.5
10
14.4
9.1
9.4
0.0
26.2
0.1
25.1
9.3
61.6
14.5
91.0
APGD- CW(50)
1
15.4
9.1
27.4
3.1
91.0
14.4
71.0
10
14.4
9.1
9.4
0.0
26.2
StarGAN. The StarGAN dataset comprises images GAN-generated images belonging to the 1000 classes in the ImageNet (Deng al., 2009) dataset. Images are generated using StarGAN (Choi et al. 2020) trained on the 256x256 CelebA-HQ dataset (resized from 1024x 1024) of celebrity head- shots Karras al. 2018). Since StarGAN is an image-to-Image translation GAN that changes the "style" 1.c., hairstyle, le, eye color, etc of an existing real image, we generate images using randomly sampled styles. Images are resized 128x128. We 21,000 images for training. 4,500 images for validation, and 4,500 for testing sets. Corresponding real images are obtained from CelehA-HQ.
FaceForensics** (DeepFakes and NeuralTextures.) This dataset comprises the DeepFakes and NeuralTextures subsets of the FaceForensicst+ dataset (Rössler et al., (011)). These are face-swap deepfake videos generated using neural, non-GAN techniques, and are thus not the primary focus of this work. Nonetheless, we include preliminary evaluation of natural performance on these videos. We extract faces from each video frame using the dlib library (King, (2009). Face images are resized to 128x128. We use 6,265 images for training, 1.342 for validation. and 1,343 for testing.
Other ensemble approaches such as EnsembleDet (Dutta al. 2021) might work better on non- GAN deepfakes such as those from FaceForensicst+. as they different architectures. instead of relying upon redundancy. Unfortunately. source code for EnsembleDet unavailable, and their official evaluation only considers attacks such FGSM and BIM, which are known to be weaker than AutoAttack, making direct comparison difficult.
A.3.4 EVALUATION ON CIFAR10
Table presents results for attacking the D3-S(4) ensemble and AT baseline, for the CIFARIO classification task. While D3-S(4) offers some improvements the smallest perturbation budgets, quickly drops off. We suspect that a more carefully chosen feature space with redundancies for animal/vehicle classification would improve these results.
17

---

Under review as a conference paper at ICLR 2023
Table StarGAN adversarial accuracies for Loo and & perturbation attacks for same configurations as in Table 2
Attacks
AT(1) ADP(4)
GAL(4)
DV(4)
D3-S(4)
20
1/255
14.6
18.5
100.0
APGD-
4/255
0.0
0.0
0.0
3.0
99.6
CE(50)
8/255
0.0
0.0
0.0
0.0
0.1
16/255
0.0
0.0
0.0
0.0
0.1
1/255
11.5
17,4
2.3
19.3
100.0
APGD-
4/255
0.0
0.0
2.7
99.8
CW(50)
8/255
0.1
16/255
0.0
0.0
0.0
0.0
0.1
82
0.1
87.5
25.3
100.0
APGD-
1
9.5
0.0
0.9
11.6
100.0
CE(50)
5
0,0
0.0
0.0
2.7
0.1
10
0.0
0.0
0.0
2.7
0.1
0.1
20.9
90.8
16.2
19.3
100,0
APGD-
1
3.3
0.1
0.3
5.8
100.0
CW(50)
0.0
2.7
0.1
10
0.0
0.0
0.0
2.7
0.1
Dataset
AT(1)
D3-S(4)
DeepFakes
98.8
90.6
Texturss
94.6
80.8
Table 8: Natural accuracies D3 and a full-spectrum classifier on non-GAN generated deepfakes from the FaceForensics++ dataset.
Table 9: CIFAR10 adversarial accuracies for foo and l perturbation attacks for same configurations as in Table 2
Attacks
AT(1)
D3-S(4)
1/255
1.7
59.8
APGD-
4/255
1.2
CE(50)
8/255
0.1
0
16/255
0.1
0
82
0.5
15.1
63.5
APGD-
13.6
39,7
CE(50)
0.9
8.2
10
0.8
6.2
A.4 PERTURBATION REQUIRED FOR SUCCESSFUL ATTACKS
A.5 UNDERSTANDING FRBQUENCY PARTITIONING
Recall that each model in our ensembles uses subset of frequencies. However, per Ilyas et al. 2019 robust features model, these frequencies can likely be categorized as either robust, or non- robust. We would like each model receive equal number of robust frequencies. In Section 3.2 we discussed two schemes for disjoint partitioning frequency components a more "naive random partitioning, and our proposed saliency partitioning. To better visualize how saliency par-
18

---

Under review as a conference paper at ICLR 2023
Unperturbed Deepfake
AT (l2)
D3-S(4) (42)
AT (loo)
D3-S(4) (Exo)
6 =
15
6 0.01
6 0.15
15
0.01
0.15
c=
15
0.01
0.15
Figure 6: Minimum Perturbation required for successful attacks
titioning achieves our goal of equal distribution, Figure 7 plots a histogram of adversarial saliency magnitudes. Recall that these saliencies are computed using the gradient (and perturbation) magni- tude at the adversarial example, i.e., our adversarial saliencies are a heuristic measure of "robust- ness" of each frequency component, with higher saliencies implying less robustness.
8000
7000
6000
5000
CDF
4000
3000
2000
1000
0.0
0.2
0.4
0.6
0.8
1.0
Saliency Magnitudes
Figure Distribution adversarial saliency magnitudes for frequency components.
19

---

Under review as a conference paper at ICLR 2023
is clear that there exists a large number of low-saliency (or highrroobussess))) frequency components with magnitudes < 0.2. If a majority of these components were allocated to single model, only components with saliency magnitudes > 0.4 would remain for the other models. Our saliency partitioning scheme thus aims to ensure that each model in the ensemble receives a fair proportion of high-robustness components with magnitudes < 0.2.
The above analysis also leads us to ask - which frequency components are of high-saliency (low- robustness), and why? To this end, visualize saliency magnitudes across all channels in Figure
0
0.25
1.0
0.125
31
0.8
31
0.100
0.075
0.10
0.4
127
127
127
(a) R-channel
(b) G-channel
(c) B-channel
Figure 8: Per-channel heatmaps of adversarial saliency magnitudes. Note the grid-like pattern, matching the pattern of artifacts used for deepfake detection itself.
We observe that low-saliency (high-robustness) components are largely those constitute the grid-
like artifact pattern crucial to deepfake detection itself (see Section 3.1). All remaining components appear be high-saliency (low-robustness), Additionally, the G-channel appears comprise the most low- www-robustness components.
20


Reviews:
[{"review_id": "kscbI5ObNdI", "reviewer": "Reviewer_9j5U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on learning a robust deepfake detector against adversarial attacks. The authors propose a method named Disjoint Deepfake Detection (D3). The method is motivated by the observation that redundancy exists in the frequency space, which means fewer signals are enough to make correct predictions. Based on this, the authors divide the DCT spectrum of an image into disjoint partitions and adopt an ensemble method to adversarially train each individual model. The authors show clear performance improvement, especially on the dataset of FFHQ with the deepfake generation by StyleGAN.", "summary_of_the_review": "In summary, I vote for the rejection of this manuscript, due to the insufficient literature survey, ambiguous experiment details, and unclear result analysis.", "strength_and_weaknesses": "**Strength**\n\n[1] This paper studies a less explored task, to robustify the deepfake detector. It should have practical needs and is quite important.\n\n[2] The method is motivated by the experimental observation that redundancy exists in the frequency space, especially for deepfake images.\n\n[3] Based on the observation, the ensemble model is natural to be considered.\n\n[4] The authors shared some theoretical analysis of the proposed ensemble method, though I didn't check the proof exactly and am not sure if it is solid.\n\n**Weakness**\n\n[1] Incomplete literature survey. Missing important related work. The authors claimed \"Defending against adversarial examples, in general, has been shown to be a difficult task (Athalye et al., 2018), and is unexplored in the deepfake detection setting.\" However, there are published papers aiming at the adversarial defense of deepfake detection. For example:\n\n*Gandhi, Apurva, and Shomik Jain. \"Adversarial perturbations fool deepfake detectors.\" 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.*\n\nIn this IJCNN paper, the authors explored two defensive mechanisms to deepfake detectors: (i) Lipschitz regularization, and (ii) Deep Image\nPrior (DIP). Thus, the statement \"Defending against adversarial examples is unexplored in the deepfake detection setting\" is inaccurate.\n\n[2] Overclaim. \"We propose D3, the first deep fake detection framework designed to be adversarially robust.\" This is an overclaim, in consideration of my last comment.\n\n[3] Missing details in experimental settings. The descriptions on the dataset are vague. The authors claimed, \"We use 50,000 images for training, 10,000 for validation, and 10,000 for testing sets\". Among these, how many are real images, and how many are deepfake images? I assume it is possible that the authors first collected 70000 images from the FFHQ dataset, and then generated a deepfake image for each by StyleGAN. Also, it is not clear how to generate deepfake images by StyleGAN in this paper. The goal of deepfake detection is not clear either. Is it going to output a 0/1 binary value to verify if an image is fake or not, or output a detection region to indicate the deepfake? This relates to the understanding of the evaluation metric AUC-ROC used in this paper, but with few descriptions of its details. This brings significant difficulty in understanding the experiments and justifying the performance of the proposed method. \n\n[4] Worse performance on FaceForensics++ dataset. In Table 8, I assume the authors are reporting accuracies (unfortunately no descriptions, and this table is even unreferred in the main text). The performance of the proposed D3 is significantly worse than the adversarial training method AT(1). Again, it seems not mentioned which types of AT the authors are using, by Madry's, or else? In a word, does this experiment indicate the proposed method only works on specific datasets, like FFHQ, but not working on video dataset FaceForensics++?\n\n[5] This paper aims to solve a deepfake detection problem, in the adversarial setting, with datasets of human faces. But unfortunately,  almost no figures are shown in this paper to show the task. Only an image of Obama is given in Figure 1 to show the method pipeline. I am still confused after reading the whole paper, that what does the deepfake image look like? And which part of the deepfake images are detected as a clue for final judgment? Only some tables and curves are shown, but some of them lack detailed descriptions as I mentioned above.\n\n[6] In Tables 1,2,3, plenty of 100%s are shown for the proposed method. This is very different in general image classification datasets, like CIFAR-10. It is hard for state-of-the-art adversarial defense methods to reach 60%, even with very large networks like WideResNet. I am not saying the proposed results are too good to be true, but I am curious if the authors could give some insights behind these perfect results, especially on FFHQ images. In comparison, the baselines are extremely worse, even worse than a random guess I assume. Does this indicate a stronger baseline should be considered?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UQuJKOHCu9", "reviewer": "Reviewer_h9Bn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an ensemble approach to deepfake detection based on a frequency-domain decomposition of input images, such that each model in the ensemble is likely to see fewer non-robust features. The authors offer a theoretical justification in terms of a reduction in the dimensionality of the adversarial subspace, building upon the work of Tramèr et al. (2017). Evaluation using GAN-generated images in both the white- and black-box settings, leveraging the AutoAttack benchmark, shows significant improvements compared to SOTA.", "summary_of_the_review": "The paper is missing important discussion of related works, acknowledging prior solutions to the same problem, as well as critical details in the experimental setup and evaluation.\n\nUpdate: reviewer discussions indicate those concern are still not resolved", "strength_and_weaknesses": "- Strengths\n  - The problem is of interest and current understanding is still lacking. The proposed approach and theoretical analysis are very valuable.\n  - The experiments do a good job establishing the advantage against using the full spectrum, and justify the saliency-based partitioning.\n\n- Weaknesses\n  - As pointed out by other reviewers, important discussion is missing regarding both related work and experiment setup.\n  - It is not clear what is the limit of this redundancy as far as classification accuracy is concerned. At which point does the model reduce to random guessing, where real and fake appear too similar? I would like to see an experiment exploring this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ha7IZSPAQ5k", "reviewer": "Reviewer_k4jq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for deepfake detection against adversarial attacks. It proposes to use an ensemble of models whose inputs are subsets of frequency spectrum. It also theoretically proves that the ensemble of models reduce the dimension of the adversarial subspace, which could increase the robustness.", "summary_of_the_review": "Overall, the paper is equipped with interesting ideas and strong experiments.", "strength_and_weaknesses": "Strength\n\n1. The idea of training ensemble of models using disjoint frequency spectrum rather than pixels conceptually makes sense. As transforming images into frequency domain can be seen as a decomposition of input features, and dispatching these features to different models can further reduce the complexity of single models (which could make them more robust).\n2. The authors also provide a theoretical proof on how their method could reduce the dimension of the adversarial subspace.\n\nWeakness\n\nI didn’t see any major weaknesses in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2yV2H7jAio", "reviewer": "Reviewer_TsrJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an adversarially robust deepfake detection approach. The approaches divided the frequency information into multiple blocks based on random bisection or saliency information. This divided information is provided to the different models and later combined for robust detection accuracy. The proposed approach was found robust in handling a few attacks and acts as a baseline for future studies. ", "summary_of_the_review": "The proposed research needs attention towards the evaluation strategies including novel attacks belonging to the black box, frequency-based, and adaptive attacks not gradient-based. The comparison in the paper is shallow. \n\nOn top of that, the generalizability of the proposed algorithm against unseen adversaries also needs to be addressed. ", "strength_and_weaknesses": "The proposed approach was found robust in handling few adversarial attacks which make the existing detection algorithms ineffective. \n\nThe claims made especially being the first work are somewhat wrong. In the literature, several works have explored the vulnerability and present robust deepfake detection approaches. \n\n[1] EnsembleDet: ensembling against adversarial attack on deepfake detection\n[2] Defending against GAN-based DeepFake Attacks via Transformation-aware Adversarial Faces\n\nThe proposed algorithm is found vulnerable against one of the adversarial attacks and hence demands extensive evaluation against state-of-the-art attacks including frequency-based attacks. \n\n[3] Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples\n[4] MD-CSDNetwork: Multi-Domain  Cross  Stitched  Network for Deepfake Detection, IEEE International  Conference on Automatic Face and Gesture Recognition, 2021\n\nThe proposed algorithm should also be compared with ensemble-based approaches the baseline must also contain the results with ensemble algorithms: (i) train different networks on frequency information and (ii) utilize multiple types of input images and training of a detection algorithm.\n\nThe majority of the content in the paper refers to the appendix only, which makes the paper hard to read and follow. The authors need to make sure that all the important information is presented in the main paper. \n\nThe comparison with SOTA work is also shallow. \n\nHow the adversarial training has been performed? The results reported by AT seem wrong and need independent verification. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject