Wavelet-Packet Powered Deepfake Image Detection
Authors: Moritz Wolter
Felix Blanke
Charles Tapley Hoyt
Jochen Garcke
Venue: ICLR 2022 Submitted (2022)
Keywords: signal processing
wavelets
wavelet packets
deepfake detection

Abstract: As neural networks become able to generate realistic artificial images, they have the potential to improve movies, music, video games and make the internet an even more creative and inspiring place.
Yet, at the same time, the latest technology potentially enables new digital ways to lie. In response, the need for a diverse and reliable method toolbox arises to identify artificial images and other content.
Previous work primarily relies on pixel-space CNN or the Fourier transform. To the best of our knowledge, synthesized fake image analysis and detection methods based on a multi-scale wavelet representation,
which is localized in both space and frequency, have been absent thus far. 
This paper proposes to learn a model for the detection of synthetic images based on the wavelet-packet representation of natural and GAN-generated images. We evaluate our method on FFHQ, CelebA, and LSUN source identification problems and find improved or competitive performance. Our forensic classifier has a small network size and can be learned efficiently.
Furthermore, a comparison of the wavelet coefficients from these two sources of images allows an interpretation and identifies significant differences.

Under review as a conference paper at ICLR 2022
WAVELET-PACKEI POWERED DEEPFAKE IMAGE DE- TECTION
Anonymous authors Paper under double-blind review
ABSTRACT
As neural networks become able generate realistic artificial images, they have the potential to improve movies, music, video games and make the internet an even more creative and inspiring place. Yet, the same time, the latest technology potentially enables new digital ways to lic. In response, the need for diverse and reliable method toolbox arises to identify artificial images and other content. Previous work primarily relies on pixel-space convolutional neural networks the Fourier transform. To the best of our knowledge, synthesized fake image a multi-scale wavelet representation,
analysis and detection methods based which is localized in both space and frequency, have been absent thus far. This paper proposes to learn model for the detection of synthetic images based on the suvelet-packer representation network
natural and generative adversarial neural images. We evaluate method FFHQ (Karras et al., 2019), CelebA, and LSUN source identification problems and find improved competitive performance. Our forensic classifier has small network size and can be learned efficiently. Furthermore, a comparison of the wavelet coefficients from these sources of images allows interpretation and identifies significant differences.
INTRODUCTION
While GANs can improve traffic prediction (Aigner & Körner, 2019), extract useful representations from data (Radford et al., 2015), translate textual descriptions into images (Zhang et al., 2017), transfer scene and style information between images (Zhu et al., 2017), and generate original art (Tan et al., 2017), they can also enable abusive actors to quickly and easily generate potentially damaging, highly realistic fake images, colloquially called deepfakes (Parkin; Frank al., 2020). As the internet becomes a more prominent virtual public space for political discourse and social media outlet (Applebaum, 2020), deepfakes present a looming threat its integrity that must be met with techniques for differentiating the real and trustworthy from the fake.
Previous algorithmic techniques for separating real from computer-hallucinated images of people have relied on identifying fingerprints in either the spatial (Yu et al., 2019; Marra et al., 2019) or frequency (Frank et al.. 2020; Dzanic et al., 2020) domain. However, the best of our knowledge. no techniques have jointly considered the two domains in a multi-scale fashion. We believe our virtual public spaces and social media outlets will benefit from growing, diverse toolbox of techniques enabling automatic detection of GAN-generated content, where the introduced wavelet- based approach provides a competitive and interpretable addition.
In this paper, we make the following contributions:
We present principled wavelet-packet-be based analysis of GGAN-gemerated images. Compared to existing work in the frequency domain, we examine the GAN-generated content for the first time. We find significant differences between real and synthetic images in both the wavelet-packet mean and standard deviation, with increasing frequency and the edges.
properties of
To the best of our knowledge, we present the first application and implementation of boundary wavelets for image analysis in the deep learning context. Proper implementation

---

Under review as a conference paper at ICLR 2022
of waaveer filters allows us to share identical network architectures across different wavelet lengths.
As result of the aforementioned wavelet-based analysis, we build classifiers to identify image sources. We work with fixed seed values for reproducibility and report mean and standard deviations over five runs whenever possible. Our systematic analysis shows slightly improved or competitive performance.
The source code for our wavelet toolbox, the first publicly available toolbox for fast boundary-wavelet transforms in the python world, and our experiments is available at https:/ / com/ develet-dettettinn--eee
RELATED WORK
2.1 GENERATIVE ADVERSARIAL NETS
The advent of GANs (Goodfellow et al., 2014) heralded several successful image generation projects. For recent review on GAN theory, architecture, algorithms, and applications, we refer the reader to Gui et (2020). We highlight the contribution of the progressive growth technique, in which small network is trained on smaller images then gradually increased in complexity/number of weights and image size, on the ability of optimization to converge at high resolutions of 1024 x 1024 pixels (Karras et al., 2018). As a supplement, style transfer methods (Gatys et al., 2016) have been integrated into new style- style-based generators. The resulting style-GANs have increased the statistical variation of the hallucinated faces (Karras et al., 2019; 2020). Finally, regularization (e.g., using spectral methods), has increased the stability of the optimization process (Miyato et al., 2018). The recent advances have allowed large-scale training (Brock et al., 2019), which in turn improves the quality further.
2.2 WAVELETS
Originally developed within applied mathematics (Mallat, 1989; Daubechies. 1992), wavelets are Long-established part of the image analysis and processing toolbox (Taubman & Marcellin. 2002; Strang & Nguyen, 1996; Jensen & la Cour-Harbo, 2001). In the world of traditional image coding, wavelet-based codes have been developed to replace techniques based on the discrete cosine transform (DCT) (Taubman Marcellin, 2002). Early applications of wavelets in machine learning studied the integration of wavelets into neural networks for function learning (Zhang et al., 1995). Within this line work, wavelets have previously served as input features (Chen et al., 2015), or as early layers of scatter nets (Mallat, 2012: Cotter, 2020). Deeper inside neural networks, wavelets have appeared within pooling layers using static Haar (Wang et al., 2020c; Williams & Li, 2018) or adaptive wavelets (Wolter & Garcke, 2021). Within the subfield of generative networks, ongoing concurrent research (Gal et al., 2021) is exploring the use of wavelets to improve GAN generated image content.
2.3 DEEPFAKE DETECTION
Previous work deepfake image detection broadly falls into two categories. The first group works in the frequency domain. Projects include the study of natural and deep network generated images in the frequency space created by the DCT, as well as detectors based Fourier features (Zhang et al., 2019; Durall et al., 2019; Dzanic et al., 2020; Frank et al., 2020; Durall et al., 2020; Giudice al., 2021). In particular, Frank al. (2020) visualizes the DCT transformed images and identifies artifacts created by different upsampling methods. We learn that the transformed images are efficient classification features, which allow significant parameter reductions in GAN identification classifiers. Instead of relying on the DCT, Dzanic et al. (2020) studied the distribution of Fourier coefficients for real and GAN-generated images. After noting significant deviations of the mean frequencies for GAN generated- and real-images, classifiers are trained. Similarly, Giudice et al. (2021) studies statistics of the DCT coefficients and uses estimations for each GAN-engine for classification. Dzanic et al. (2020) found lighfrreuency discrepancies for GAN- generated imagery, built detectors, and spoofed the newly trained bourier-classifier by manually adapting the coefficients in Fourier-spacee space. Finally, He et al. (2021) combine 2d-FFT and DCT features and observed improved performance.
2

---

Under review as a conference paper at ICLR 2022
The second group of classifiers works in the spatial pixel domain, among others, (Yu et al., 2019; Wang et al., 2020a;b; Zhao et al., 2021a;b) train (convolutional) neural networks directly on the raw images identify various GAN architectures. Building upon pixel-CNN Wang & Deng (2021) adds neural attention. According to (Yu et al., 2019), the classifier features in the final layers constitute the fingerprint for each GAN and are interesting to visualize. Instead of relying on deep-learning. (Marra et al., 2019) proposed GAN-fingerprints working exclusively with denoising-filet and mean operations, whereas (Guarnera et al., 2020) computes convolutional traces.
To the best of our knowledge, we are the first to combine both schools of thought by proposing to work directly with apatio-frequena wavelet packet representation multi-scale fashion. As we demonstrate in the next section, wavelet packets allow visualization of frequency information while, at the same time, preserving local information to some degree. Note that Tang et al. (2021) uses the first level of a discrete wavelet transform, this means no multi-scale representation as pre-processing step to compute spectral correlations between the color bands of an image, which are then further processed to obtain features for classifier.
3 METHODS
"Wavelets are localized waves, instead of oscillating forever, they drop to zero." " TThis observation, in the words of Strang & Nguyen (1996), leads to the first motivation for using wavelet transforms. In contrast to the periodic sine and cosine waves in the Fourier-basis, which are localized in frequency, wavelets are localized in both space and frequency. Since images often contain sharp borders, where pixel values change rapidly, slowly decaying Fourier coefficients caused by such short pulses can fill up the spectrum. Reconstruction of the original pulse depends on the cancellation of most coefficients. The locality of wavelets makes it easier to deal with such pulses.
While previous approaches based on the fast Fourier transform (FFT) by Dzanic et al. (2020) and DCT from Frank et al. (2020) and Giudice et al. (2021) provide abundant frequency space information, the global nature of these bases means all spatial relations are missing. The desire to extend our knowledge by exploiting spatial information is our second reason to work with wavelets. Wavelets allow the representation hierarchical levels of detail space and frequency.
Before discussing the mechanics of the fast wavelet transform (FWT) and its packet variant, we present a proof of concept experiment in Figure 1. We investigate the coefficients of the level Haar wavelet packet transform. Computations use 5K 1024 x 1024 pixel images from Flickr Faces High Quality (FFHQ) (Karras et al., 2019) and 5K 1024 X 1024 pixel images generated by StyleGAN, The leftmost columns of Figure show sample from both FFHQ and a StyleGAN generated one. Mean wavelet packets appear in the second, and their standard deviation in the third column. Finally, the rightmost column plots the absolute mean packets and standard deviation differences. For improved visibility, we re-scale the absolute values of the packet representation, averaged first over the three color bands, using the natural logarithm loge-
We find that the mean wavelet packets of GAN-generated images are often significantly brighter. The differences become more pronounced as the frequency increases from the top left the bottom right. The differences in high-frequene most significant differences for freuuenyyynncy Fourier-coefffcients.
packets independently confirm Dzanic et al. (2020), who saw the
The locality of the wavelet filters allows face-like shapes to appear, but the image edges stand out, allowing us to pinpoint the frequency disparities. We now have an intuition regarding the local origin of the differences. We note that Haar wavelet packets do not use padding or orthogonalization and conclude that the differences stem from GAN generation. For the standard deviations, the picture is reversed. Instead of the GAN the FFHQ-packets appear brighter. The StyleGAN packets do not deviate as much as the data in the original dataset. The observation suggests that the GAN did not capture the complete variance of the original data. Our evidence indicates that GAN-generated backgrounds are of particular monotony across all frequency bands.
In the next section, we discuss how the fast wavelet transform (FWT) and wavelet packets in Figure 1 are computed. For mualti-channel color images, we transform each color separately, but for simplicity, will only consider single channels in the following exposition.
3

---

Under review as a conference paper at ICLR 2022
Style-Can image
StyleGIAN mean packets
StyleGAN standard packet deviation
difference
PPHQ image
PR#Q menn packets
PPHQ standard packet deviation
This figure compares single time-domain images and absolute lsga-s mean wavelet packets and their standard deviations. The first column shows two example images, the second mean packets, the third standard deviations, and the fourth mean and standard deviation differences. Mean values and standard deviations have been computed on 5K 1024 by 1024 FFHQ- and StyleGAN- generated images each. The order of the packets has the frequency increasing from the top left to the bottom right. We observe significant differences. The mean is significantly different for higher frequencies and at image edges. The standard deviations are different on the background across the entire spectrum. The filter application order of each packet is shown on the right of Figure 2. Best viewed color.
Figure 1:
3.1 FAST WAVELET TRANSFORM
Wavelets can be defined by low- and corresponding high-pass filter or by a wavelet function and scaling function, where we follow the filter construction, as common in signal processing. The FWT (Mallat, 1989; Strang & Nguyen, 1996; Jensen & la Cour-Harbe 2001) utilizes convolutions to decompose input signal into its frequency components, where repeated application results in multi-scale analysis. The resulting wavelet coefficients reflect scale (of level ②). which related to frequeney, and location, i.e., the position in the transformed and scaled image of level
Even though 2D wavelets such as the Mexican hat wavelet exist, one dimensional wavelet pairs are often transformed to 2D quadruples using the outer products (Vyas et al., 2018):
f
f, = f
f, = futE
fa fg f
Common choices for ID wavelets are the Daubechies-warvee variant, the symlets ("sym"). We refer the reader to Daubechies (1992) for an in-depth discussion.
("db") and their less asymmetrical
In the equations above, denotes filter vector. For one-dimensional filters, the subscripts C denote the one-dimensional low-pass and H the high pass filter, respectively. In the two-dimensional case, a denotes the approximation coefficients, denotes the horizontal coefficients, v denotes vertical coefficients, and d denotes the diagonal coefficients. The 2D transformation at representation level + requires the input x,, as well as a filter quadruple fa for k E [a, h, v, d] and is computed by
Xg * fL kg+1.
(2)
where * denotes stride two convolution. Note the input image is at level zero, i.e. I, while for stage y the low pass result of q is used as input.
The wavelet transform is invertible. The forward or analysis transform relies on the analysis filters often denoted as H2 (Strang & Nguyen, 1996). The inverse or synthesis transform filters are typically written Ft. The down- 12 and up- 12 arrow symbols describe the sampling operations in what is known as stride two convolution and transposed convolution in today's machine learning literature. In order make the transform invertible and plots interpretable, not just any filter will do. The perfect reconstruction and anti- aliisilis conditions must hold. We refer to Strang & Nguyen (1996) and Jensen & la Cour-Harbo (2001) for excellent further discussion of these conditions.
4

---

Under review as a conference paper at ICLR 2022
3.2 BOUNDARY WAVELETS
So far we described the wavelet transform without considering the finite size of the images. For example, the simple Haar wavelets can be used without modifications in such a case. But, for the transform to preserve all information and be invertible, higher-order wavelets require modifications at the boundary (Strang & Nguyen, 1996). There are different ways handle the boundary, including proo-padinn, symmetrization, periodic extension, and specific filters on the boundary. The disad- vantage of zero-padding periodic extensions is that discontinuities are artificially created the border, whereas with symmetrization discontinuities of the first derivative arise at the border (Jensen la Cour-Harbo, 2001). While for large images, the boundary effects might be negligible, for the employed multi-scale approach of wavelet-packets as introduced in the next subsection, the artifacts become too severe. Furthermore, zero-pas pro-ppading increases the number of coefficients, which in our application would need different neural network architectures per wavelet. Therefore we employ special boundary filters in the form of the so-called Gram-Schmidt boundary filters (Jensen la Cour-Harbo, 2001).
The idea now to replace the filters at the boundary with specially constructed, shorter filters that preserve both the length and the perfect reconstruction property or other properties of the wavelet transform. In the appendix Figures 7 and 8 illustrate the sparsity patterns of the resulting sparse analysis and synthesis matrices for the Daubechies two case.
3.3 WAVELET PACKETS
Presuming to find the essential information in the lower frequencies, standard wavelet transformations decompose only the low-pass coefficients further. The h, v, and coefficients are left untouched. While this is often reasonable assumption (Strang & Nguyen, 1996), previous work (Dzanic et al., 2020) found the higher frequencies equally relevant for deepfake detection. Our analysis, therefore. relies on wavelet packets. For a wavelet packet representation, one recursively continues to filter the low- and high-pass results. Each recursion leads new level of filter coefficients, starting with an input image I E Rh.", and using no.0 I. A node nag at position j of level q. is convolved with all filters fg, k E [a, h, v, d]:
ng. fg 11+
Therefore every node at level y will spawn four nodes at the next level q + 1. The result at the final level Q. assuming Haar or boundary wavelets without padding, will be 4 x A x w tensor, i.c. the number of coefficients is the same as before and is denoted by Q Thereby wavelet packets provide filtering of the input into progressively finer equal- blocks, with no redundancy. For excellent presentations of the one-dimensional case we again refer to Strang & Nguyen (1996) and Jensen & la Cour-Harbo (2001).
We show a schematic drawing of the process the left of Figure 2. The upper half shows the analysis transform, which leads to the coefficients we are after. The lower half shows the synthesis transform, which allows inversion for completeness. Finally, for the correct interpretation of Figure I the right of Figure 2 lists the filter combinations for packet plots previously shown in Figure 1.
4 IMAGE SOURCE SEPARATION EXPERIMENTS
Considering Figure I once more. note that the absolute differences of the mean wavelet-coefffcienns and their standard deviations, shown in the rightmost column, appeared significantly different. In the absolute mean difference plot, we saw widening disparity with increasing frequency along the diagonal. Additionally, background and edge coefficients appeared to diverge. To numerically relate these to each other, we remove the spatial dimensions by averaging over these as well. We observe in the left of Figure 3 increasing mean differences across the board. Differences are especially pronounced at the higher frequencies the right. In comparison the FFHQ standard deviation, the variance produced by the StyleGAN, shown in orange, is smaller for all coefficients. In the following sections, aim to exploit these differences for the identification of artificial images.
5

---

Under review as a conference paper at ICLR 2022
11
11 kg
Hh 4o
H. Lo
H.
aaa
Bav avv ava
vva
vvv
vav
vaa
aah
aad avd avh
vvh
vvd
vad
vah
ahh
ahd
add adh
vdh
vdd
vhd
vhh
aha
ahv
adv
ada
vda
vdv
vhv
vha
To F.
Fs.
To
To Fa
hha
hhv
hdv
hda
dda
ddv
dhv
dha
hhh
hhd
hdd
hdh
ddh
ddd
dhd
1s Pa
10 Fh
1o
to Fa
hah had hvd hvh
dvh
dvd
dad
hav hvv
hva
dva
dvv
dav
daa
Figure 2: Visualization of the 2D wavelet packet transform analysis and synthesis packet transform
(left). The analysis filters are written as Hg, synthesis filters as Fg. We show all first level coefficients well as some second level coefficients aa, ah, av, od. The dotted lines indicate the omission of further possible analysis and synthesis steps. The transform is invertible in principle, i denotes the reconstructed original input. The right side shows the labels for the transforms of level we showed previously in the frequency order (Jensen & la Cour-Harbo, 2001).
Mean absolute coefficient comparison EEHO-Sttl:GAN
Accuracy FFHO--yyleGANNaayyGGANN2 idemtfficctooo
mean absolute coeflictent magnitude
FFHQ
StyleGAN
0.95
mean accuacy
0.9
2-
0.8
validation
-2
Haar va validation acc
Haar test accuracy accuracy
0.75
avv
add
hiha
hdv
vad
vall
daav
dhd
del oidl
500
1,000
1,500 2,000 2,500 3,000 3,500
lilter
training sleps
Figure 3: The mean 3 Haar wavelet packet coefficient values for each 63K 128 128 pixel FFHQ (blue) and StyleGAN (orange) images are shown on the left. The shaded area indicates a single standard deviation a. We find higher mean coefficient values for the StyleGAN samples across the board. As the frequency increases from left to right, the differences become more pronounced. The plot on the right side shows validation and test accuracy for the identification experiment. Linear regression networks where used identify FFHQ, StyleGAN, and StyleGAN2 (Karras al., 2020) images. The blue line shows the pixel and the orange line the training accuracy using log, -scaled absolute wavelet packet coefficients. Shaded areas indicate single standard deviation for different initializations. We find that working with log, log,-scaled -scaled absolute packets allowed linear separation of all three images sources. Furthermore, it significantly improves the convergence speed and final result.
Predicted label
True label
FFHQ
StyleGAN
StyleGAN2
FFHQ
4896
100
StyleGAN
25
4973
2
StyleGAN2
4901
Table 1: Confusion matrix for the Haar wavelet regression experiment. Overall 98.5% of all samples where classified correctly. Linear Wavelet regression works particularly well for the StyleGAN. 99. 99.46% of all StyleGAN samples have been properly identified.

---

Under review as a conference paper at ICLR 2022
4.1 FFHQ
Encouraged by the differences we saw in Figures 1 and 3, we attempt to linearly separate the log.- scaled 3rd level Haar wavelet packets by training linear regression model. We work per class with 63K images for training, 2K for validation, and 5K for testing. All images have a 128 x 128 pixel resolution. Wavelet coefficients, as well as raw pixels, are normalized by mean subtraction and division by a, using their respective mean and standard deviations computed on the train set. On both normalized features, we train identical networks using the Adam optimizer (Kingma Ba, 2015) with step size of 0.001 using PyTorch (Paszke et al., 2017). We plot the mean validation and test accuracy over runs with identical seeds of 0, 1, 2,3, 4 in Figure 3 on the right. The shaded areas indicate a single a o-deviation. On the test set, we observe the confusion table 1. We conclude that in the Haar packet coefficient space, we are able to separate the three sources linearly. Overall, 98.5% of all samples and 99, 46% of all StyleGAN images are identified correctly. Observe that working in the Haar-wavelet space improved both the final result and convergence speed.
4.2 LARGE-SCALE CELEB FACES ATTRIBUTES AND LARGE-SCALE SCENE UNDERSTANDING
In the previous experiment, essentially two image sources, authentic FFHQ and the related StyleGAN or StyleGAN2 generated images, had to be identified, where we additionally could also distinguish the two artificial sources. In this section, we will study a standardized problem with more image sources. To allow comparison with previous work, we exactly reproduce the experimental setup from Yu et al. (2019) and Frank et al. (2020). Note that we choose this setup since we, in particular want to compare our space and frequency approach with the frequency only DCT-rrepresentation. We cite their results for comparison. Additionally, we benchmark against the photoresponse non- uniformity (PRNU) (Marra et al., 2019) approach, well as the eigenfaces (Sirovich & Kirby, 1987), and straightforward regression as baselines. The experiments in this section use four GANs; CramerGAN (Bellemare al., 2017), MMDGAN (Binkowski et al., 2018), ProGAN (Karras et al., 2018), and SN-DCGAN (Miyato et al., 2018).
150K images were randomly selected from the Large-scale Celeb Faces Attributes (CelebA) (Liu et al., 2015) and Large-scale Scene UNderstanding (LSUN) bedroom (Yu et al., 2015) datasets. Our pre-processing is identical for both. The real images are cropped and resized to 128 x 128 pixels each. With each of the four GANs an additional 150K images are generated at the same resolution. The 750K total images are split into 500K training, 100K validation, and 150K test images. To ensure stratified sampling, we draw the same amount of samples from each generator or the original dataset. As result, the train, validation, and test sets contain equally many images from each source.
We compute wavelet packets with three levels for each image. We explore the use of Haar and Daubechies wavelets as well as symlets (Daubechies, 1992: Strang & Nguyen, 1996; Jensen & la Cour-Harbo, 2001). The Haar wavelet is also the first Daubechies wavelet. Daubechies-wavelets and symlets are identical up to a degree of 3. We, therefore, start comparing both families above a degree of 4.
For a fair comparison, we normalize both the raw pixels and the wavelet coefficients. Normalization always the last step before the models. We normalize by subtracting the training-set color-channel mean and dividing each color channel by its standard deviation. The log, -scaled coefficients are normalized after the rescaling. We train identical CNN-classifiers on top of pixel and various wavelet representations. Additionally, we evaluate eigenface and PRNU baselines, Our wavelet packet transform, regression, and convolutional models are implemented using Py PyToc (Paszke et al., 2019). We use Adam (Kingma& & Ba, 2015) to optimize our convolutional-modde models for 20 epochs. The batch size is set 512 and the learning rate to 0.001. See the supplementary for the exact network layouts.
Table 2 lists the results for comparison. We obtain beneficial packet representation in every case. On celebA the eigenface approach, in particular, was able to classify 24. / /% more images correctly when run on log, -scaled -avelet packages instead of pixels. For the more complex CNN, Haar wavelets are not enough. More complex wavelets, however, significantly improve the accuracy. With a CNN comparable to Frank et al. (2020), we find accuracy maxima superior to the DCT approach for five wavelets. Demonstrating the robustness of the wavelet approach, the db3 and db4 packets are better on average. On LSUN, we gains over the raw pixel representation for our log,- -scaled wavelets for CNN as well as eigenface- sagenface-based classifiers. While we do not outperform the DCT features in this case, we improvements of the pixel baseline and overall competitive performance.
7

---

Under review as a conference paper at ICLR 2022
Table 2: CNN source identification results the CelebA and LSUN bedroom datasets. We explone the use of boundary-waver packets up to a wavelet degree of 5. We report the test set accuracy. Whenever possible report mean accuracy and standard deviation over runs.
Accuracy[%]
CelobA
LSUN bedroom
Method
max
1 + 0
max
14 + 0
Eigenfaces (ours)
68.56
62.24
Eigenfaces-DCT (Frank et al., 2020)
88.39
94.31
Ligenfaces-log,- -Haar (ours)
92.67
87.91
PRNU (ours)
83.13
66.10
CNN-Pixel (ours)
98.87
98.74
97.06
95.02 1.14
CNN-log.-Haad (ours)
97.09
6..99+0..
97.14
+.19
CNN-log.-db2 (ours)
99.20
0.96
98.90
9.73
CNN-log. (ours)
99.38
99.19
99.01 0
CNN-log.ddh4 (ours)
99.43
99.27 0.15
99.02
98.46 0.67
CNN-log,-db5 (ours)
99.02
98.66 + 0.59
98.32
97.64 + 0.83
CNN-log,-Sym4 (ours)
99,43
98.98 + 0.36
99.09
98.57 + 0.72
CNN-log.ssymS (ours)
99.49
98.79 + 0.48
99.07
98.78 + 0.24
CNN-Pixel (Yu et al., 2019)
99.43
98.58
CNN-Pixel (Frank et al., 2020)
97.80
98.95
CNN-DCT (Frank et al., 2020)
99.07
99.64
Table 3: Confusion matrix for our best performing CNN using db4-wavelet-pacets on CelebA. Classification errors for the original dataset as well the CramerGAN, MMDGAN, ProGAN, and SN-DCGAN architectures are shown. The test set contains 30,000 entries per label. The detector classifies 99.43% of all images correctly.
Predicted label
True label
CelebA
CramerGAN
MMDGAN
ProGAN
SN-DCGAN
CelebA CramerGAN
29,759
11
107
116
19
29,834
144
3
MMDGAN
17
120
29,862
ProGAN
136
29,755
108
SN-DCGAN
55
5
29,940
We show the confusion matrix for our best performing network on CelebA in Table Among the GANs considered the ProGAN-architecture and SN-DCGANN-rchitees
Images drawn from both architectures were almost exclusively misclassified as original data and rarely attributed to another GAN. The CramerGAN and MMDGAN generated images were often misattributed each other, but rarely confused with the original dataset. Of all misclassified real CelebA images most were confused with ProGAN images, making it the most advanced network of the four. Overall find our, in comparison to GAN, lightweight 1 109k classifier able to accurately spot 99.45% of all fakes this case. Note that all our convolutional networks have only 109,061 parameters, while Yu et al. (2019) used roughly 9 million parameters and Frank et al. (2020) utilized around 170,000 parameters. We observed that using our packet representation improved convergence speed, Figure 10 in the supplementary material illustrates this for the Haar- Wavelet case.
We study the effect drastic reduction of the training data size, by reducing the training set size by 80%. Table 4 reports test acturacusss for retrained classifiers. We find that our Daubechies packet-CNi CNN classifiers are robust to training data reductions, in particular in comparison to the pixel representation.
8

---

Under review as a conference paper at ICLR 2022
Table 4: CNN source identification results the CelebA dataset with only 20 % of the original data. We report the test set accuracy mean and standard deviation over 5 runs as well as the accuracy loss compared to the corresponding network trained on the full data set.
Accuracy on CelebA[%]
Method
max
Loss
/ +
Loss
CNN-Pixel (ours)
96.37
-2.50
95.16 + 0.86
-3.58
CNN-log,-dbe (ours)
99.01
-0.41
96.96
+ 3.47
-2.58
CNN-Pixel (Frank et al., 2020)
96.33
-1.47
CNN-DCT (Frank et al., 2020)
98.47
-0.60
DETECTION OF IMAGES FROM AN UNKNOWN GENERATOR
To exemplarily study the detection of images from an unknown GAN, we remove the SN-DCGAN generated images from our LSUN training and validation sets. 10,000 SN-DCGAN images now appear exclusively in the test set, where they never influence the learning process. The training hyperparameters identical to those discussed in section 4.2. We rebalance the data contain equally many real and genneeeertted images. The task now is to produce a real or fake label in the presence of fake images which were not present during the optimization process. For this initial proof-of- cooff-o--concet investigation, we apply only the simple Haar wavelet. We find that our approach does allow detection of the SN-DCGAN samples on LSUN- training set. Using the -ogaled Haar- Wavelet packages, we achieve for the unknown GAN a result of 78.8 1.8%, with a max of 81.7%. For the real and artificial generators present in the training set, we achieve 98.6 + 0.1%, with max of 98.6%. The accuracy improvement in comparison to Table 2 likely due to the binary classification problem here instead of the custi-classification one before.
without their presence in the
This result indicates that the wavelet representation alone can allow a transfer to unseen generators, or other manipulations, to some extent. More detailed studies, also with wavelets of higher degree, and in comparison, or together, with transfer learning approaches (Jeon et al., 2020) or autoencoders particularly designed for transferability (Cozzolino et al., 2018) are warranted future work.
6 CONCLUSION AND OUTLOOK
This paper presented wavelet macket-based approach for deepfake analysis and detection, which is based on a multi-scale image representation in space and frequency.
We saw that wavelet-packets allow the visualization of frequency domain differences while preserving some spatial information. We found diverging mean values for packets at high frequencies. At the same time, the bulk of the standard deviation differences were the edges and within the background portions of the images. This observation suggests contemporary GAN architectures still fail to capture the backgrounds and high-frequency information in full detail. The proposed wavelet packet representations allowed linear separation of the STHHO--tyleGAN-Stylecca problem. We studied additional wavelets on the CelebA and LSUN -UN--bdrooo problems. We found that coupling higher-order wavelets and CNN led to an improved or competitive performance in comparison to a DCT approach or working directly on the raw images. The employed lean neural network architecture allows efficient training and can achieve similar accuracy with only 20% of the training data. Overall our classifiers deliver state-of-the-ar performance, require few learnable parameters, and converge quickly.
source identification
Even though releasing our detection code will allow potential bad actors to test against it, we hope approach will complement the publicly available deepfake identification toolkits. We propose to investigate the resilience of multi-classifier ensembles in future work and envision a framework of multiple forensic classifiers, which together give strong and robust results for artificial image identification.
Future work could also explore the use of curvelets and shearlets for deepfake detection.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of this research project, we release our wavelet toolbox and the source code for our experiments in the supplementary material. The wavelet toolbox ships multiple unit tests to ensure correctness. Our code is carefully documented to make it as accessible as possible. We have consistently seeded the Mersenne twister used by PyTorch to initialize our neural networks with 0, 1, 2, 3, 4. Since the seed is always set, rerunning our code always produces the same results. Outliers can make hard to reproduce results if the seed unknown. ensure we do not share outliers without context, report mean and standard deviations of five runs for all neural network experiments.
REFERENCES
Sandra Aigner and Marco Körner. Futuregan: Anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing gans. The International Archives of the Photogrammetry. Remote Sensing and Spatial Information Sciences, Volume XLII-2/W16, 2019, 2019.
Anne Applebaum. Twilight of democracy; the seductive lure of authoritarianism, 2020.
Marc G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as a solution to biased wasserstein gradients. CoRR, abs/1705.10743, 2017. URL http: /arxiv..vvv.. org/ hblgg/////// abs/ab///515/ 10743.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD gans. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: http:///ppeeeeveieee net/ forumiid=:11002NCC
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representattions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpss://opereevie.t net/forum?? (irrrmmidd=l1ss009995
Lan-lan Chen, Yu Zhao, Jian Zhang, and Jun-zhong Zou. Automatic detection alertness/drrowsiness from physiological signals using wavelet-based nonlinear features and machine learning. Expert Systems with Applications, 42((1):7334-7353 7355, 2015.
Fergal Cotter. Uses of Complex Wavelets in Deep Convolutional Neural Networks. PhD thesis,
University of Cambridge, 2020.
Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, and Luisa Verdoliva. ForensicTransfer: Weakly Domain Adaptation for Forgery Detection. 2018.
ISSN 2331-8422. URL http: http://axvvvvv./// org. 1812. 02510.
Ingrid Daubechies. Ten Lectures on Wavelets. SIAM, 1992. ISBN 978-0-89871-2744-. doi:
10.1 90.11377119781111990111 URL battps://dat org 10. 11111117119711111197700000
Ricard Durall, Margret Keuper, Franz-Josef Pfreundt, and Janis Keuper. Unmasking deepfakes with simple features. arXiv preprint 1911::191100006 2019.
Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Cnn based genera- tive deep neural networks are failing to reproduce spectral distributions. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7890-7899, 2020.
Tarik Dzanic, Karan Shah, and Freddie D. Witherden. Fourier spectrum discrepancies in deep network generated images. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Essua-Tiie Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https: httpp/ppcccee....e/ee neurips...... tttpcc//pp///e wwtpppppp//aaeeeeeeeeeee 2020/ hash/1 f8d87e1 61 6868 wa..8aaaceeee....cc.....aaaaaaaa 8 88 ec Abstract html.
10

---

Under review as a conference paper at ICLR 2022
Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020. Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3247-3258. PMLR, 2020. URL http: / http://ooceeeeee ngs wwm.m.r.pp. pre.s..pe////.. / vv1 frank20a .tml..
Rinon Gal, Dana Cohen, Amit Bermano, and Daniel Cohen-Or Or. Swagan: A style-based wavelet-driven generative model. arXiv preprint arXiv:2 16108, 2021.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pultern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2414-2423. IEEE Computer Society, 2016. doi: 10.1 URL https: /oi. org/ 101010.0000 1...0.119/// 111911111PP// 2016. 265.
Oliver Giudice, Luca Guarnera, and Sebastiano Battiato. Fighting Deepfakes by Detecting GAN DCT Anomalies. Journal of Imaging, 7(8): jul 2021. ISSN 2313-433X. doi: 10.3390/
jimaging7080128 URL htth::::: /www. www.wwppp.....ooooow.wpwpwwt.w.w .tm.siccom/////ommm 1 8/28..
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Luca Guarnera, Oliver Giudice, and Sebastiano Battiato. Fighting deepfake by exposing the con- volutional traces on images. IEEE Access, 8:165085-165098, 2020. ISSN 21693536. doi:
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications. arXiv preprint, 2001.06937, 2020.
URL https:tt//// /arxiv.iviv..///ai org/ abs///a1..000.. 06937.
Yang He, Ning Yu, Margret Keuper, and Mario Fritz. Beyond the spectrum: Detecting deepfakes via re-synthesis. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp.
ijcai.org, 2021. doi: 00.24963/jjcci.22221/383 URL https: ttt./ooooooooooo wttp/.o.or.0.... 24963/ jeaii. 2021/349
Ame Jensen and Anders la Cour-Harbo Ripples in mathematics: the discrete wavelet transform. Springer Science & Business Media, 2001.
Hyconseong Jeon, Youngoh Bang, Junyaup Kim, and Simon S. Woo. T-GD; Transferable GAN- generated Images Detection Framework. 37th International Conference Machine Learning, ICML 2020, PartF168 11664446666411111 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: : /pwp//peeeee...e////////p net wwwp:nneew.ooneeeeeeeeeo /t/f/oormmm//// iim zCeAb.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015, URL http: //arxiv. orgiiv.orrg/aa abs www.////....... 6980.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Stéphane Mallat. A wavelet tour of signal processing, the sparse way, 2009.
11

---

Under review as a conference paper at ICLR 2022
Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 65 (10):1331-1398, 2012.
Stephane Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions pattern analysis and machine intelligence, 11(7):674-6933 1989.
Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. Do gans leave artificial fingerprints? In 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 506-511. IEEE, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL https::/p//// http://ppppr....eeeeeee net/ forum? iide QRgziT-
Simon Parkin. The rise of the deepfake and the threat democracy. https: /ttpe///21//////////// Accessed:
//www. theguardi an ccom/ttcchoolloggngniineeee hha.ggaardaa..cm//eeeeollooooooo the-rise- teeeee-ofthheee te--tteeeeeepppe -eee--eeeaaa-------------eeeeeee 2021-05-10.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. neurips cccppppr//2119 9/hash/
8024-8035, 2019. URL pettps::/proceeediie bdbca288 fee 92 ( -tst html.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: ////////11044444 2015.
Lawrence Sirovich and Michael Kirby. Low-dimensional procedure for the characterization of human faces. Josa a, 43):559-5 1987.
Gilbert Strang and Truong Nguyen. Wavelets and filter banks. SIAM, 1996.
Wei Ren Tan, Chee Seng Chan, Hernán E Aguirre, and Kiyoshi Tanaka. Artgan: Artwork synthesis with conditional categorical gans. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 3760-3766 IEEE, 2017.
Guihua Tang, Lei Sun, Xiuging Mao. Song Guo, Hongmeng Zhang, and Xiaoqin Wang. Detection of GAN-Synthesized Image Based on Discrete Wavelet Transform. Security and Communication Networks, 2021:1-10, 2021. ISSN 1939-0114, 14. doi: 10. 10.11555202//5511443
David Taubman and Michael W Marcellin. Jpeg2000: Standard for interactive imaging, Proceedings of the IEEE, 90(8): 2002.
Apama Vyas, Soohwan Yu, and Joonki Paik. Multiscale transforms with application to image processing. Springer, 2018.
Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14923-14932, 2021.
12

---

Under review as a conference paper at ICLR 2022
Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, and Yang Liu. FakeSpotter: A Simple yet Robust Baseline for Spotting Al-Synthesized Fake Faces. Pro- ceedings the Twenty-Ninth International Joint Conference Artificial Intelligence, volume 2021-Janua, pp. 3444-3451, California, jul 2020a. International Joint Conferences on Artifi- cial Intelligence Organization. ISBN 978-0-9992211114449 1-65.. doi: 10.24963/jjci 2020/446. URL
ht www http://www......poooaeaaaa//////
Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cun-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8695-8704, 2020b.
Yu Guang Wang, Ming Li, Zheng Ma, Guido Montufar, Xiaosheng Zhuang, and Yanan Fan. Haar graph pooling. In International Conference on Machine Learning, pp. 9952-9962. PMLR, 2020c.
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International Conference Learning Representations, 2018.
Moritz Wolter and Jochen Garcke. Adaptive wavelet pooling for convolutional neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 1936-1944. PMLR, 2021.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.08 03365, 2015.
Ning Yu, Larry S Davis, and Mario Fritz. Attributing fake images to gans: Learning and analyzing gan fingerprints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Jun Zhang, Gilbert G Walter, Yubo Miao, and Wan Ngai Wayne Lee. Wavelet neural networks for function learning. IEEE transactions on Signal Processing, 43(6):1485-1499) 1995.
Xu Zhang, Svebor Karaman, and Shih-Fu Chang. Detecting and simulating artifacts in gan fake images. In 2019 IEEE International Workshop Information Forensics and Security (WIFS), pp. 1-6. IEEE, 2019.
Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi- attentional deepfake detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2185-185211 2194, 2021a.
Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia. Learning self- consistency for deepfake detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15023-15033, 2021b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
ACRONYMS
CelebA Large-scale Celeb Faces Attributes
CNN convolutional neural network
DCT discrete cosine transform
FFHQ Flickr Faces High Quality
FFT fast Fourier transform
FWT fast wavelet transform
13

---

Under review as a conference paper at ICLR 2022
GAN generative adversarial neural network LSUN Large-scale Scene UNderstanding
PRNU photoresponse non-uniformity
SUPPLEMENTARY
In this supplementary. show the full architectures for all networks. In terms of training convergence, we give the validation accuracy mean and standard deviation during training. Furthermore, we show additional visualizations of the wavelet packets.
8.1 NETWORK ARCHITECTURES
Pixcl-CNN
Convolution
(3x8x3x3),
Wavelet-Packet-CNN
ReLU,
Convolution
(8x8x3),
Convolution (192x24x3x3),
ReLU,
ReLU,
Pool
(2, 2), (8x (8x16x3x3),
Convolution
(24x24x6x6),
Convolution
ReLU,
ReLU,
Convolution
(24x24x9),
Pool
(2, 2), (16x32x3x3),
ReLU,
Convolution
Linear (24, classes)
ReLU,
Linear
232*28*28, classes)
Table 5: The CNN architectures used in our experiments. We show the convolution and pooling kernel, as well as matrix sizes in brackets. As the packet representation generates additional channels we are forced adapt the CNN architecture. The parameter counts vary depending on the number of classes, but has have approximately 100k for all experiments.
Unless explicitly stated otherwise, we train with batch size of 512 and learning rate of 0.001 using Adam (Kingma & Ba, 2015) for 20 epochs. For the 5 repetitions we report the seed values are set to 0,1,2,3,4.
8.2 PERTURBATION ANALYSIS
We study the effect of image cropping, rotation, and jpeg compression our classifiers in table 6. re-trained both classifiers on the perturbed data as described in section 4.2. The center crop perturbation randomly extracts images centers while preserving least 80% of the original width height. Both the pixel and packet approaches are affected by center cropping yet can classify more
Table 6: Perturbation analysis of the db3 wavelet-packet- and pixel- CNN classifiers LSUN- bedrooms.
Accuracy on LSUN[%]
Method
Perturbation
max
+
CNN-log, -db3 (ours)
center-crop
95.68 92.03
95.49- 0.26
CNN-Pixel (ours)
center-crop
90.041 1.18
CNN-log. (ours)
rotation rotation
91.74
90.841 0.90
CNN-Pixel (ours)
92.63
91.994 0.97
CNN-log.-dhe (ours)
jpeg jpeg
84.73
84.25 = 00.33
CNN-Pixel (ours)
89.95
89.25 = 00.48
14

---

Under review as a conference paper at ICLR 2022
Daubechies 5
Symlet 5
2 he
hx
0.3
0.5
2 fr.
hg
hN
-0.5
O
L
-0.5
fH
6
8
10
2
4
10
Figure 4: Fifth degree Daubechies and Symlet filters side by side.
than 90% of all images correctly. In this case, the lca-scaled db3-waveler packets can outperform the pixel-space representation average. For rotation and jpeg compression the picture changes. The rotation perturbation randomly rotates all images by up to 15 degrees. The jpeg perturbation randomly compresses images with compression factor drawn from (700, 90), without subsampling. The wavelet-packer representation is less robust to rotation than its pixel counterpart. For jpeg- compression, the effect is more pronounced. Looking at Figure 3 suggests that our classifiers rely on gigg-frreuency information. Since neggccopresssion removes this part of the spectrum, perhaps explains the larger drop in mean accuracy.
8.3 THE PERFECT RECONSTRUCTION AND ALIAS CANCELLATION CONDITIONS
When using wavelet filters, we expect a lossless representation free of aliasing. Consequently, we want restrict ourselves filters, which guarantee both. Classic literature like Strang & Nguyen (1996) presents two equations to enforce this, the perfect reconstruction and alias cancellation conditions. Starting from the analysis filter coefficients and the synthesis filter coefficients For a complex number E C their z-transformed counterparts are H(z) Σn ((n)=-" and F(z) respectively. The reconstruction condition can now be expressed as
Hc(z)Fo(z) + Hz(2)Fu(2)) = 2
(4)
and the anti-aliasing condition as
Hc(-=)F2(=)) Hx(-=))H(() 0.
(5)
For the perfect reconstruction condition in Eq. 4, the center term s' of the resulting z-transformed expression must be two; all other coefficients should be zero. denotes the power of the center. The effect of the alias cancellation condition can be studied by looking at Daubechies wavelets and symlets, which we will do the next section. The perfect reconstruction condition is visible too, but harder to see. For an in-depth discussion of both conditions, we refer the interested reader to the excellent textbooks by Strang Nguyen (1996) and Jensen & Cour- Harbo (2001).
8.4 DAUBECHIES WAVELETS AND SYMLETS
Figure 4 plots both Daubechies filter pairs the left. A possible way to solve the anti-aliasing condition formulated in equation is to require, Fz(=)= = (=(= and Fx(2) - (=((=) (Strang & Nguyen, 1996). The patterns this solution produces as well as their symmetric counterparts shown above. To see why substitute (-z). It will produce a minus sign odd powers in the coefficient polynomial. Multiplication with (-1) shifts the pattern
visible in for both the Daubechies filters
to even powers. Whenever Fz and Hμ share the same sign FH and He do not and the other way around. The same pattern is visible for the Symlet the right of figure 4.
The Daubechies wavelets are very anti anti-symmetric (Mallat, 2009). Symlets have been designed as an alternative with similar properties. But, as shown in 4, Symlets are symmetric and centered.
15

---

Under review as a conference paper at ICLR 2022
analysis Haar level
synthesis Haar level 1.tex
Figure 5: Sparsity pattern of the 2d level Haar fast wavelet analysis and synthesis matrices.
analysis Haar level
synthesis haar level 2.tex
Figure 6: Sparsity pattern of the 2d level Haar fast wavelet analysis and synthesis matrices.
8.5 SPARSE FAST WAVELET TRANSFORMATION MATRIX PLOTS
16

---

Under review as a conference paper at ICLR 2022
analysis db2 level
synthesis db2 level
Figure 7: Sparsity pattern of the 2d level Daubechies-2 fast wavelet analysis and synthesis matrices.
analysis db2 level
synthesis db2 level 2
Figure 8: Sparsity pattern of the 2d level 2 Daubechies-2 fast wavelet analysis and synthesis matrices.
17

---

Under review as a conference paper at ICLR 2022
Accuracy source identification
0.9
mean accuracy
0.8
0.7
validation acc
0.6
log.-db4 -b4 validation acc
raw test acc
0.5
log,- db4 test acc
1,000
2,000
3,000
training steps
Figure 9: Mean validation and test set accuracy of 5 runs of source identification on CelebA for CNN trained the raw images and log, log.- wavelet packets, each using only 20 % of the original training data. The shaded areas indicate a single standard deviation a.
source identification
LSUN-GAN
CelebA-GAN
0.9
0.8
accuracy
0.7
raw validition ace
0.6
validation logracket validition ace pucket test ace logpacket test
0.5
5,000
10,000
15,000
20,000
5,000
10,000
15,000
20,000
training steps
training steps
Figure 10: Haar-Waveler packet CNN training accuracy on the LSUN (left) and the CelebA (right) datasets.
18


Reviews:
[{"review_id": "tGw67b3DrcU", "reviewer": "Reviewer_feXB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper uses a different representation of images i.e wavelet-packets to solve source identification problem which has not been used before. \n2. The analysis on using different levels  for Daubechies and symlets wavelets provides better understanding of which level is useful for detecting GAN generated images.\n\nWeaknesses:\n\n1. The main idea of using wavelet coefficients to perform deepfake detection is not a novel approach. Wavelet representation is localized in space and frequency domain. However, both domains have already been studied in the past [1], [2]. Therefore, using a representation combining both of them is not a very exciting idea. Looking at the source identification results in Tab. 2 highlights the fact that the combined representation is performing very similar to previous approaches with worse performance for LSUN dataset. This makes me question the contribution of the proposed approach for the community. Using a new representation although gives an idea to look at a new direction, but the performance doesn't contribute positively for the same. \n\n2. The authors provide a thorough analysis of their approach in Tab. 2 which talks about the source identification task. However, the problem of source identification is quite simple and previous approaches already have made significant advancement in the performance. The authors try to go one step further by removing one GAN from the training set and evaluate their approach. However, they don't provide any baseline results to compare with for proposed testing protocol. This gives no conclusion whether the proposed approach works better than pervious works or not. \n\n\n3. The paper proposes to use wavelet coefficients at a certain level to perform a transformation of input image using quadruple filters as shown in Eq. 2. The filters defined in Eq. 1 are used to perform the above transformation. These filters denote the low-low, low-high, high-low and high-high coefficients. However, the intuition for using these four types of coefficients is still not clear. Each type of coefficient can have different performance for deepfake detection. It is not clear which of the four is responsible for the reported result in Tab. 2 and 4.\n\n4.  In page 4, there is a discussion for the wavelet transform being invertible. This is done using analysis filters H and inverse filters F. The paragraph further talks about perfect reconstruction and anti-aliasing conditions to select these filters. However, a brief introduction of the above mentioned conditions and how these conditions are used to select filters for the proposed framework should be mentioned clearly. \n\n5. The authors tackle the problem of source identification. But it would be interesting to see how the proposed method would perform when trained and tested on deepfake datasets like FaceForensics++, DFDC, UADFV, etc. Real vs fake binary classification performance would provide much more clarity on whether the proposed approach actually works for deepfake detection or not as depicted by the main title of the paper. \n\n6. In section 4.1, the authors use a linear regression model to separate the 3rd level Haar wavelet packets for classification. However, in section 4.2, the authors use a CNN-classifier to perform classification. it is not clear why different models were used for different datasets. \n\n7. The authors use Haar and Daubechies wavelets as well as symlets wavelets in their experiments. However, it would be beneficial to include a brief introduction for both type of wavelets in section 3 to better understand the intuition behind selecting these two for this work.\n\n\n8. The setting of experiment on FFHQ images dataset mentions that the images are used at a resolution of 128 x 128. However, FFHQ has 1024 x 1024 images. Same is the output for StyleGAN and StyleGAN2. So my concern is that the patterns observed in the wavelet visualization in Figure 1 might be formed due to downsampling of these images. Therefore, the difference in the visualization might come from downsampling rather than GAN itself. \n\nReferences: \n1. Ning Yu, Larry S Davis, and Mario Fritz. Attributing fake images to gans: Learning and analyzing gan fingerprints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7556–7566, 2019\n2. Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3247–3258. PMLR, 2020. ", "summary_of_the_paper": "This paper proposes a framework to perform source identification using wavelet-packet-based analysis of GAN-generated images. They implement boundary-wavelet filters which make use of the share identical network architectures across different wavelet lengths to perform GAN source classification.", "summary_of_the_review": "The paper lacks enough novelty in the technical approach. The authors tackle a very simple problem of source identification. The authors try to perform evaluation on unseen GANs but lack of baselines to compare it with makes the analysis redundant. The experimental evaluation is not thorough across different datasets for deepfake detection which seems to be the main motivation of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qsJV8oFQRMJ", "reviewer": "Reviewer_qN3N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a very simple solution for deepfake detection based on wavelet representation.\nStrength:\n+ The wavelet has not been exploited in deepfake detection, which can provide multiscale representation from low-frequency representation to high-frequency representation.\n+ From the experiments shown in this paper, the wavelet representation looks promising in deepfake detection, that many features at different stages can successful perform deepfake detection, and they achieve better results than image or DCT based representation.=\n+ Overall, the paper is clearly written and easy to follow.\n\nWeakness:\n- The solution is trivial. It seems more study and designs can be exploited. Since wavelet provides multiscale representation, how can you leverage those features together for deepfake detection. However, this paper only simply learns a CNN to achieve classification. The idea of applying wavelet looks good and promising, but need a better solution for more favorable detection.\n- The experiments are not solid. Even though authors test their methods on a set of GAN models, they don't test it on the synthetic images with attacks, such as cropping, JPEG compression, low-pass filtering etc. Besides, detecting GAN generated images is interesting, however, there are many existing deepfake detection benchmarks. The authors need to show results on the public benchmark and compare your method with more state-of-the-art models.\n- References are not enough. The authors focus on discussing the image and frequency based representation. However, there are many other works in this area recently. Such as CVPR21, ICCV21, IJCAI21.\n  * C. Wang et al., Representative Forgery Mining for Fake Face Detection, CVPR, 2021.\n  * H. Zhao et al., Multi-Attentional Deepfake Detection, CVPR, 2021.\n  * T.Zhao et al., Learning Self-Consistency for Deepfake Detection, ICCV, 2021.\n  * Y. He et al., Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis, IJCAI, 2021.\n\nAs a result, I think this work requires a lot of extensions for publication, and rate this submission as marginally below the acceptance threshold.\n\n\n", "summary_of_the_paper": "This paper leverage wavelet-based representation in deepfake detection. By applying wavelet transformation, several features are obtained. In this work, the authors shows the wavelet features at different stages are more robust than previous widely used frequency or image based representation. The representation can be applied like image itself to perform deepfake detection, for example CNN and PRNU. This paper leverage synthetic images from different GAN models to conduct their experiments, including cross-domain detection from an unknown domain.", "summary_of_the_review": "My main concerns are that the experiments are solid enough and the method is trivial. The authors only apply wavelet to extract multiscale features and test them separately.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dZ7QpOavQzS", "reviewer": "Reviewer_zoMp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nUsage of wavelets to detect GAN generated images is a definite strength of this paper. Both pixel domain and frequency domain methods have been applied in the past to detect GAN generated images. Usage of wavelets integrates these. \n\nThe source separation experiments are also strengths in this paper.\n\nWeakness:\n\nOne main drawback of the paper is that no robustness analysis is done on the proposed method. While the authors mention that they will do this as future work, this analysis is an important part in media forensics and it is essential to know how a detection method would perform when post processing operations are performed on the image. It would have been good if the authors presented the results on at least some post processing operations like JPEG compression and noise addition. \n\nIt will also be good to consider more GAN datasets such as CycleGAN, StarGAN, BIGGAN and others, such that the GANs are all diverse. \n\nMany past papers have shown patterns in the frequency domain for GAN generated images that are flagged due to upsampling. It will be good to compare those patterns and the wavelet patterns side by side. Patterns can be found in:\n\nRef (a). Zhang, Xu, Svebor Karaman, and Shih-Fu Chang. \"Detecting and simulating artifacts in gan fake images.\" In 2019 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1-6. IEEE, 2019.\n\nRef (b). Wang, Sheng-Yu, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. \"CNN-generated images are surprisingly easy to spot... for now.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8695-8704. 2020.\n\n", "summary_of_the_paper": "This paper presents an approach to detect GAN generated Deepfake images using wavelet features. The paper also showed linear separation of GANs using the proposed approach. Experiments presented on many datasets showed that the approach is promising and presents a method that integrates both spatial and frequency domain information. ", "summary_of_the_review": "While the paper provides an interesting method to detect GAN generated images using wavelets, thus combining pixel and frequency domain, more experiments are needed to provide a good assessment of the method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W89SVcwZvxT", "reviewer": "Reviewer_sHXs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well structured, mathematical details are well presented and experimental results demonstrate the effectiveness of the technique. However I have some concerns:\n\nSection 4.2 and 4.3 presents results obtained on different image-sets. Please compare with more advanced SOTA techniques (as reported in related work)\n\nGiven the demonstration that first layers of CNNs encode something extremely similar to wavelets (Mallat, Stéphane. \"Understanding deep convolutional networks.\" Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374.2065 (2016): 20150203.) please provide explanation of the contribution of employing wavelet + CNN and not directly a CNN without the wavelet extraction phase.\n\nMoreover the method does not take into account any kind of manipulation or transformation on images. How does compression affects the technique? Resizing? Rotation? Etc.\n\nMany cited papers obtain better results in detection and classification. The authors compare for parameters. There are works that employs analytical solutions with extremely few parameters. They should be taken into account and improvements w.r.t. them should be emphasized", "summary_of_the_paper": "This paper presents a wavelet-based approach for deepfake image detection. This approach is the first using wavelet for the task and could be of great contribution to the community mixing spatial and frequency information. Experimental results are presented for:\n\nBinary classification (FFHQ vs. Stylegan 1/2)\nCelebA+LSUN vs. GANs\nUnknown generator test\nComparison with SOTA are carried out only vs. none of the best paper on the field", "summary_of_the_review": "The paper is ok... but lacks of a contribution for community in my opinion and does not advance state of the art in any terms (computations speed, parameters, overall accuracy results, explaibaility).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5Rk-uEGIMpo", "reviewer": "Reviewer_SoZC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#Strengths:\n1) The study pursued should be pursued with importance given the rise in deepfake images in the internet.\n2) The methods section is well written.\n\n#Weaknesses:\nThe paper can be improved based on the following points:\n1) In section 1, it would be better to include a motivational case study to emphasize the importance of the research being pursued for a broader group of readers to be motivated.\n2) Please briefly mention why previous works only relied on identifying fingerprints in either the spatial or frequency domain? What were the main challenges associated and why/how is this research work is able to overcome that? This wasn’t mentioned properly in section 1 or 2.\n3) In section 3.3, for Fast Wavelet Transform (FWT) please briefly mention the use of FWT and its associated justification. Please do the same for 3.4 i.e. Boundary Wavelet as well. \n4) Though in section 3 the different wavelet methods are mentioned well, for the equations the symbols used are not intuitive and explanatory. Please specify them properly.\n5) In the experimental section, the comparative study seemed to be weak. Why were the methods chosen? Could you please provide a variety of comparative study to prove efficacy?", "summary_of_the_paper": "In this paper, the authors propose to learn a model for the detection of synthetic images based on the wavelet-packet representation of natural and generative adversarial neural network (GAN)-generated images. In experimental evaluation, the proposed method was compared with FFHQ, CelebA, and LSUN source identification problems.", "summary_of_the_review": "The paper is well written in some places, however, does has the opportunity to be improved, especially it was lacking justification in some places and could be helpful to include some more comparative study to prove the efficacy. Please refer to the points mentioned under weaknesses section in the main review to get details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 5 quality reviewers, where 3 of them rated 5 and 2 rated 3. While this work has merits, many concerns are raised by various reviewers. The AC agrees with the reviewers that this paper is not ready for publication at its current form.
Decision: Reject