Wavelet-Packet Powered Deepfake Image Detection

1 INTRODUCTION . While GANs can improve traffic prediction ( Aigner & Körner , 2019 ) , extract useful representations from data ( Radford et al. , 2015 ) , translate textual descriptions into images ( Zhang et al. , 2017 ) , transfer scene and style information between images ( Zhu et al. , 2017 ) , and generate original art ( Tan et al. , 2017 ) , they can also enable abusive actors to quickly and easily generate potentially damaging , highly realistic fake images , colloquially called deepfakes ( Parkin ; Frank et al. , 2020 ) . As the internet becomes a more prominent virtual public space for political discourse and social media outlet ( Applebaum , 2020 ) , deepfakes present a looming threat to its integrity that must be met with techniques for differentiating the real and trustworthy from the fake . Previous algorithmic techniques for separating real from computer-hallucinated images of people have relied on identifying fingerprints in either the spatial ( Yu et al. , 2019 ; Marra et al. , 2019 ) or frequency ( Frank et al. , 2020 ; Dzanic et al. , 2020 ) domain . However , to the best of our knowledge , no techniques have jointly considered the two domains in a multi-scale fashion . We believe our virtual public spaces and social media outlets will benefit from a growing , diverse toolbox of techniques enabling automatic detection of GAN-generated content , where the introduced waveletbased approach provides a competitive and interpretable addition . In this paper , we make the following contributions : • We present a principled wavelet-packet-based analysis of GAN-generated images . Compared to existing work in the frequency domain , we examine the spatio-frequency properties of GAN-generated content for the first time . We find significant differences between real and synthetic images in both the wavelet-packet mean and standard deviation , with increasing frequency and at the edges . • To the best of our knowledge , we present the first application and implementation of boundary wavelets for image analysis in the deep learning context . Proper implementation of boundary-wavelet filters allows us to share identical network architectures across different wavelet lengths . • As a result of the aforementioned wavelet-based analysis , we build classifiers to identify image sources . We work with fixed seed values for reproducibility and report mean and standard deviations over five runs whenever possible . Our systematic analysis shows slightly improved or competitive performance . The source code for our wavelet toolbox , the first publicly available toolbox for fast boundary-wavelet transforms in the python world , and our experiments is available at https : //github.com/ wavelet-detection-review . 2 RELATED WORK . 2.1 GENERATIVE ADVERSARIAL NETS . The advent of GANs ( Goodfellow et al. , 2014 ) heralded several successful image generation projects . For a recent review on GAN theory , architecture , algorithms , and applications , we refer the reader to Gui et al . ( 2020 ) . We highlight the contribution of the progressive growth technique , in which a small network is trained on smaller images then gradually increased in complexity/number of weights and image size , on the ability of optimization to converge at high resolutions of 1024× 1024 pixels ( Karras et al. , 2018 ) . As a supplement , style transfer methods ( Gatys et al. , 2016 ) have been integrated into new style-based generators . The resulting style-GANs have increased the statistical variation of the hallucinated faces ( Karras et al. , 2019 ; 2020 ) . Finally , regularization ( e.g. , using spectral methods ) , has increased the stability of the optimization process ( Miyato et al. , 2018 ) . The recent advances have allowed large-scale training ( Brock et al. , 2019 ) , which in turn improves the quality further . 2.2 WAVELETS . Originally developed within applied mathematics ( Mallat , 1989 ; Daubechies , 1992 ) , wavelets are a long-established part of the image analysis and processing toolbox ( Taubman & Marcellin , 2002 ; Strang & Nguyen , 1996 ; Jensen & la Cour-Harbo , 2001 ) . In the world of traditional image coding , wavelet-based codes have been developed to replace techniques based on the discrete cosine transform ( DCT ) ( Taubman & Marcellin , 2002 ) . Early applications of wavelets in machine learning studied the integration of wavelets into neural networks for function learning ( Zhang et al. , 1995 ) . Within this line of work , wavelets have previously served as input features ( Chen et al. , 2015 ) , or as early layers of scatter nets ( Mallat , 2012 ; Cotter , 2020 ) . Deeper inside neural networks , wavelets have appeared within pooling layers using static Haar ( Wang et al. , 2020c ; Williams & Li , 2018 ) or adaptive wavelets ( Wolter & Garcke , 2021 ) . Within the subfield of generative networks , ongoing concurrent research ( Gal et al. , 2021 ) is exploring the use of wavelets to improve GAN generated image content . 2.3 DEEPFAKE DETECTION . Previous work on deepfake image detection broadly falls into two categories . The first group works in the frequency domain . Projects include the study of natural and deep network generated images in the frequency space created by the DCT , as well as detectors based on Fourier features ( Zhang et al. , 2019 ; Durall et al. , 2019 ; Dzanic et al. , 2020 ; Frank et al. , 2020 ; Durall et al. , 2020 ; Giudice et al. , 2021 ) . In particular , Frank et al . ( 2020 ) visualizes the DCT transformed images and identifies artifacts created by different upsampling methods . We learn that the transformed images are efficient classification features , which allow significant parameter reductions in GAN identification classifiers . Instead of relying on the DCT , Dzanic et al . ( 2020 ) studied the distribution of Fourier coefficients for real and GAN-generated images . After noting significant deviations of the mean frequencies for GAN generated- and real-images , classifiers are trained . Similarly , Giudice et al . ( 2021 ) studies statistics of the DCT coefficients and uses estimations for each GAN-engine for classification . Dzanic et al . ( 2020 ) found high-frequency discrepancies for GAN-generated imagery , built detectors , and spoofed the newly trained Fourier-classifiers by manually adapting the coefficients in Fourier-space . Finally , He et al . ( 2021 ) combine 2d-FFT and DCT features and observed improved performance . The second group of classifiers works in the spatial or pixel domain , among others , ( Yu et al. , 2019 ; Wang et al. , 2020a ; b ; Zhao et al. , 2021a ; b ) train ( convolutional ) neural networks directly on the raw images to identify various GAN architectures . Building upon pixel-CNN Wang & Deng ( 2021 ) adds neural attention . According to ( Yu et al. , 2019 ) , the classifier features in the final layers constitute the fingerprint for each GAN and are interesting to visualize . Instead of relying on deep-learning , ( Marra et al. , 2019 ) proposed GAN-fingerprints working exclusively with denoising-filter and mean operations , whereas ( Guarnera et al. , 2020 ) computes convolutional traces . To the best of our knowledge , we are the first to combine both schools of thought by proposing to work directly with a spatio-frequency wavelet packet representation in a multi-scale fashion . As we demonstrate in the next section , wavelet packets allow visualization of frequency information while , at the same time , preserving local information to some degree . Note that Tang et al . ( 2021 ) uses the first level of a discrete wavelet transform , this means no multi-scale representation as a pre-processing step to compute spectral correlations between the color bands of an image , which are then further processed to obtain features for a classifier . 3 METHODS . “ Wavelets are localized waves , instead of oscillating forever , they drop to zero. ” This observation , in the words of Strang & Nguyen ( 1996 ) , leads to the first motivation for using wavelet transforms . In contrast to the periodic sine and cosine waves in the Fourier-basis , which are localized in frequency , wavelets are localized in both space and frequency . Since images often contain sharp borders , where pixel values change rapidly , slowly decaying Fourier coefficients caused by such short pulses can fill up the spectrum . Reconstruction of the original pulse depends on the cancellation of most coefficients . The locality of wavelets makes it easier to deal with such pulses . While previous approaches based on the fast Fourier transform ( FFT ) by Dzanic et al . ( 2020 ) and DCT from Frank et al . ( 2020 ) and Giudice et al . ( 2021 ) provide abundant frequency space information , the global nature of these bases means all spatial relations are missing . The desire to extend our knowledge by exploiting spatial information is our second reason to work with wavelets . Wavelets allow the representation of hierarchical levels of detail in space and frequency . Before discussing the mechanics of the fast wavelet transform ( FWT ) and its packet variant , we present a proof of concept experiment in Figure 1 . We investigate the coefficients of the level 3 Haar wavelet packet transform . Computations use 5K 1024× 1024 pixel images from Flickr Faces High Quality ( FFHQ ) ( Karras et al. , 2019 ) and 5K 1024× 1024 pixel images generated by StyleGAN . The leftmost columns of Figure 1 show a sample from both FFHQ and a StyleGAN generated one . Mean wavelet packets appear in the second , and their standard deviation in the third column . Finally , the rightmost column plots the absolute mean packets and standard deviation differences . For improved visibility , we re-scale the absolute values of the packet representation , averaged first over the three color bands , using the natural logarithm loge . We find that the mean wavelet packets of GAN-generated images are often significantly brighter . The differences become more pronounced as the frequency increases from the top left to the bottom right . The differences in high-frequency packets independently confirm Dzanic et al . ( 2020 ) , who saw the most significant differences for high-frequency Fourier-coefficients . The locality of the wavelet filters allows face-like shapes to appear , but the image edges stand out , allowing us to pinpoint the frequency disparities . We now have an intuition regarding the local origin of the differences . We note that Haar wavelet packets do not use padding or orthogonalization and conclude that the differences stem from GAN generation . For the standard deviations , the picture is reversed . Instead of the GAN the FFHQ-packets appear brighter . The StyleGAN packets do not deviate as much as the data in the original dataset . The observation suggests that the GAN did not capture the complete variance of the original data . Our evidence indicates that GAN-generated backgrounds are of particular monotony across all frequency bands . In the next section , we discuss how the fast wavelet transform ( FWT ) and wavelet packets in Figure 1 are computed . For multi-channel color images , we transform each color separately , but for simplicity , we will only consider single channels in the following exposition . 3.1 FAST WAVELET TRANSFORM . Wavelets can be defined by low- and corresponding high-pass filter or by a wavelet function and a scaling function , where we follow the filter construction , as common in signal processing . The FWT ( Mallat , 1989 ; Strang & Nguyen , 1996 ; Jensen & la Cour-Harbo , 2001 ) utilizes convolutions to decompose an input signal into its frequency components , where repeated application results in a multi-scale analysis . The resulting wavelet coefficients reflect scale ( of level q ) , which is related to frequency , and location , i.e. , the position in the transformed and scaled image of level q . Even though 2D wavelets such as the Mexican hat wavelet exist , one dimensional wavelet pairs are often transformed to 2D quadruples using the outer products ( Vyas et al. , 2018 ) : fa = fLf T L , fh = fLf T H , fv = fHf T L , fd = fHf T H. ( 1 ) Common choices for 1D wavelets are the Daubechies-wavelets ( “ db ” ) and their less asymmetrical variant , the symlets ( “ sym ” ) . We refer the reader to Daubechies ( 1992 ) for an in-depth discussion . In the equations above , f denotes a filter vector . For one-dimensional filters , the subscripts L denote the one-dimensional low-pass and H the high pass filter , respectively . In the two-dimensional case , a denotes the approximation coefficients , h denotes the horizontal coefficients , v denotes vertical coefficients , and d denotes the diagonal coefficients . The 2D transformation at representation level q + 1 requires the input xq as well as a filter quadruple fk for k ∈ [ a , h , v , d ] and is computed by xq ∗ fk = kq+1 , ( 2 ) where ∗ denotes stride two convolution . Note the input image is at level zero , i.e . x0 = I , while for stage q the low pass result of q − 1 is used as input . The wavelet transform is invertible . The forward or analysis transform relies on the analysis filters often denoted as Hk ( Strang & Nguyen , 1996 ) . The inverse or synthesis transform filters are typically written as Fk . The down- ↓2 and up- ↑2 arrow symbols describe the sampling operations in what is known as stride two convolution and transposed convolution in today ’ s machine learning literature . In order to make the transform invertible and plots interpretable , not just any filter will do . The perfect reconstruction and anti-aliasing conditions must hold . We refer to Strang & Nguyen ( 1996 ) and Jensen & la Cour-Harbo ( 2001 ) for an excellent further discussion of these conditions .

Summary: This paper presents a wavelet-based approach for deepfake image detection. This approach is the first using wavelet for the task and could be of great contribution to the community mixing spatial and frequency information. Experimental results are presented for:  Binary classification (FFHQ vs. Stylegan 1/2) CelebA+LSUN vs. GANs Unknown generator test Comparison with SOTA are carried out only vs. none of the best paper on the field