Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
Ning Yu 1,2

Larry Davis 1
Mario Fritz 3
1
University of Maryland, College Park
2
Max Planck Institute for Informatics
Saarland Informatics Campus, Germany
3
CISPA Helmholtz Center for Information Security
Saarland Informatics Campus, Germany

ningyu@mpi-inf.mpg.de

lsd@cs.umd.edu

fritz@cispa.saarland

Abstract
Recent advances in Generative Adversarial Networks(GANs) have shown increasing success in generating photorealistic images. But they also raise challenges to visual forensics and model attribution. We present the firststudy of learning GAN fingerprints towards image attribution and using them to classify an image as real or GANgenerated. For GAN-generated images, we further identify
their sources. Our experiments show that (1) GANs carry
distinct model fingerprints and leave stable fingerprints intheir generated images, which support image attribution;(2) even minor differences in GAN training can result indifferent fingerprints, which enables fine-grained model authentication; (3) fingerprints persist across different image
frequencies and patches and are not biased by GAN artifacts; (4) fingerprint finetuning is effective in immunizing
against five types of adversarial image perturbations; and(5) comparisons also show our learned fingerprints consistently outperform several baselines in a variety of setups 1 .

1. Introduction
In the last two decades, photorealistic image generationand manipulation techniques have rapidly evolved. Visualcontents can now be easily created and edited without leaving obvious perceptual traces [72]. Recent breakthroughs
in generative adversarial networks (GANs) [31, 52, 10, 32,
38, 19] have further improved the quality and photorealism of generated images. The adversarial framework of
GANs can also be used in conditional scenarios for image translation [36, 70, 71] or manipulation in a given context [60, 61, 57, 12, 64], which diversifies media synthesis.
1 Code, data, models, and supplementary material are available at
GitHub.

Figure 1. A t-SNE [43] visual comparison between our fingerprintfeatures (right) and the baseline inception features [52] (left) for
image attribution. Inception features are highly entangled, indicating the challenge to differentiate high-quality GAN-generated
images from real ones. However, our result shows any single difference in GAN architectures, training sets, or even initialization
seeds can result in distinct fingerprint features for effective attribution.

At the same time, however, the success of GANs has
raised two challenges to the vision community: visual
forensics and intellectual property protection.
GAN challenges to visual forensics. There is a
widespread concern about the impact of this technology
when used maliciously. This issue has also received increasing public attention, in terms of disruptive consequences to visual security, laws, politics, and society in general [6, 1, 3]. Therefore, it is critical to look into effective
visual forensics against threats from GANs.
While recent state-of-the-art visual forensics techniquesdemonstrate impressive results for detecting fake visual media [16, 53, 27, 13, 22, 11, 35, 67, 68, 26], they have
only focused on semantic, physical, or statistical inconsistency of specific forgery scenarios, e.g., copy-move
manipulations[16, 26] or face swapping [67]. Forensics on
GAN-generated images [44, 47, 59] shows good accuracy,
but each method operates on only one GAN architecture byidentifying its unique artifacts and results deteriorate when
the GAN architecture is changed. It is still an open questionof whether GANs leave stable marks that are commonly

7556

shared by their generated images. That motivates us to investigate an effective feature representation that differentiates GAN-generated images from real ones.
GAN challenges to intellectual property protection.
Similar to other successful applications of deep learningtechnology to image recognition [33] or natural language
processing [30], building a product based on GANs is nontrivial [37, 4, 5]. It requires a large amount of training data,
powerful computing resources, significant machine learning
expertise, and numerous trial-and-error iterations for identifying optimal model architectures and their model hyperparameters. As GAN services become widely deployedwith commercial potential, they will become increasingly
vulnerable to pirates. Such copyright plagiarism may jeopardize the intellectual property of model owners and takefuture market share from them. Therefore, methods for attributing GAN-generated image origins are highly desirable
for protecting intellectual property.
Given the level of realism that GAN techniques alreadyachieve today, attribution by human inspection is no longer
feasible (see the mixture of Figure 4). The state-of-theart digital identification techniques can be separated into
two categories: digital watermarking and digital fingerprint
detection. Neither of them is applicable to GAN attribution. Previous work on watermarking deep neural networks [65, 62] depends on an embedded security scheme
during “white-box” model training, requires control of the
input, and is impractical when only GAN-generated images
are accessible in a “black-box” scenario. Previous work on
digital fingerprints is limited to device fingerprints [42, 21]
or in-camera post-processing fingerprints [24], which cannot be easily adapted to GAN-generated images. That motivates us to investigate GAN fingerprints that attribute different GAN-generated images to their sources.
We present the first study addressing the two GAN challenges simultaneously by learning GAN fingerprints for image attribution: We introduce GAN fingerprints and use
them to classify an image as real or GAN-generated. For
GAN-generated images, we further identify their sources.
We approach this by training a neural network classifier andpredicting the source of an image. Our experiments showthat GANs carry distinct model fingerprints and leave stablefingerprints in their generated images, which support image
attribution.
We summarize our contributions as demonstrating theexistence, uniqueness, persistence, immunizability, and visualization of GAN fingerprints. We address the following
questions:
Existence and uniqueness: Which GAN parameters differentiate image attribution? We present experiments
on GAN parameters including architecture, training data, as
well as random initialization seed. We find that a differencein any one of these parameters results in a unique GAN fin-gerprint for image attribution. See Figure 1, Section 3.1 and
4.2.
Persistence: Which image components contain fingerprints for attribution? We investigate image components in different frequency bands and in different patch
sizes. In order to eliminate possible bias from GAN artifact components, we apply a perceptual similarity metric
to distill an artifact-free subset for attribution evaluation.
We find that GAN fingerprints are persistent across different frequencies and patch sizes, and are not dominated byartifacts. See Section 3.2 and 4.3.
Immunizability: How robust is attribution to image perturbation attacks and how effective are the defenses?
We investigate common attacks that aim at destroying image fingerprints. They include noise, blur, cropping, JPEG
compression, relighting, and random combinations of them.
We also defend against such attacks by finetuning our attribution classifier. See Section 4.4.
Visualization: How to expose GAN fingerprints? We
propose an alternative classifier variant to explicitly visualize GAN fingerprints in the image domain, so as to betterinterpret the effectiveness of attribution. See Section 3.3
and 4.5.
Comparison to baselines. In terms of attribution accuracy, our method consistently outperforms three baseline
methods (including a very recent one [45]) on two datasets
under a variety of experimental conditions. In terms offeature representation, our fingerprints show superior distinguishability across image sources compared to inception
features [52].

2. Related work
Generative Adversarial Networks (GANs). GANs [31,52, 10, 32, 38, 19] have shown improved photorealism inimage synthesis [40, 15, 69], translation [36, 70, 71], or manipulation [9, 60, 61]. We focus on unconditional GANs as
the subject of our study. We choose the following four GAN
models as representative candidates of the current state of
the art: ProGAN [38], SNGAN [46], CramerGAN [14],and MMDGAN [17], considering their outstanding performances on face generation.
Visual forensics. Visual forensics targets detecting statistical or physics-based artifacts and then recognizing the authenticity of visual media without evidence from an embedded security mechanism [28, 27]. An example is asteganalysis-based method [29], which uses hand-craftedfeatures plus a linear Support Vector Machine to detect forgeries. Recent CNN-based methods [13, 22, 18, 11, 35, 67,
68, 7, 23, 26] learn deep features and further improve tampering detection performance on images or videos. Rössler
et al. [49, 50] introduced a large-scale face manipulation
dataset to benchmark forensics classification and segmentation tasks, and demonstrated superior performance when using additional domain-specific knowledge. For forensics on7557

GAN-generated images, several existing works [44, 47, 59]
show good accuracy. However, each method considers only
one GAN architecture and results do not generalize acrossarchitectures.
Digital fingerprints. Prior digital fingerprint techniques
focus on detecting hand-crafted features for either device
fingerprints or postprocessing fingerprints. The device fingerprints rely on the fact that individual devices, due tomanufacturing imperfections, leave a unique and stable
mark on each acquired image, i.e., the photo-response nonuniformity (PRNU) pattern [42, 21]. Likewise, postprocessing fingerprints come from the specific in-camera postprocessing suite (demosaicking, compression, etc.) during
each image acquisition procedure [24]. Recently, Marra etal. [45] visualize GAN fingerprints based on PRNU, andshow their application to GAN source identification. We
replace their hand-crafted fingerprint formulation with alearning-based one, decoupling model fingerprint from image fingerprint, and show superior performances in a variety
of experimental conditions.
Digital watermarking. Digital watermarking is a complementary forensics technique for image authentication [58, 39, 51]. It involves embedding artificial watermarks in images. It can be used to reveal image source
and ownership so as to protect their copyright. It has beenshown that neural networks can also be actively watermarked during training [65, 62]. In such models, a characteristic pattern can be built into the learned representation
but with a trade-off between watermarking accuracy and theoriginal performance. However, such watermarking has notbeen studied for GANs. In contrast, we utilize inherent fingerprints for image attribution without any extra embedding
burden or quality deterioration.

3. Fingerprint learning for image attribution
Inspired by the prior works on digital fingerprints [42,24], we introduce the concepts of GAN model fingerprint
and image fingerprint. Both are simultaneously learned
from an image attribution task.
Model fingerprint. Each GAN model is characterized bymany parameters: training dataset distribution, network architecture, loss design, optimization strategy, and hyperparameter settings. Because of the non-convexity of theobjective function and the instability of adversarial equilibrium between the generator and discriminator in GANs, thevalues of model weights are sensitive to their random initializations and do not converge to the same values during eachtraining. This indicates that even though two well-trained
GAN models may perform equivalently, they generate highquality images differently. This suggests the existence anduniqueness of GAN fingerprints. We define the model fingerprint per GAN instance as a reference vector, such thatit consistently interacts with all its generated images. In a

specifically designed case, the model fingerprint can be an
RGB image the same size as its generated images. See Section 3.3.
Image fingerprint. GAN-generated images are the outcomes of a large number of fixed filtering and non-linear
processes, which generate common and stable patterns
within the same GAN instances but are distinct across different GAN instances. That suggests the existence of image
fingerprints and attributability towards their GAN sources.
We introduce the fingerprint per image as a feature vector
encoded from that image. In a specifically designed case,an image fingerprint can be an RGB image the same size as
the original image. See Section 3.3.

3.1. Attribution network
Similar to the authorship attribution task in natural language processing [56, 8], we train an attribution classifierthat can predict the source of an image: real or from a GAN
model.
We approach this using a deep convolutional neural network supervised by image-source pairs {(I, y)} where I ∼I is sampled from an image set and y ∈ Y is the source
ground truth belonging to a finite set. That set is composed of pre-trained GAN instances plus the real world.
Figure 2(a) depicts an overview of our attribution network.
We implicitly represent image fingerprints as the finalclassifier features (the 1 × 1 × 512 tensor before the final fully connected layer) and represent GAN model fingerprints as the corresponding classifier parameters (the1× 1× 512 weight tensor of the final fully connected layer).
Why is it necessary to use such an external classifier
when GAN training already provides a discriminator? The
discriminator learns a hyperplane in its own embedding
space to distinguish generated images from real ones. Different embedding spaces are not aligned. In contrast, theproposed classifier necessarily learns a unified embedding
space to distinguish generated images from different GAN
instances or from real images.
Note that our motivation to investigate “white-box”
GANs subject to known parameters is to validate the attributability along different GAN parameter dimensions. In
practice, our method also applies to “black-box” GAN API
services. The only required supervision is the source labelof an image. We can simply query different services, collect
their generated images, and label them by service indices.
Our classifier would test image authenticity by predicting if
an image is sampled from the desired service. We also test
service authenticity by checking if most of their generated
images have the desired source prediction.

3.2. Component analysis networks
In order to analyze which image components contain fingerprints, we propose three variants of the network.
7558

(a)

(b)

(c)

(d)

Figure 2. Different attribution network architectures. Tensor representation is specified by two spatial dimensions followed by the
number of channels. The network is trained to minimize crossentropy classification loss. (a) Attribution network. (b) Predownsampling network example that downsamples input image to8 × 8 before convolution. (c) Pre-downsampling residual network
example that extracts the residual component between 16×16 and8 × 8 resolutions. (d) Post-pooling network example that starts average pooling at 64 × 64 resolution.

Pre-downsampling network. We propose to test whether
fingerprints and attribution can be derived from different
frequency bands. We investigate attribution performance
w.r.t. downsampling factor. Figure 2(b) shows an architecture example that extracts low-frequency bands. We replace
the trainable convolution layers with our Gaussian downsampling layers from the input end and systematically control at which resolution we stop such replacement.
Pre-downsampling residual network. Complementary
to extracting low-frequency bands, Figure 2(c) shows an architecture example that extracts a residual high-frequency
band between one resolution and its factor-2 downsampled
resolution. It is reminiscent of a Laplacian Pyramid [20].
We systematically vary the resolution at which we extractsuch residual.
Post-pooling network. We propose to test whether fingerprints and attribution can be derived locally based onpatch statistics. We investigate attribution performance
w.r.t. patch size. Figure 2(d) shows an architecture example.
Inspired by PatchGAN [36], we regard a “pixel” in a neuraltensor as the feature representation of a local image patch
covered by the receptive field of that “pixel”. Therefore,
post-pooling operations count for patch-based neural statistics. Earlier post-pooling corresponds to a smaller patch
size. We systematically vary at which tensor resolution we
start this pooling in order to switch between more local andmore global patch statistics.

Figure 3. Fingerprint visualization diagram. We train an AutoEncoder and GAN fingerprints end-to-end. ⊙ indicates pixel-wise
multiplication of two normalized images.

learn fingerprint images from image-source pairs ({I, y}).
We also decouple the representation of model fingerprints
from image fingerprints. Figure 3 depicts the fingerprint visualization model.
Abstractly, we learn to map from input image to its fingerprint image. But without fingerprint supervision, wechoose to ground the mapping based on a reconstruction
task with an AutoEncoder. We then define the reconstruction residual as the image fingerprint. We simultaneously
learn a model fingerprint for each source (each GAN instance plus the real world), such that the correlation index
between one image fingerprint and each model fingerprint
serves as softmax logit for classification.
Mathematically, given an image-source pair (I, y) wherey ∈ Y belongs to the finite set Y of GAN instances plus thereal world, we formulate a reconstruction mapping R fromI to R(I). We ground our reconstruction based on pixelwise L1 loss plus adversarial loss:
Lpix (I) = ||R(I) − I||1
(1)



Ladv (I) = Drec R(I) − Drec I + GP R(I), I|Drec (2)where Drec is an adversarially trained discriminator, andGP(·) is the gradient penalty regularization term definedin [32].
I
We then explicitly define image fingerprint Fimas theI
reconstruction residual Fim = R(I) − I.
y
We further explicitly define model fingerprint FmodasI
freely trainable parameters with the same size as Fim , suchyI
Ithat corr(Fim, Fmod), the correlation index between Fim
andyFmod , is maximized over Y. This can be formulated as thesoftmax logit for the cross-entropy classification loss supervised by the source ground truth:
Lcls (I, y) = − log P

3.3. Fingerprint visualization
Alternatively to our attribution network in Section 3.1
where fingerprints are implicitly represented in the featuredomain, we describe a model similar in spirit to Marra etal. [45] to explicitly represent them in the image domain.
But in contrast to their hand-crafted PRNU-based representation, we modify our attribution network architecture andyIcorr(Fim, Fmod)ŷIŷ∈Y corr(Fim , Fmod )(3)

where corr(A, B) = Â ⊙ B̂, Â and B̂ are the zero-mean,
unit-norm, and vectorized version of images A and B, and⊙ is the inner product operation.
Our final training objective isminỹR,{Fmod|ỹ∈Y}

7559

max
Drec

E

{(I,y)}(λ1 Lpix + λ2 Ladv + λ3 Lcls ) (4)(a) CelebA real data(b) ProGAN(c) SNGAN(d) CramerGAN(e) MMDGAN

Figure 4. Face samples from difference sources.

where λ1 = 20.0, λ2 = 0.1, and λ3 = 1.0 are used tobalance the order of magnitude of each loss term, which are
not sensitive to dataset and are fixed.
Note that this network variant is used to better visualize
and interpret the effectiveness of image attribution. However, it introduces extra training complexity and thus is not
used if we only focus on attribution.

4. Experiments
We discuss the experimental setup in Section 4.1. From
Section 4.2 to 4.5, we explore the four research questions
discussed in the Introduction.

4.1. Setup
Datasets . We employ CelebA human face dataset [41]and LSUN bedroom scene dataset [63], both containing
20, 000 real-world RGB images.
GAN models. We consider four recent state-of-the-art
GAN architectures: ProGAN [38], SNGAN [46], CramerGAN [14], and MMDGAN [17]. Each model is trained
from scratch with their default settings except we fix thenumber of training epochs to 240 and fix the output size ofa generator to 128 × 128 × 3.
Baseline methods. Given real-world datasets and four
pre-trained GAN models, we compare with three baselineclassification methods: k-nearest-neighbor (kNN) on raw
pixels, Eigenface [55], and the very recent PRNU-based fingerprint method from Marra et al. [45].
Evaluation. We use classification accuracy to evaluate
image attribution performance.
In addition, we use the ratio of inter-class and intra-classFréchet Distance [25], denoted as FD ratio, to evaluate thedistinguishability of a feature representation across classes.
The larger the ratio, the more distinguishable the feature
representation across sources. See supplementary material
for more detail. We compare our fingerprint features to image inception features [52]. The FD of inception features
is also known as FID for GAN evaluation [34]. Therefore,
the FD ratio of inception features can serve as a reference toshow how challenging it is to attribute high-quality GANgenerated images manually or without fingerprint learning.

4.2. Existence and uniqueness: which GAN parameters differentiate image attribution?
We consider GAN architecture, training set, and initialization seed respectively by varying one type of parameterand keeping the other two fixed.
Different architectures. First, we leverage all the real
images to train ProGAN, SNGAN, CramerGAN, and
MMDGAN separately. For the classification task, we configure training and testing sets with 5 classes: {real, ProGAN, SNGAN, CramerGAN, MMDGAN}. We randomly
collect 100, 000 images from each source for classification
training and another 10, 000 images from each source fortesting. We show face samples from each source in Figure 4 and bedroom samples in the supplementary material.
Table 1 shows that we can effectively differentiate GANgenerated images from real ones and attribute generated images to their sources, just using a regular CNN classifier.
There do exist unique fingerprints in images that differentiate GAN architectures, even though it is far more challenging to attribute those images manually or through inception
features [52].
Different GAN training sets. We further narrow down
the investigation to GAN training sets. From now we only
focus on ProGAN plus real dataset. We first randomly select a base real subset containing 100, 000 images, denoted
as real subset diff 0. We then randomly select 10 otherreal subsets also containing 100, 000 images, denoted asreal subset diff #i, where i ∈ {1, 10, 100, 1000, 10000,
20000, 40000, 60000, 80000, 100000} indicates the number of images that are not from the base subset. We collectsuch sets of datasets to explore the relationship between attribution performance and GAN training set overlaps.
For each real subset diff #i, we separately train a ProGAN model and query 100, 000 images for classifier
training and another 10, 000 images for testing, labeled
as ProGAN subset diff #i. In this setup of {real, ProGAN subset diff #i}, we show the performance evaluationin Table 2. Surprisingly, we find that attribution performance remains equally high regardless of the amount of
GAN training set overlap. Even GAN training sets that differ in just one image can lead to distinct GAN instances.
That indicates that one-image mismatch during GAN training results in a different optimization step in one iteration

7560

Table 1. Evaluation on {real, ProGAN, SNGAN, CramerGAN,
MMDGAN}. The best performance is highlighted in bold.

Accuracy(%)
FD ratio

CelebA

LSUN

kNN
Eigenface [55]PRNU [45]
Ours

28.00
53.28
86.61
99.43

36.30
67.84
98.58

Inception [52]
Our fingerprint

2.36
454.765.27
226.59

Table 4. Classification accuracy (%) of our network w.r.t. downsampling factor on low-frequency or high-frequency componentsof {real, ProGAN seed v#i}. “L-f” column indicates the lowfrequency components and represents the performances from thepre-downsampling network. “H-f” column indicates the highfrequency components and represents the performances from thepre-downsampling residual network.

Table 2. Evaluation on {real, ProGAN subset diff #i}. The best
performance is highlighted in bold.

Accuracy(%)
FD ratio

CelebA

LSUN

kNN
Eigenface [55]PRNU [45]
Ours

11.46
27.98
92.28
99.50

10.72
70.55
97.66

Inception [52]
Our fingerprint

1.08
111.41

1.64
39.96

Accuracy(%)

FD ratio

10.88
23.12
89.40
99.14
97.07

10.58
69.73
97.04
96.58

Inception [52]
Our fingerprint

1.10
80.28

1.29
36.48

and finally results in distinct fingerprints. That motivates
us to investigate the attribution performance among GAN
instances that were trained with identical architecture anddataset but with different random initialization seeds.
Different initialization seeds. We next investigate theimpact of GAN training initialization on image attributability. We train 10 ProGAN instances with the entire real
dataset and with different initialization seeds. We sample 100, 000 images for classifier training and another
10, 000 images for testing. In this setup of {real, ProGAN seed v#i} where i ∈ {1, ..., 10}, we show the performance evaluation in Table 3. We conclude that it is the difference in optimization (e.g., caused by different randomness) that leads to attributable fingerprints. In order to verify our experimental setup, we ran sanity checks. For example, two identical ProGAN instances trained with the sameseed remain indistinguishable and result in random-chance
attribution performance.

Resolution

CelebA
L-f
H-f

LSUN
L-f
H-f

1
2
4
8
16
32

1282
642
322
162
82
42

99.14
98.74
95.50
87.20
67.44
26.58

97.04
96.78
91.08
83.02
63.80
28.24

99.14
98.64
98.52
92.90
78.74
48.42

97.04
96.84
96.04
91.58
80.58
54.50

Table 5. Classification accuracy (%) of our network w.r.t. patch
size on {real, ProGAN seed v#i}.
Pooling starts at
2

4
82
162
322
642
1282

Table 3. Evaluation on {real, ProGAN seed v#i}. The best performance is highlighted in bold. “Our visNet” row indicates our fingerprint visualization network described in Section 3.3 and evaluated in Section 4.5.
CelebA LSUNkNN
Eigenface [55]PRNU [45]Ours
Our visNet

Downsample
factorPatch size
2

128
1082
522
242
102
32

CelebA

LSUN

99.34
99.32
99.30
99.24
89.60
13.42

97.44
96.30
95.94
88.36
18.26
17.10

4.3. Persistence: which image components containfingerprints for attribution?
We systematically explore attribution performance w.r.t.
image components in different frequency bands or with different patch sizes. We also investigate possible performancebias from GAN artifacts.
Different frequencies. We investigate if band-limited images carry effective fingerprints for attribution. We separately apply the proposed pre-downsampling network andpre-downsampling residual network for image attribution.
Given the setup of {real, ProGAN seed v#i}, Table 4 shows
the classification accuracy w.r.t. downsampling factors.
We conclude that (1) a wider frequency band carries more
fingerprint information for image attribution, (2) the lowfrequency and high-frequency components (even at the resolution of 8×8) individually carry effective fingerprints andresult in attribution performance better than random, and (3)at the same resolution, high-frequency components carry
more fingerprint information than low-frequency components.
Different local patch sizes. We also investigate if local
image patches carry effective fingerprints for attribution.
We apply the post-pooling network for image attribution.
Given the setup of {real, ProGAN seed v#i}, Table 5 shows
the classification accuracy w.r.t. patch sizes. We conclude
that for CelebA face dataset a patch of size 24 × 24 or larger
carries sufficient fingerprint information for image attribution without deterioration; for LSUN, a patch of size 52×527561

4.4. Immunizability: how robust is attribution toimage perturbation attacks and how effective
are the defenses?

(a) Non-selected samples(b) Selected samples

Figure 5. Visual comparisons between (a) arbitrary face samplesand (b) selected samples with top 10% Perceptual Similarity [66]
to CelebA real dataset. We notice the selected samples have higher
quality and fewer artifacts. They are also more similar to eachother, which challenge more on attribution.
Table 6. Evaluation on the 10% selected images of {real, ProGAN seed v#i}. The best performance is highlighted in bold.

Accuracy(%)
FD ratio

CelebA

LSUN

kNN
Eigenface [55]PRNU [45]
Ours

11.99
26.69
93.50
99.93

10.35
74.49
98.16

Inception [52]
Our fingerprint

1.04
15.63

1.22
6.27

or larger carries a sufficient fingerprint.

Artifact-free subset. Throughout our experiments, thestate-of-the-art GAN approaches are capable of generating
high-quality images – but are also generating obvious artifacts in some cases. There is a concern that attribution
might be biased by such artifacts. In order to eliminate thisconcern, we use Perceptual Similariy [66] to measure the 1nearest-neighbor similarity between each testing generated
image and the real-world dataset, and then select the 10%
with the highest similarity for attribution. We compare facesamples between non-selected and selected sets in Figure 5
and compare bedroom samples in the supplementary material. We notice this metric is visually effective in selecting
samples of higher quality and with fewer artifacts.
Given the setup of 10% selected {real, ProGAN seed v#i}, we show the performance evaluationin Table 6. All the FD ratio measures consistently decreased compared to Table 3. This indicates our selection
also moves the image distributions from different GAN
instances closer to the real dataset and consequently
closer to each other. This makes the attribution task morechallenging. Encouragingly, our classifier, pre-trainedon non-selected images, can perform equally well on theselected high-quality images and is hence not biased byartifacts.

Attacks. We apply five types of attacks that perturb testing images [48]: noise, blur, cropping, JPEG compression,
relighting, and random combination of them. The intention
is to confuse the attribution network by destroying image
fingerprints. Examples of the perturbations on face images
are shown in Figure 6. Examples on bedroom images areshown in the supplementary material.
Noise adds i.i.d. Gaussian noise to testing images. The
Gaussian variance is randomly sampled from U [5.0, 20.0].
Blur performs Gaussian filtering on testing images with kernel size randomly picked from {1, 3, 5, 7, 9}. Cropping crops testing images with a random offset between 5%
and 20% of the image side lengths and then resizes back to
the original. JPEG compression performs JPEG compression processing with quality factor randomly sampled fromU [10, 75]. Relighting uses SfSNet [54] to replace the current image lighting condition with another random one fromtheir lighting dataset. The combination performs each attack with a 50% probability in the order of relighting, cropping, blur, JPEG compression, and noise.
Given perturbed images and the setup of {real, ProGAN seed v#i}, we show the pre-trained classifier performances in the “Akt” columns in Table 7 and Table 8. All
performances decrease due to attacks. In detail, the classifier completely fails to overcome noise and JPEG compression attacks. It still performs better than random whenfacing the other four types of attacks. The relighting attack is the least effective one because it only perturbs lowfrequency image components. The barely unchanged fingerprints in high-frequency components enables reasonableattribution.
Defenses. In order to immunize our classifier against attacks, we finetune the classifier under the assumption that
we know the attack category. Given perturbed images andthe setup of {real, ProGAN seed v#i}, we show the finetuned classifier performance in the “Dfs” columns in Table 7 and Table 8. It turns out that the immunized classifiercompletely regains performance over blur, cropping and relighting attacks, and partially regains performance over theothers. However, the recovery from combination attack isminimal due to its highest complexity. In addition, ourmethod consistently outperforms the method of Marra etal. [45] under each attack after immunization, while theirs
does not effectively benefit from such immunization.

4.5. Fingerprint visualization
Given the setup of {real, ProGAN seed v#i}, we alternatively apply the fingerprint visualization network (Section 3.3) to attribute images. We show the attribution performance in the “Our visNet” row in Table 3, which are com-7562

(a) No attack(b) Noise(c) Blur(d) Cropping(e) Compression(f) Relighting(g) Combination

Figure 6. Image samples for the attacks and defenses of our attribution network.
Table 7. Classification accuracy (%) of our network w.r.t. different perturbation attacks before or after immunization on CelebA {real,
ProGAN seed v#i}. The best performance is highlighted in bold.

PRNU [45]
Ours

Noise
Atk
Dfs

Atk

Dfs

CelebA
Cropping
Compression
Atk
Dfs
Atk
Dfs

57.88
9.14

27.37
49.64

42.43
97.20

9.84
46.80

63.82
93.02

Blur

10.68
98.28

26.15
8.77

44.55
88.02

Relighting
Atk
Dfs

Combination
Atk
Dfs

86.59
94.02

19.93
19.31

87.02
98.66

21.77
72.64

Table 8. Classification accuracy (%) of our network w.r.t. different perturbation attacks before or after immunization on LSUN bedroom{real, ProGAN seed v#i}. The best performance is highlighted in bold.

PRNU [45]
Ours

Noise
Atk
Dfs

Atk

39.59
11.80

26.92
74.48

40.97
95.30

Dfs

LSUN
Cropping
Compression
Atk
Dfs
Atk
Dfs

Relighting
Atk
Dfs

Combination
Atk
Dfs

30.79
96.68

9.30
86.20

60.86
62.21

16.54
24.44

Blur

9.42
97.30

18.27
24.73

23.66
92.40

63.31
97.36

16.89
83.42

approximate realistically [2].

5. Conclusion

Figure 7. Visualization of model and image fingerprint samples.
Their pairwise interactions are shown as the confusion matrix.

parable to that of the attribution model. Figure 7 visualizes
face fingerprints. Bedroom fingerprints are shown in the
supplementary material. It turns out that image fingerprints
maximize responses only to their own model fingerprints,
which supports effective attribution. To attribute the realworld image, it is sufficient for the fingerprint to focus only
on the eyes. To attribute the other images, the fingerprints
also consider clues from the background, which, compared
to foreground faces, is more variant and harder for GANs to

We have presented the first study of learning GAN fingerprints towards image attribution. Our experiments showthat even a small difference in GAN training (e.g., the difference in initialization) can leave a distinct fingerprint that
commonly exists over all its generated images. That enables
fine-grained image attribution and model attribution. Further encouragingly, fingerprints are persistent across different frequencies and different patch sizes, and are not biased
by GAN artifacts. Even though fingerprints can be deteriorated by several image perturbation attacks, they are effectively immunizable by simple finetuning. Comparisonsalso show that, in a variety of conditions, our learned fingerprints are consistently superior to the very recent baseline [45] for attribution, and consistently outperform inception features [52] for cross-source distinguishability.

Acknowledgement
This project was partially funded by DARPA MediFor
program under cooperative agreement FA87501620191.
We acknowledge the Maryland Advanced Research Computing Center for providing computing resources. We thank
Hao Zhou for helping with the relighting experiments. We
also thank Yaser Yacoob and Abhinav Shrivastava for constructive advice in general.

7563

References[1] Deep fakes: How they are made and how they can bedetected.
https://www.nbcnews.com/mach/video/deepfakes-how-they-are-made-and-how-they-can-be-detected1354417219989. 1[2] Howtorecognize
fakeai-generated
images.
https://medium.com/@kcimc/how-to-recognize-fake-aigenerated-images-4d1f6f9a2842. 8[3] In the age of a.i., is seeing still believing?
https://www.newyorker.com/magazine/2018/11/12/inthe-age-of-ai-is-seeing-still-believing. 1[4] Model gallery. https://www.microsoft.com/en-us/cognitivetoolkit/features/model-gallery. 2[5] The value of stolen data on the dark web.
https://darkwebnews.com/dark-web/value-of-stolen-datadark-web. 2[6] You thought fake news was bad?
deep
fakes
are
where
truth
goes
to
die.
https://www.theguardian.com/technology/2018/nov/12/deepfakes-fake-news-truth. 1[7] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao
Echizen. Mesonet: a compact facial video forgery detection
network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–7. IEEE, 2018.
2[8] Sadia Afroz, Aylin Caliskan Islam, Ariel Stolerman, RachelGreenstadt, and Damon McCoy. Doppelgänger finder: Taking stylometry to the underground. In Security and Privacy (SP), 2014 IEEE Symposium on, pages 212–226. IEEE,
2014. 3[9] Grigory Antipov, Moez Baccouche, and Jean-Luc Dugelay.
Face aging with conditional generative adversarial networks.
In Image Processing (ICIP), 2017 IEEE International Conference on, pages 2089–2093. IEEE, 2017. 2[10] Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214–223,
2017. 1, 2
[11] Jawadul H Bappy, Amit K Roy-Chowdhury, Jason Bunk,
Lakshmanan Nataraj, and BS Manjunath. Exploiting spatial
structure for localizing manipulated image regions. In Proceedings of the IEEE International Conference on Computer
Vision, pages 4970–4979, 2017. 1, 2
[12] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou,
Joshua B. Tenenbaum, William T. Freeman, and Antonio
Torralba. Visualizing and understanding generative adversarial networks. In International Conference on Learning
Representations, 2019. 1[13] Belhassen Bayar and Matthew C Stamm. A deep learning
approach to universal image manipulation detection using
a new convolutional layer. In Proceedings of the 4th ACM
Workshop on Information Hiding and Multimedia Security,
pages 5–10. ACM, 2016. 1, 2
[14] Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi
Munos. The cramer distance as a solution to biased wasser-

stein gradients. arXiv preprint arXiv:1705.10743, 2017. 2,
5[15] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial gan. In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 469–477. JMLR. org, 2017. 2[16] Paolo Bestagini, Simone Milani, Marco Tagliasacchi, andStefano Tubaro. Local tampering detection in video sequences. In 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP), pages 488–493. IEEE,
2013. 1[17] Mikoaj Bińkowski, Dougal J. Sutherland, Michael Arbel,
and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. 2,
5[18] Luca Bondi, Silvia Lameri, David Guera, Paolo Bestagini,
Edward J Delp, Stefano Tubaro, et al. Tampering detection
and localization through clustering of camera-based cnn features. In IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), pages 1855–1864, 2017.
2[19] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large
scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. 1, 2
[20] Peter Burt and Edward Adelson. The laplacian pyramid asa compact image code. IEEE Transactions on Communications, 31(4):532–540, 1983. 4[21] Mo Chen, Jessica Fridrich, Miroslav Goljan, and Jan Lukás.
Determining image origin and integrity using sensor noise.
IEEE Transactions on Information Forensics and Security,3(1):74–90, 2008. 2, 3
[22] Davide Cozzolino, Giovanni Poggi, and Luisa Verdoliva.
Recasting residual-based local descriptors as convolutional
neural networks: an application to image forgery detection. In Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security, pages 159–164. ACM,
2017. 1, 2
[23] Davide Cozzolino, Justus Thies, Andreas Rössler, Christian
Riess, Matthias Nießner, and Luisa Verdoliva. Forensictransfer: Weakly-supervised domain adaptation for forgery detection. arXiv preprint arXiv:1812.02510, 2018. 2[24] Davide Cozzolino and Luisa Verdoliva. Noiseprint: a
cnn-based camera model fingerprint.
arXiv preprint
arXiv:1808.08396, 2018. 2, 3
[25] DC Dowson and BV Landau. The fréchet distance between
multivariate normal distributions. Journal of Multivariate
Analysis, 12(3):450–455, 1982. 5[26] Luca DAmiano, Davide Cozzolino, Giovanni Poggi, andLuisa Verdoliva. A patchmatch-based dense-field algorithm
for video copy–move detection and localization. IEEE
Transactions on Circuits and Systems for Video Technology,
29(3):669–682, 2019. 1, 2
[27] Hany Farid. Photo forensics. MIT Press, 2016. 1, 2
[28] Jessica Fridrich. Digital image forensics: there is more to a
picture than meets the eye. Springer New York, 2012. 2

7564

[29] Jessica Fridrich and Jan Kodovsky. Rich models for steganalysis of digital images. IEEE Transactions on Information
Forensics and Security, 7(3):868–882, 2012. 2[30] Yoav Goldberg. A primer on neural network models for natural language processing. Journal of Artificial Intelligence
Research, 57:345–420, 2016. 2[31] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, andYoshua Bengio. Generative adversarial nets. In Advances in
Neural Information Processing Systems, pages 2672–2680,
2014. 1, 2
[32] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent
Dumoulin, and Aaron C Courville. Improved training ofwasserstein gans. In Advances in Neural Information Processing Systems, pages 5767–5777, 2017. 1, 2, 4[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 770–778, 2016. 2[34] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,
Bernhard Nessler, and Sepp Hochreiter. Gans trained by a
two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626–6637, 2017. 5[35] Minyoung Huh, Andrew Liu, Andrew Owens, and Alexei A
Efros. Fighting fake news: Image splice detection via
learned self-consistency. In Proceedings of the European
Conference on Computer Vision (ECCV), pages 101–117,
2018. 1, 2[36] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A
Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 1125–
1134, 2017. 1, 2, 4[37] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey
Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama,
and Trevor Darrell. Caffe: Convolutional architecture for fastfeature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia, pages 675–678. ACM,
2014. 2[38] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning
Representations, 2018. 1, 2, 5[39] Gerhard C Langelaar, Iwan Setyawan, and Reginald L Lagendijk. Watermarking digital image and video data. astate-of-the-art overview. IEEE Signal Processing Magazine,
17(5):20–46, 2000. 3[40] Chuan Li and Michael Wand. Precomputed real-time texture
synthesis with markovian generative adversarial networks. In
European Conference on Computer Vision, pages 702–716.
Springer, 2016. 2[41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
Deep learning face attributes in the wild. In Proceedings ofInternational Conference on Computer Vision (ICCV), 2015.
5[42] Jan Lukas, Jessica Fridrich, and Miroslav Goljan. Digital
camera identification from sensor pattern noise. IEEE Transactions on Information Forensics and Security, 1(2):205–
214, 2006. 2, 3
[43] Laurens van der Maaten and Geoffrey Hinton. Visualizing
data using t-sne. Journal of Machine Learning Research,
9(Nov):2579–2605, 2008. 1[44] Francesco Marra, Diego Gragnaniello, Davide Cozzolino,
and Luisa Verdoliva. Detection of gan-generated fake images
over social networks. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pages
384–389. IEEE, 2018. 1, 3[45] Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, andGiovanni Poggi. Do gans leave artificial fingerprints? In
2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pages 506–511. IEEE, 2019. 2, 3,
4, 5, 6, 7, 8
[46] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, andYuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning
Representations, 2018. 2, 5[47] Huaxiao Mo, Bolin Chen, and Weiqi Luo. Fake faces identification via convolutional neural network. In Proceedings of
the 6th ACM Workshop on Information Hiding and Multimedia Security, pages 43–47. ACM, 2018. 1, 3
[48] Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario
Fritz. Towards reverse-engineering black-box neural networks. In Internation Conference on Representation Learning (ICLR), 2018. 7[49] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics: A large-scale video dataset for forgery detection in human faces. arXiv preprint arXiv:1803.09179, 2018. 2[50] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. arXiv
preprint arXiv:1901.08971, 2019. 2[51] Lalit Kumar Saini and Vishal Shrivastava. A survey of digital watermarking techniques and its applications. CoRR,
abs/1407.4735, 2014. 3[52] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki
Cheung, Alec Radford, and Xi Chen. Improved techniques
for training gans. In Advances in Neural Information Processing Systems (NIPS), pages 2226–2234, 2016. 1, 2, 5, 6,
7, 8[53] Husrev Taha Sencar and Nasir Memon. Digital image forensics. Counter-Forensics: Attacking Image Forensics, pages
327–366, 2013. 1[54] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo,
and David W Jacobs. Sfsnet: Learning shape, reflectanceand illuminance of facesin the wild’. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 6296–6305, 2018. 7[55] Lawrence Sirovich and Michael Kirby. Low-dimensional
procedure for the characterization of human faces. OpticalSociety of America, 4(3):519–524, 1987. 5, 6, 77565

[56] Efstathios Stamatatos. A survey of modern authorship attribution methods. Journal of the Association for Information
Science and Technology, 60(3):538–556, 2009. 3[57] Supasorn Suwajanakorn, Steven M Seitz, and Ira
Kemelmacher-Shlizerman. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics(TOG), 36(4):95, 2017. 1[58] Mitchell D Swanson, Mei Kobayashi, and Ahmed H Tewfik. Multimedia data-embedding and watermarking technologies. Proceedings of the IEEE, 86(6):1064–1087, 1998.
3[59] Shahroz Tariq, Sangyup Lee, Hoyoung Kim, Youjin Shin,and Simon S Woo. Detecting both machine and human created fake face images in the wild. In Proceedings of the
2nd International Workshop on Multimedia Privacy and Security, pages 81–87. ACM, 2018. 1, 3[60] Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. Real-time
expression transfer for facial reenactment. ACM Transactions on Graphics (TOG), 34(6):183–1, 2015. 1, 2[61] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time
face capture and reenactment of rgb videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 2387–2395, 2016. 1, 2[62] Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, andShin’ichi Satoh. Embedding watermarks into deep neuralnetworks. In Proceedings of the 2017 ACM on International
Conference on Multimedia Retrieval, pages 269–277. ACM,
2017. 2, 3[63] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas
Funkhouser, and Jianxiong Xiao. Lsun: Construction of alarge-scale image dataset using deep learning with humans
in the loop. arXiv preprint arXiv:1506.03365, 2015. 5[64] Ning Yu, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, and Michal Lukac. Texture mixer: A network for controllable synthesis and interpolation of texture. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 12164–12173, 2019. 1[65] Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Marc Ph
Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking.
In Proceedings of the 2018 on Asia Conference on Computerand Communications Security, pages 159–172. ACM, 2018.
2, 3[66] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness ofdeep features as a perceptual metric. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, 2018. 7[67] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis.
Two-stream neural networks for tampered face detection.
In 2017 IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), pages 1831–1839. IEEE,
2017. 1, 2
[68] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis.
Learning rich features for image manipulation detection. In

Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 1053–1061, 2018. 1, 2
[69] Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel
Cohen-Or, and Hui Huang. Non-stationary texture synthesis
by adversarial expansion. ACM Transactions on Graphics(TOG), 37(4):49:1–49:13, 2018. 2[70] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A
Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. In Proceedings of the IEEE
International Conference on Computer Vision, pages 2223–
2232, 2017. 1, 2[71] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advancesin Neural Information Processing Systems, 2017. 1, 2[72] Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek
Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger,
Matthias Nießner, and Christian Theobalt. State of the art
on monocular 3d face reconstruction, tracking, and applications. In Computer Graphics Forum, volume 37, pages 523–
550. Wiley Online Library, 2018. 1

7566