Fawkes: Protecting Privacy against Unauthorized Deep Learning Models
Shawn Shan† , Emily Wenger† , Jiayun Zhang, Huiying Li, Haitao Zheng, Ben Y. Zhao
† denotes co-first authors with equal contribution
Computer Science, University of Chicago{shansixiong, ewillson, jiayunz, huiyingli, htzheng, ravenben}@cs.uchicago.edu
Opportunities for misuse of this technology are numerous
and potentially disastrous. Anywhere we go, we can be identified at any time through street cameras, video doorbells, security cameras, and personal cellphones. Stalkers can findout our identity and social media profiles with a single snapshot [47]. Stores can associate our precise in-store shopping
behavior with online ads and browsing profiles [31]. Identity
thieves can easily identify (and perhaps gain access to) our
personal accounts [13].
We believe that private citizens need tools to protect themselves from being identified by unauthorized facial recognition models. Unfortunately, previous work in this space issparse and limited in both practicality and efficacy. Somehave proposed distorting images to make them unrecognizable and thus avoiding facial recognition [27, 52, 64]. Others produce adversarial patches in the form of bright patterns
printed on sweatshirts or signs, which prevent facial recognition algorithms from even registering their wearer as a person [55, 65]. Finally, given access to an image classification
model, “clean-label poison attacks” can cause the model tomisidentify a single, preselected image [42, 71].
Instead, we propose Fawkes, a system that helps individuals to inoculate their images against unauthorized facialrecognition models at any time without significantly distorting their own photos, or wearing conspicuous patches.
Fawkes achieves this by helping users adding imperceptible
pixel-level changes (“cloaks”) to their own photos. For example, a user who wants to share content (e.g. photos) on
social media or the public web can add small, imperceptible
alterations to their photos before uploading them. If collectedby a third-party “tracker” and used to train a facial recognition model to recognize the user, these “cloaked” images
would produce functional models that consistently misidentify them.
Our distortion or “cloaking” algorithm takes the user’s
photos and computes minimal perturbations that shift themsignificantly in the feature space of a facial recognition
model (using real or synthetic images of a third party as alandmark). Any facial recognition model trained using theseimages of the user learns an altered set of “features” of whatmakes them look like them. When presented with a clean, uncloaked image of the user, e.g. photos from a camera phoneor streetlight camera, the model finds no labels associated

Abstract
Today’s proliferation of powerful facial recognition systems poses a real threat to personal privacy. As Clearview.ai
demonstrated, anyone can canvas the Internet for data andtrain highly accurate facial recognition models of individuals without their knowledge. We need tools to protect ourselves from potential misuses of unauthorized facial recognition systems. Unfortunately, no practical or effective solutions exist.
In this paper, we propose Fawkes, a system that helps
individuals inoculate their images against unauthorized facial recognition models. Fawkes achieves this by helping
users add imperceptible pixel-level changes (we call them“cloaks”) to their own photos before releasing them. Whenused to train facial recognition models, these “cloaked” images produce functional models that consistently cause normal images of the user to be misidentified. We experimentally demonstrate that Fawkes provides 95+% protection
against user recognition regardless of how trackers train theirmodels. Even when clean, uncloaked images are “leaked” tothe tracker and used for training, Fawkes can still maintainan 80+% protection success rate. We achieve 100% successin experiments against today’s state-of-the-art facial recognition services. Finally, we show that Fawkes is robust against
a variety of countermeasures that try to detect or disrupt image cloaks.

1 Introduction
Today’s proliferation of powerful facial recognition models
poses a real threat to personal privacy. Facial recognition systems are scanning millions of citizens in both the UK andChina without explicit consent [33, 41]. By next year, 100%of international travelers will be required to submit to facial recognition systems in top-20 US airports [38]. Perhaps
more importantly, anyone with moderate resources can now
canvas the Internet and build highly accurate facial recognition models of us without our knowledge or awareness, e.g.
MegaFace [21]. Kashmir Hill from the New York Times recently reported on Clearview.ai, a private company that collected more than 3 billion online photos and trained a massive model capable of recognizing millions of citizens, allwithout knowledge or consent [20].
1Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

with the user in the feature space near the image, and classifies the photo to another label (identity) nearby in the feature
space.
Our exploration of Fawkes produces several key findings:

showed that carefully computed adversarial stickers on a hat
can reduce its wearer’s likelihood of being recognized. Others propose “adversarial patches” that target “person identification” models, making it difficult for models to recognize
the wearer as a person in an image [55, 65].
All of these approaches share two limitations. First, theyrequire the user to wear fairly obvious and conspicuous accessories (hats, glasses, sweaters) that are impractical for normal use. Second, in order to evade tracking, they require fulland unrestricted access (white box access) to the precise
model tracking them. Thus they are easily broken (and user
privacy compromised) by any tracker that updates its model.
Another line of work seeks to edit facial images so thathuman-like characteristics are preserved but facial recognition model accuracy is significantly reduced. Methods used
include k-means facial averaging [35], facial inpainting [51],and GAN-based face editing [27,52,64]. Since these dramatically alter the user’s face in her photos, we consider them
impractical for protecting shared content.

• We can produce significant alterations to images’ feature
space representations using perturbations imperceptible tothe naked eye (DSSIM ≤ 0.007).
• Regardless of how the tracker trains its model (via transfer
learning or from scratch), image cloaking provides 95+%
protection against user recognition (adversarial training
techniques help ensure cloaks transfer to tracker models).
• Experiments show 100% success against state-of-the-art
facial recognition services from Microsoft (Azure Face
API), Amazon (Rekognition), and Face++. We first “share”
our own (cloaked) photos as training data to each service,
then apply the resulting models to uncloaked test imagesof the same person.
• In challenging scenarios where clean, uncloaked images
are “leaked” to the tracker and used for training, we show
how a single Sybil identity can boost privacy protection.
This results in 80+% success in avoiding identificationeven when half of the training images are uncloaked.
• Finally, we consider a tracker who is aware of our image
cloaking techniques and evaluate the efficacy of potential
countermeasures. We show that image cloaks are robust(maintain high protection rates against) to a variety ofmechanisms for both cloak disruption and cloak detection.

2 Background and Related WorkTo protect user privacy, our image cloaking techniques leverage and extend work broadly defined as poisoning attacks
in machine learning. Here, we set the context by discussing
prior efforts to help users evade facial recognition models.
We then discuss relevant data poisoning attacks, followedby related work on privacy-preserving machine learning andtechniques to train facial recognition models.
Note that to protect user privacy from unauthorized deeplearning models, we employ attacks against ML models. In
this scenario, users are the “attackers,” and third-party trackers running unauthorized tracking are the “targets.”

2.1 Protecting Privacy via Evasion Attacks
Privacy advocates have considered the problem of protecting individuals from facial recognition systems, generally
by making images difficult for a facial recognition model torecognize. Some rely on creating adversarial examples, inputs to the model designed to cause misclassification [54].
These attacks have since been proven possible “in the wild,”
Sharif et al. [44] create specially printed glasses that cause
the wearer to be misidentified. Komkov and Petiushko [24]

2.2 Protecting Privacy via Poisoning Attacks
An alternative to evading models is to disrupt their training.
This approach leverages “data poisoning attacks” against
deep learning models. These attacks affect deep learningmodels by modifying the initial data used to train them, usually by adding a set of samples S and associated labels LS .
Previous work has used data poisoning to induce unexpectedbehaviors in trained DNNs [66]. In this section, we discuss
two data poisoning attacks related to our work, and identify
their key limitations when used to protect user privacy.
Clean Label Attacks. A clean-label poisoning attack injects “correctly” labeled poison images into training data,causing a model trained on this data to misclassify a specificimage of interest [42, 71]. What distinguishes clean-label attacks from normal poisoning attacks is that all image labels
remain unchanged during the poisoning process – only thecontent of the poisoned images changes.
Our work (Fawkes) works with similar constraints. Our action to affect or disrupt a model is limited to altering a groupof images with a correct label, i.e. a user can alter her images
but cannot claim these are images of someone else.
Current clean label attacks cannot address the privacy
problem because of three factors. First, they only cause misclassification on a single, preselected image, whereas user
privacy protection requires the misclassification of any current or future image of the protected user (i.e. an entire model
class). Second, clean label attacks do not transfer well to different models, especially models trained from scratch. Even
between models trained on the same data, the attack onlytransfers with 30% success rate [71]. Third, clean label attacks are easily detectable through anomaly detection in thefeature space [19].
Model Corruption Attacks. Other recent work proposes2

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

User
Original

FawkesTracker / Model Trainer
CloakedTraining Data(cloaked)
Web Crawl

Images from Target T

Testing Data
(uncloaked)

Model
Training

Wrong
Label

Feature Extractor Φ:

Figure 1: Our proposed Fawkes system that protects user privacy by cloaking their online photos. (Left) A user U applies
cloaking algorithm (given a feature extractor Φ and images from some target T ) to generate cloaked versions of U’s photos,
each with a small perturbation unnoticeable to the human eye. (Right) A tracker crawls the cloaked images from online sources,
and uses them to train an (unauthorized) model to recognize and track U. When it comes to classifying new (uncloaked) imagesof U, the tracker’s model misclassifies them to someone not U. Note that T does not have to exist in the tracker’s model.
techniques to modify images such that they degrade the accuracy of a model trained on them [45]. The goal is to spreadthese poisoned images in order to discourage unauthorized
data collection and model training. We note that Fawkes’
goals are to mislead rather than frustrate. Simply corrupting
data of a user’s class may inadvertently inform the tracker ofthe user’s evasion attempts and lead to more advanced countermeasures by the tracker. Finally, [45] only has a 50% success rate in protecting a user from being recognized.

ample, a facial recognition model trained on faces extracted
from YouTube videos might serve well as a feature extractor
for a model designed to recognize celebrities in magazines.
Finally, the concept of protecting individual privacy
against invasive technologies extends beyond the image domain. Recent work [12] proposes wearable devices that restore personal agency using digital jammers to prevent audio
eavesdropping by ubiquitous digital home assistants.

3 Protecting Privacy via Cloaking
2.3 Other Related Work
Privacy-Preserving Machine Learning. Recent work has
shown that ML models can memorize (and subsequentlyleak) parts of their training data [48]. This can be exploited
to expose private details about members of the training
dataset [17]. These attacks have spurred a push towards differentially private model training [6], which uses techniquesfrom the field of differential privacy [15] to protect sensitive characteristics of training data. We note these techniques
imply a trusted model trainer and are ineffective against an
unauthorized model trainer.
Feature Extractors & Transfer Learning. Transfer learning uses existing pretrained models as a basis for quickly
training models for customized classification tasks, using
less training data. Today, it is commonly used to deploy complex ML models (e.g. facial recognition or image segmentation [70]) at reasonable training costs.
In transfer learning, the knowledge of a pre-trained feature extractor Φ is passed on to a new model Fθ . Typically,a model Fθ can be created by appending a few additional
layers to Φ and only training those new layers. The original layers that composed Φ will remain unmodified. As such,
pre-existing knowledge “learned” by Φ is passed on to themodel Fθ and directly influences its classification outcomes.
Finally, transfer learning is most effective when the feature
extractor and model are trained on similar datasets. For ex-

We propose Fawkes, a system designed to help protect the privacy of a user against unauthorized facial recognition models
trained by a third-party tracker on the user’s images. Fawkes
achieves this by adding subtle perturbations (“cloaks”) to theuser’s images before sharing them. Facial recognition models trained on cloaked images will have a distorted view of
the user in the “feature space,” i.e. the model’s internal understanding of what makes the user unique. Thus the models
cannot recognize real (uncloaked) images of the user, and instead, misclassify them as someone else.
In this section, we first describe the threat model and assumptions for both users and trackers. We then present theintuition behind cloaking and our methodology to generate
cloaks. Finally, we discuss why cloaking by individuals is
effective against unauthorized facial recognition models.

3.1 Assumptions and Threat Model
User. The user’s goal is to share their photos online without
unknowingly helping third party trackers build facial recognition models that can recognize them. Users protect themselves by adding imperceptible perturbations (“cloaks”) to
their photos before sharing them. This is illustrated in theleft part of Figure 1, where a cloak is added to this user’s
photos before they are uploaded.
The design goals for these cloaks are:3

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

• cloaks should be imperceptible and not impact normal
use of the image;• when classifying normal, uncloaked images, models
trained on cloaked images should recognize the underlying person with low accuracy.

gram. Users can also leverage privacy laws such as “Right
to be Forgotten” to remove and untag online content related
to themselves. The online curation of personal images is achallenging problem, and we leave the study of minimizing
online image footprints to future work.

We assume the user has access to moderate computing resources (e.g., a personal laptop) and applies cloaking to theirown images locally. We also assume the user has accessto some feature extractor, e.g. a generic facial recognition
model, represented as Φ in Figure 1. Cloaking is simplified
if the user has the same Φ as the tracker. We begin with thiscommon assumption (also used by prior work [42, 59, 71]),
since only a few large-scale face recognition models areavailable in the wild. Later in §3.4, we relax this assumption
and show how our design maintains the above properties.
We initially consider the case where the user has the ability to apply cloaking to all their photos to be shared, thus the
tracker can only collect cloaked photos of the user. Later in§7, we explore a scenario where a stronger tracker has obtained access to some number of their uncloaked images.
Tracker/Model Trainer. We assume that the tracker (theentity training unauthorized models) is a third party without
direct access to user’s personal photos (i.e. not Facebook or
Flickr). The tracker could be a company like Clearview.ai, agovernment entity, or even an individual. The tracker has significant computational resources. They can either use transfer learning to simplify their model training process (leveraging existing feature extractors), or train their model completely from scratch.
We also assume the tracker’s primary goal is to build apowerful model to track many users rather than targeting asingle specific person1. The tracker’s primary data source isa collection of public images of users obtained via web scraping. We also consider scenarios where they are able to obtainsome number of uncloaked images from other sources (§7).
Real World Limitations. Privacy benefits of Fawkes relyon users applying our cloaking technique to the majority ofimages of their likeness before posting online. In practice,
however, users are unlikely to control all images of themselves, such as photos shared online by friends and family,
media, employer or government websites. While it is unclear
how easy or challenging it will be for trackers to associatethese images with the identity of the user, a tracker who obtains a large number of uncloaked images of the user cancompromise the effectiveness of Fawkes.
Therefore, Fawkes is most effective when used in conjunction with other privacy-enhancing steps that minimize the online availability of a user’s uncloaked images. For example,
users can curate their social media presence and remove tagsof their names applied to group photos on Facebook or Insta1 Tracking a specific person can be easily accomplished through easier,offline methods, e.g. a private investigator who follows the target user, andis beyond the scope of our work.

3.2 Overview and Intuition
DNN models are trained to identify and extract (often hidden) features in input data and use them to perform classification. Yet their ability to identify features is easily disrupted
by data poisoning attacks during model training, where small
perturbations on training data with a particular label (l) can
shift the model’s view of what features uniquely identifyl [42, 71]. Our work leverages this property to cause misclassification of any existing or future image of a single class,providing one solution to the challenging problem of protecting personal privacy against the unchecked spread of facialrecognition models.
Intuitively, our goal is to protect a user’s privacy by modifying their photos in small and imperceptible ways before
posting them online, such that a facial recognition modeltrained on them learns the wrong features about what makes
the user look like the user. The model thinks it is successful,
because it correctly recognizes its sample of (modified) images of the user. However, when unaltered images of the user,e.g. from a surveillance video, are fed into the model, themodel does not detect the features it associates with the user.
Instead, it identifies someone else as the person in the video.
By simply modifying their online photos, the user successfully prevents unauthorized trackers and their DNN models
from recognizing their true face.

3.3 Computing Cloak Perturbations
But how do we determine what perturbations (we call them“cloaks”) to apply to Alice’s photos? An effective cloak
would teach a face recognition model to associate Alice witherroneous features that are quite different from real features
defining Alice. Intuitively, the more dissimilar or distinct
these erroneous features are from the real Alice, the lesslikely the model will be able to recognize the real Alice.
In the following, we describe our methodology for computing cloaks for each specific user, with the goal of making
the features learned from cloaked photos highly dissimilarfrom those learned from original (uncloaked) photos.
Notation. Our discussion will use the following notations.
• x: Alice’s image (uncloaked)• xT : target image (image from another class/user T ) used
to generate cloak for Alice• δ(x, xT ): cloak computed for Alice’s image x based on animage xT from label T• x ⊕ δ(x, xT ): cloaked version of Alice’s image x
4

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

• Φ: Feature extractor used by facial recognition model• Φ(x): Feature vector (or feature representation) extracted
from an input x
Cloaking to Maximize Feature Deviation. Given eachphoto (x) of Alice to be shared online, our ideal cloaking design modifies x by adding a cloak perturbation δ(x, xT ) to xthat maximize changes in x’s feature representation:maxδ Dist (Φ(x), Φ(x ⊕ δ(x, xT ))) ,(1)

subject to |δ(x, xT )| < ρ,
where Dist(.) computes the distance of two feature vectors,|δ| measures the perceptual perturbation caused by cloaking,and ρ is the perceptual perturbation budget.
To guide the search for the cloak perturbation in eq (1), we
use another image xT from a different user class (T ). Since
the feature space Φ is highly complex, xT serves as a landmark, enabling fast and efficient search for the input perturbation that leads to large changes in feature representation.
Ideally, T should be very dissimilar from Alice in the feature
space. We illustrate this in Figure 1, where we use Patrick
Dempsey (a male actress) as a dissimilar target T for the original user (female actor Gwyneth Paltrow).
We note that our design does not assume that the cloak target (T ) and the associated xT are used by any tracker’s facerecognition model. In fact, any user whose feature representation is sufficiently different from Alice’s would suffice (see§3.4). Alice can easily check for such dissimilarity by running the feature extractor Φ on other users’ online photos.
Later in §4 we will present the detailed algorithm for choosing the target user T from public datasets of facial images.
Image-specific Cloaking. When creating cloaks for her
photos, Alice will produce image-specific cloaks, i.e. δ(x, xT )is image dependent. Specifically, Alice will pair each original
image x with a target image xT of class T . In our current implementation, the search for δ(x, xT ) replaces the ideal optimization defined by eq. (1) with the following optimization:minδ Dist (Φ(xT ), Φ(x ⊕ δ(x, xT ))) ,(2)

subject to |δ(x, xT )| < ρ.
Here we search for the cloak for x that shifts its feature
representation closely towards xT . This new form of optimization also prevents the system from generating extreme
Φ(x ⊕ δ(x, xT )) values that can be easily detected by trackers
using anomaly detection.
Finally, our image-specific cloak optimization will create
different cloak patterns among Alice’s images. This “diversity” makes it hard for trackers to detect and remove cloaks.

3.4 Cloaking Effectiveness & Transferability
Now a user (Alice) can produce cloaked images whose feature representation is dissimilar from her own but similar toWithout Cloak
x2

T

x2 T

A

B

(a)

A
B

U

Decision Boundary

With Cloak
O

x1

x1

(b)

Figure 2: The intuition for why a tracker’s model trained on
U’s cloaked photos will misclassify U’s original photos, visualized on a simplified 2D feature space with four user classes
A, B, U (aka Alice), T . (a) decision boundaries of the modeltrained on U’s uncloaked photos. (b) decision boundaries
when trained on U’s cloaked photos (with target T ).

that of a target user T . But does this translate into the desired
misclassification behavior in the tracker model? Clearly, if T
is a class in the tracker model, Alice’s original (uncloaked)
images will not be classified as Alice. But under the more
likely scenario where T is not in the tracker model, does
cloaking still lead to misclassification?
We believe the answer is yes. Our hypothesis is that aslong as the feature representations of Alice’s cloaked and uncloaked images are sufficiently different, the tracker’s model
will not classify them as the same class. This is because there
will be another user class (e.g. B) in the tracker model, whosefeature representation is more similar to Φ(x) (true Alice)than Φ(x ⊕ δ) (Alice learned by the model). Thus, the model
will classify Alice’s normal images as B.
We illustrate this in Figure 2 using a simplified 2D visualization of the feature space. There are 4 classes (A, B, U aka
Alice, and T ) that a tracker wishes to distinguish. The two figures show the tracker model’s decision boundary when U’s
training data is uncloaked and cloaked, respectively. In Figure 2(a), the model will learn U’s true feature representation
as the bottom right corner. In Figure 2(b), U uses T as the
cloak target, and the resulting tracker model will learn U’s
feature representation Φ(x ⊕ δ) as green triangles near T (topleft corner). This means that the area corresponding to U’s
original feature representation Φ(x) will be classified as B.
More importantly, this (mis)classification will occur whether
or not T is a class in the tracker’s model.
Our above discussion assumes the tracker’s model contains a class whose feature representation is more similar to
the user’s original feature representation than her cloaked feature representation. This is a reasonable assumption when
the tracker’s model targets many users (e.g. 1,000) rather
than a few users (e.g. 2). Later in §5 we confirm that cloaking
is highly effective against multiple facial recognition models
with anywhere from 65 to 10,575 classes.
Transferability. Our above discussion also assumes thatthe user has the same feature extractor Φ as is used to train
the tracker model. Under the more general scenario, the ef5

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

fectiveness of cloaking against any tracker models relies on
the transferability effect, the property that models trained
for similar tasks share similar properties and vulnerabilities,
even when they were trained on different architectures anddifferent training data [14, 39, 50, 70].
This transferability property suggests that cloaking shouldstill be effective even if the tracker performs transfer learning using a different feature extractor or trains their modelfrom scratch. Because the user’s and tracker’s feature extractors/models are designed for similar tasks (i.e. facial recognition), cloaks should be effective regardless of the tracker’s
training method. Later, we empirically evaluate cloaking success rate when trackers use different feature extractors (§5.3)
or train models from scratch (§5.4). In all scenarios, cloaking
is highly effective (> 95% protection rate).

the optimization in eq.(2) as follows:

4 The Fawkes Image Cloaking System
We now present the detailed design of Fawkes, a practicalimage cloaking system that allows users to evade identification by unauthorized facial recognition models. Fawkes uses
three steps to help a user modify and publish her online photos.
Given a user U, Fawkes takes as input the set of U’s photos
to be shared online XU , the (generic) feature extractor Φ, andthe cloak perturbation budget ρ.
Step 1: Choosing a Target Class T . First, Fawkes examines a publicly available dataset that contains numerous
groups of images, each identified with a specific class label,e.g. Bob, Carl, Diana. Fawkes randomly picks K candidate
target classes and their images from this public dataset anduses the feature extractor Φ to calculate Ck , the centroid of
the feature space for each class k = 1..K. Fawkes picks as thetarget class T the class in the K candidate set whose feature
representation centroid is most dissimilar from the feature
representations of all images in XU , i.e.
T = argmax min Dist(Φ(x), Ck ).
k=1..K x∈XU

(3)

We use L2 as the distance function in feature space, Dist(.).
Step 2: Computing Per-image Cloaks. Let XT represent
the set of target images available to user U. For each imageof user U, x ∈ XU , Fawkes randomly picks an image xT ∈ XT ,and computes a cloak δ(x, xT ) for x, following the optimization defined by eq. (2), subject to |δ(x, xT )| < ρ.
In our implementation, |δ(x, xT )| is calculated using the
DSSIM (Structural Dis-Similarity Index) [61, 62]. Different from the L p distance used in previous work [9, 25, 43],DSSIM has gained popularity as a measure of user-perceived
image distortion [23,28,59]. Bounding cloak generation withthis metric ensures that cloaked versions of images are visually similar to the originals.
We apply the penalty method [37] to reformat and solve

min Dist (Φ(xT ), Φ(x ⊕ δ(x, xT ))) + λ · max(|δ(x, xT )| − ρ, 0)δ

Here λ controls the impact of the input perturbation causedby cloaking. When λ → ∞, the cloaked image is visually identical to the original image. Finally, to ensure the input pixel
intensity remains in the correct range ([0, 255]), we transformthe intensity values into tanh space as proposed in previous
work [10].
Step 3: Limiting Content. Now the user U has createdthe set of cloaked images that she can post and share online. However, the user must be careful to ensure that no
uncloaked images are shared online and associated with her
identity. Any images shared by friends and labeled or taggedwith her name would provide uncloaked training data for a
tracker model. Fortunately, a user can proactively “untag”
herself on most photo sharing sites.
Even so, a third party might be able to restore those labels and re-identify her in those photos using friendlist intersection attacks [63]. Thus, in §7, we expand the design of
Fawkes to address trackers who are able to obtain uncloaked
images in addition to cloaked images of the user.

5 System Evaluation
In this section, we evaluate the effectiveness of Fawkes. We
first describe the datasets, models, and experimental configurations used in our tests. We then present results for cloaking in three different scenarios: 1) the user produces cloaks
using the same feature extractor as the tracker; 2) the user
and tracker use different feature extractors; and 3) the tracker
trains models from scratch (no feature extractor).
Our key findings are: cloaking is highly effective whenusers share a feature extractor with the tracker; efficacy could
drop when feature extractors are different, but can be restored
to near perfection by making the user’s feature extractor robust (via adversarial training); and, similarly, cloaks generated on robust feature extractors work well even when trackers train models from scratch.

5.1 Experiment Setup
Our experiments require two components. First, we need feature extractors that form the basis of facial recognition models for both the user’s cloaking purposes and the tracker’s
model training. Second, we need datasets that emulate a setof user images scraped by the tracker and enable us to evaluate the impact of cloaking.
Feature Extractors. There are few publically available,
large-scale facial recognition models. Thus we train feature
extractors using two large (≥ 500K images) datasets on different model architectures (details in Table 2).
6

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

Teacher Dataset

Model Architecture

Abbreviation

WebFace
WebFace
VGGFace2
VGGFace2

InceptionResNet
DenseNet
InceptionResNet
DenseNet

Web-Incept
Web-Dense
VGG2-Incept
VGG2-Dense

Teacher Testing
Accuracy
74%
76%
81%
82%

Student Testing Accuracy
PubFigFaceScrub
96%
92%
96%
94%
95%
90%
96%
92%

Table 1: The four feature extractors used in our evaluation, their classification efficacy and those of their student models.
DatasetPubFigFaceScrub
WebFace
VGGFace2# of Labels
65
344
10, 575
8, 631

Input Size
224 × 224 × 3
224 × 224 × 3
224 × 224 × 3
224 × 224 × 3# of Training Images
5, 85037, 905
475, 137
3, 141, 890

Table 2: Datasets emulating user images in experiments.
• VGGFace2 contains 3.14M images of 8, 631 subjects downloaded from Google Image Search [7].
• WebFace has 500, 000 images of faces covering roughly
10, 000 subjects collected from the Internet [69].
Using these two datasets, we build four feature extractors, two from each. We use two different model architectures: a) DenseNet-121 [22], a 121 layer neural networkwith 7M parameters, and b) InceptionResNet V2 [53], a572 layer deep neural network with over 54M parameters.
Our trained models have comparable accuracy with previous
work [7, 34, 59] and perform well in transfer learning scenarios. For clarity, we abbreviate feature extractors based on
their dataset/architecture pair. Table 1 lists the classification
accuracy for our feature extractors and student models.
Tracker’s Training Datasets. Under the scenario where
the tracker trains its facial recognition model from scratch(§5.4), we assume they will use the above two large datasets(VGGFace2, WebFace). Under the scenario where they apply
transfer learning (§5.2 and §5.3), the tracker uses the following two smaller datasets (more details in Table 2).
• PubFig contains 5, 850 training images and 650 testing
images of 65 public figures2 [5].
• FaceScrub contains 100, 000 images of 530 public figureson the Internet [36]3 .
To perform transfer learning, the tracker adds a softmax layer
at the end of the feature extractor (see §2.3), and fine-tunes
the added layer using the above dataset.
Cloaking Configuration.
In our experiments, we randomly choose a user class U in the tracker’s model, e.g. a
random user in PubFig, to be the user seeking protection.
We then apply the target selection algorithm described in§4 to select a target class T from a small subset of users in
2 We exclude 18 celebrities also used in the feature extractor datasets.

3 We could only download 60,882 images for 530 people, as some URLs
were removed. Similarly, prior work [68] only retrieved 48,579 images.

VGGFace2 and WebFace. Here we ensure that T is not a userclass in the tracker’s model.
For each given U and T pair, we pair each image x of U
with an image xT from T , and compute the cloak for x. For
this we run the Adam optimizer for 1000 iterations with alearning rate of 0.5.
As discussed earlier, we evaluate our cloaking under three
scenarios, U and tracker model sharing the same feature
extractor (§5.2), the two using different feature extractors(§5.3), and the tracker training model from scratch without
using any pre-defined feature extractor (§5.4).
Evaluation Metrics. In each scenario, we evaluate cloak
performance using two metrics: protection success rate,
which is the tracker model’s misclassification rate for clean(uncloaked) images of U, and normal accuracy, which is
the overall classification accuracy of the tracker’s model onusers beside U. When needed, we indicate the configuration of user/tracker feature extractors using the notation <entity>:<feature extractor>.

5.2 User/Tracker Sharing a Feature Extractor
We start from the simple case where the user uses the samefeature extractor as the tracker to generate cloaks. We randomly select a label from PubFig or FaceScrub to be theFawkes user U. We then compute “cloaks” for a subset ofU’s images, using each of the four feature extractors in Table 1. On the tracker side, we perform transfer learning on thesame feature extractor (with cloaked images of U) to build amodel that recognizes U. Finally, we evaluate whether the
tracker model can correctly identify other clean images of U
it has not seen before.
Results show that cloaking offers perfect protection, i.e. U
is always misclassified as someone else, for all four feature
extractors and under the perturbation budget ρ = 0.007. To
explore the impact of ρ, Figure 4 plots protection successrate vs. ρ when the tracker runs on the FaceScrub dataset.
Fawkes achieves 100% protection success rate when ρ >0.005. Figure 5 shows original and cloaked images, demonstrating that cloaking does not visually distort the original
image. Even when ρ = 0.007, the perturbation is barely detectable by the naked eye on a full size, color image. For calibration, note that prior work [28] claims much higher DSSIM
values (up to 0.2) are imperceptible to the human eye. Finally,
the average L2 norm of our cloaks is 5.44, which is smaller
7

Original Images
Other Images
Target Images

-0.4

-0.2

0

0.2

Dimension 1(a) Before Cloaking

0.4

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao
0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3

Protection Success Rate

0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3

Dimension 2

Dimension 2

Proc. of USENIX Security Symposium 2020

Cloaked Images
Other Images
Target Images

-0.4

-0.2

0

0.2

1
0.8
0.6
0.4
0.2
0
0

0.4

0.002

0.004

0.006

0.008

0.01

DSSIM Perturbation Budget

Dimension 1(b) After Cloaking

Figure 3: 2-D PCA visualization of VGG2-Dense feature space representations ofuser images (sampled from FaceScrub) before/after cloaking. Triangles are user’s
images, red crosses are target images, grey dots are images from another class.
than that of perturbations used in prior works [29, 59].
Feature Space Deviation. The goal of a cloak is to change
the image’s feature space representation in the tracker’s
model. To examine the effect of the cloak in the tracker
model, we visualize feature space representations of user images before and after cloaking, their chosen target images,
and a randomly chosen class from the tracker’s dataset. We
use principal components analysis (PCA, a common dimensionality reduction technique) to reduce the high dimensionalfeature space to 2 dimensions. Figure 3 shows the PCA results for cloaked images from a PubFig class, using cloaks
constructed on the Web-Incept feature extractor. Figure 3(a)shows the feature space positions of the original and target
images before cloaking, along with a randomly selected class.
Figure 3(b) shows the updated feature space after the original
images have been cloaked. It is clear that feature space representations of the cloaked images are well-aligned with thoseof the target images, validating our intuition for cloaking (anabstract view in Figure 2).
Impact of Label Density. As discussed in §3, the numberof labels present in the tracker’s model impacts performance.
When the tracker targets fewer labels, the feature space is“sparser,” and there is a greater chance the model continues toassociate the original feature space (along with the cloakedfeature space) with the user’s label. We empirically evaluate the impact of fewer labels on cloaking success using the
PubFig and FaceScrub datasets (65 and 530 labels, respectively). We randomly sample N labels (varying N from 2 to10) to construct a model with fewer labels. Figure 6 showsthat for PubFig, cloaking success rate grows from 68% for
2 labels to > 99% for more than 6 labels, confirming that ahigher label density improves cloaking effectiveness.

5.3 User/Tracker Using Different Feature Extractors
We now consider the scenario when the user and tracker
use different feature extractors to perform their tasks. While
the model transferability property suggests that there are significant similarities in their respective model feature spaces

Original

CloakedFigure 4: Protection performance as
DSSIM perturbation budget increases.
(User/Tracker: Web-Incept)
Original

CloakedOriginal

CloakedFigure 5: Pairs of original and cloaked images (ρ = 0.007).
(since both are trained to recognize faces), their differences
could still reduce the efficacy of cloaking. Cloaks that shift
image features significantly in one feature extractor may produce a much smaller shift in a different feature extractor.
To illustrate this, we empirically inspect the change in feature representation between two different feature extractors.
We take the cloaked images (optimized using VGG2-Dense),
original images, and target images from the PubFig dataset
and calculate their feature representations in a different feature extractor, Web-Incept. The result is visualized using
two dimensional PCA and shown in Figure 7. From the PCA
visualization, the reduction in cloak effectiveness is obvious.
In the tracker’s feature extractor, the cloak “moves” the original image features only slightly towards the target image features (compared to Figure 3(b)).
Robust Feature Extractors Boost Transferability. To address the problem of cloak transferability, we draw on recent
work linking model robustness and transferability. Demontis
et al. [14] argue that an input perturbation’s (in our case,cloak’s) ability to transfer between models depends on the“robustness” of the feature extractor used to create it. Theyshow that more “robust” models are less reactive to small
perturbations on inputs. Furthermore, they claim that perturbations (or, again, cloaks) generated on more robust models
will take on “universal” characteristics that are able to effectively fool other models.
Following this intuition, we propose to improve cloak
8

Proc. of USENIX Security Symposium 2020
User’s Robust
FeatureExtractor
VGG2-Incept
VGG2-Dense
Web-Incept
Web-Dense

VGG2-Incept
PubFig FaceScrub
100%
100%
100%
100%
100%
100%
100%
100%

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao
Model Trainer’s Feature Extractor
VGG2-Dense
Web-Incept
PubFig FaceScrub PubFig FaceScrub
100%
100%
95%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
97%

Web-Dense
PubFig FaceScrub
100%
100%
100%
100%
99%
99%
100%
96%

Table 3: Protection performance of cloaks generated on robust feature extractors.
transferability by increasing the user feature extractor’s robustness. This is done by applying adversarial training [18,30], which trains the model on perturbed data to make itless sensitive to similar small perturbations on inputs. Specifically, for each feature extractor, we generate adversarial examples using the PGD attack [25], a widely used method
for adversarial training. Following prior work [30], we run
the PGD4 algorithm for 100 steps using a step size of 0.01.
We train each feature extractor for an additional 10 epochs.
These updated feature extractors are then used to generateuser cloaks on the PubFig and FaceScrub datasets.
Results in Table 3 show that each robust feature extractor
produces cloaks that transfer almost perfectly to the tracker’s
models. Cloaks now have protection success rates > 95%
when the tracker uses a different feature extractor. We visualize their feature representation using PCA in Figure 8 andsee that, indeed, cloaks generated on robust extractors transfer better than cloaks computed on normal ones.

5.4 Tracker Models Trained from Scratch
Finally, we consider the scenario in which a powerful tracker
trains their model from scratch. We select the user U to bea label inside the WebFace dataset. We generate cloaks onuser images using the robust VGG2-Incept feature extractor
from §5.3. The tracker then uses the WebFace dataset (but
U’s cloaked images) to train their model from scratch. Again
our cloas achieve a success rate of 100%. Other combinations of labels and user-side feature generators all have 100%
protection success.

6 Image Cloaking in the Wild
Our results thus far have focused on limited configurations,
including publicly available datasets and known model architectures. Now, we wish to understand the performance of
Fawkes on deployed facial recognition systems in the wild.
We evaluate the real-world effectiveness of image cloaking by applying Fawkes to photos of one of the co-authors.
We then intentionally leak a portion of these cloaked photos
to public cloud-based services that perform facial recognition, including Microsoft Azure Face [3], Amazon Rekogni4 We found that robust models trained on CW attack samples [10] produce similar results

Face
RecognitionAPI
Microsoft Azure
Face API
Amazon Rekognition
Face Verification
Face++
Face Search API

Protection Success Rate
Without
Protected by Protected byprotection normal cloak robust cloak
0%

100%

100%

0%

34%

100%

0%

0%

100%

Table 4: Cloaking is highly effective against cloud-based facerecognition APIs (Microsoft, Amazon and Face++).
tion [2], and Face++ [4]. These are the global leaders in facialrecognition and their services are used by businesses, police,private entities, and governments in the US and Asia.

6.1 Experimental Setup
We manually collected 82 high-quality pictures of a coauthor that feature a wide range of lighting conditions, poses,
and facial expressions. We separate the images into two
subsets, one set of 50 images for “training” and one set of
32 images for “testing.” We generate both normal and robust cloaks for the “training” images using the setup discussed in Section 5 (using normal and robust versions of the
Web-Incept feature extractor). This allows us to compare
the relative effectiveness of normal and robust user feature
extractors in real life.
For each API service, we experiment with three scenarios:• Unprotected: We upload original training images, and test
the model’s classification accuracy on testing images.
• Normal Cloak: We upload training images protected bya nonrobust cloak and then test the model’s classification
accuracy on the testing images.
• Robust Cloak: We upload training images protected bya robust cloak and test the model’s classification accuracy
on the testing images.
For each scenario, we use the online service APIs to upload training images to the API database, and then query theAPIs using the uncloaked testing images. The reported protection success rate is the proportion of uncloaked test images that the API fails to correctly identify as our co-author.
9

0.8

0.6

0.6

0.4

0.4

0.2

0.2

Protection Success Rate
Normal Classification Accuracy

0
2

4

6

8

0
10

0.6
0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3

Original Images
Other Images
Target Images
Cloaked Images

-0.4

-0.2

Number of Labels in Student Dataset

Figure 6: Protection performance improves as the number of labels intracker’s model increases. (User/Tracker:
Web-Incept)

Dimension 2

1

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

Dimension 21
0.8Classification Accuracy

Protection Success Rate

Proc. of USENIX Security Symposium 2020

0

0.2

0.4

0.6

Dimension 1

Figure 7: Cloaking is less effective whenusers and trackers use different feature
extractors. (User: VGG2-Dense, Tracker:
Web-Incept)

6.2 Real World Protection Performance
Microsoft Azure Face API. Microsoft Azure Face API [3]is part of Microsoft Cognitive Services, and is reportedlyused by many large corporations including Uber and Jet.com.
The API provides face recognition services. A client uploads
training images of faces, and Microsoft trains a model to recognize these faces. The API has a “training” endpoint thatmust be called before the model will recognize faces, whichleads us to believe that Microsoft uses transfer learning totrain a model on user-submitted images.
Our normal cloaking method is 100% effective against theMicrosoft Azure Face API. Our robust cloaks also provide
100% protection against the Azure Face API. Detailed protection results are shown in Table 4.
Amazon Rekognition Face Verification. Amazon Rekognition [2] provides facial search services that the client canuse to detect, analyze, and compare faces. The API is usedby various large corporations including the NFL, CBS, andNational Geographic, as well as law enforcement agencies in
Florida and Oregon, and the U.S. Immigration and Customs
Enforcement agency (ICE).
It is important to note that Amazon Rekognition does not
specifically train a neural network to classify queried images.
Instead, it computes an image similarity score between thequeried image and the ground truth images for all labels. If
the similarity score exceeds a threshold for some label, Amazon returns a match. Our cloaking technique is not designed
to fool a tracker who uses similarity matching. However,
we believe our cloaking technique should still be effective
against Amazon Rekognition, since cloaks create a feature
space separation between original and cloaked images thatshould result in low similarity scores between them.
Table 4 shows that our normal cloaks only achieve a protection success rate of 34%. However, our robust cloaks
again achieve a 100% protection success rate.
Face++. Face++ [4] is a well-known face recognition system developed in China that claims to be extremely robust
against a variety of attacks (i.e. adversarial masks, makeup,etc.). Due to its high performance and perceived robust-

0.6
0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3

Original Images
Other Images
Target Images
Cloaked Images

-0.4

-0.2

0

0.2

0.4

0.6

Dimension 1

Figure 8: Cloaks generated on robustmodels transfer better between feature
extractors. (User: VGG2-Dense, Tracker:
Web-Incept)

ness, Face++ is widely used by financial services providers
and other security-sensitive customers. Notably, Alipay uses
Face++’s services to authenticate users before processing
payments. Lenovo also uses Face++ services to perform facebased authentication for laptop users.
Our results show that normal cloaking is completely ineffective against Face++ (0% protection success rate; see Table 4). This indicates that their model is indeed extremely
robust against input perturbations. However, as before, ourrobust cloaks achieve a 100% success rate.
Summary. Microsoft Azure Face API, Amazon Rekognition and Face++ represent three of the most popular andwidely deployed facial recognition services today. The success of Fawkes cloaking techniques suggests our approach isrealistic and practical against production systems. While we
expect these systems to continue improving, we expect cloaking techniques to similarly evolve over time to keep pace.

7 Trackers with Uncloaked Image AccessThus far we have assumed that the tracker only has access to
cloaked images of a user, i.e. the user is perfect in applying
her cloaking protection to her image content, and disassociating her identity from images posted online by friends. In real
life, however, this may be too strong an assumption. Users
make mistakes, and unauthorized labeled images of the user
can be taken and published online by third parties such as
newspapers and websites.
In this section, we consider the possibility of the tracker
obtaining leaked, uncloaked images of a target user, e.g. Alice. We first evaluate the impact of adding these images tothe tracker’s model training data. We then consider possible
mechanisms to mitigate this impact by leveraging the use oflimited sybil identities online.

7.1 Impact of Uncloaked Images
Intuitively, a tracker with access to some labeled, uncloaked
images of a user has a much greater chance of training a10

Proc. of USENIX Security Symposium 2020
model M that successfully recognizes clean images of thatuser. Training a model with both cloaked and uncloaked user
images means the model will observe a much larger spreadof features all designated as the user. Depending on how M
is trained and the presence/density of other labels, it can a)classify both regions of features as the user; b) classify bothregions and the region between them as the user; or c) ignore
these feature dimensions and identify the user using some alternative features (e.g. other facial features) that connect both
uncloaked and cloaked versions of the user’s images.
We assume the tracker cannot visually distinguish between
cloaked and uncloaked images and trains their model on both.
We quantify the impact of training with uncloaked images
using a simple test with cloaks generated from §5.2 and amodel trained on both cloaked and uncloaked images. Figure 10 shows the drop in protection success for FaceScrub
dataset as the ratio of uncloaked images in the training
dataset increases. The protection success rate drops below
39% when more than 15% of the user’s images are uncloaked.
Next, we consider proactive mitigation strategies against
leaked images. The most direct solution is to intentionallyrelease more cloaked images, effectively flooding a potential
tracker’s training set with cloaked images to dominate any
leaked uncloaked images. In addition, we consider the use ofa cooperating secondary identity (more details below). For
simplicity, we assume that: trackers have access to a small
number of a user’s uncloaked images; the user is unaware of
the contents of the uncloaked images obtained by the tracker;
and users know the feature extractor used by the tracker.

7.2 Sybil Accounts
In addition to proactive flooding of cloaked images, we explore the use of cooperative Sybil accounts to induce model
misclassification. A Sybil account is a separate account controlled by the user that exists in the same Internet community (i.e. Facebook, Flickr) as the original account. Sybils
already exist in numerous online communities [67], and areoften used by real users to curate and compartmentalize content for different audiences [26]. While there are numerous
detection techniques for Sybil detection, individual Sybil accounts are difficult to identify or remove [60].
In our case, we propose that privacy-conscious users create a secondary identity, preferably not connected to theirmain identity in the metadata or access patterns. Its content can be extracted from public sources, from a friend,or even generated artificially via generative adversarial networks (GANs) [32]. Fawkes modifies Sybil images (in a manner similar to cloaking) to provide additional protection for
the user’s original images. Since Sybil and user images reside in the same communities, we expect trackers will collect
both. While there are powerful re-identification techniquesthat could be used to associate the Sybil back to the original

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao
Without Sybil
x2

A

Leaked image of U

With Sybil

Test image of U

A

Cloaked image of U
Sybil image

S

(a)

x1

Decision Boundary

(b)

x1

Figure 9: Intuition behind Sybil integration visualized in a
2D feature space. Without Sybils, a tracker’s model will use
leaked training images of U to learn U’s true feature space(left), leading to the correct classification of images of U.
Sybil images S complicate the model’s decision boundaryand cause misclassification of U’s images, even when leaked
images of U are present (right).
user, we assume they are impractical for the tracker to apply
at scale to its population of tracked users.
Sybil Intuition.
To bolster cloaking effectiveness, theuser modifies Sybil images so they occupy the same feature space as a user’s uncloaked images. These Sybil images
help confuse a model trained on both Sybil images and uncloaked/cloaked images of a user, increasing the protection
success rate. Figure 9 shows the high level intuition. Without
Sybil images, models trained on a small portion of uncloaked(leaked) images would easily associate test images of the user
with the user’s true label (shown on left). Because the leaked
uncloaked images and Sybil images are close by in their feature space representations, but labeled differently (i.e. “ User
1” and “User 2”), the tracker model must create additional
decision boundaries in the feature space (right figure). These
additional decision boundaries decrease the likelihood of associating the user with her original feature space.
For simplicity, we explore the base case where the user
is able to obtain one single Sybil identity to perform feature space obfuscation on her behalf. Our technique becomeseven more effective with multiple Sybils, but provides muchof its benefit with images labeled with a single Sybil identity.
Creating Sybil images.
Sybil images are created byadding a specially designed cloak to a set of candidate images. Let xC be an image from the set of candidates the user
obtains (i.e. images generated by a GAN) to populate the
Sybil account. To create the final Sybil image, we create acloak δ(xC , x) that minimizes the feature space separation between xC and user’s original image x, for each candidate. The
optimization is equivalent to setting x as the target and optimizing to create xC ⊕ δ(xC , x) as discussed in §4. After choosing the final xc from all the candidates, a ready-to-upload
Sybil image xS = xC ⊕ δ(xC , x).

7.3 Efficacy of Sybil Images
Sybil accounts can increase a user’s protection success rate
when the tracker controls a small number of a user’s un11

0.8
0.6
0.4
0.2
0
0

0.2

0.4

Ratio of Leaked Uncloaked Images

Figure 10: Protection success rate
decreases when the tracker has more
original user images. (User/Tracker:
Web-Incept)

1

Protection Success Rate

1

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao
Protection Success Rate

Protection Success Rate

Proc. of USENIX Security Symposium 2020

0.8
0.6
0.4
Without Sybil
With Sybil
With Sybil (x2)

0.2
0
0.4

0.6

0.8

1

1
0.8
0.6
0.4
Without Sybil
With Sybil
With Sybil (x2)

0.2
0
0.4

Ratio of Cloaked Images

Figure 11: Protection success rate ishigh when the user has a Sybil account,even if tracker has original user images.
(User/Tracker: Web-Incept)

cloaked images. To experimentally validate this claim, wechoose a label from the tracker’s dataset to be the Sybil account (controlled by the user), and split the user’s images into
two disjoint sets: A contains images that were processed by
Fawkes, and whose cloaked versions have been shared online; and B contains original images leaked to the tracker. For
each synthetic image of the Sybil, we randomly select an uncloaked image of the user in set A. We select one Sybil image
per uncloaked image in A. Then, we cloak all the candidate
images using the methodology discussed in §4. The resulting Sybil images mimic the feature space representation of
uncloaked user images. From the tracker’s perspective, they
have access to cloaked user images from set A, uncloaked
images from set B, and the Sybil images.
Figure 11 compares the protection success rate with andwithout Sybil accounts (with Web-Incept as user’s andtracker’s feature extractor). The use of a Sybil account significantly improves the protection success rate when an attacker
has a small number of original images. The protection success rate remains above 87% when the ratio of the original
images owned by the tracker is less than 31%.
As discussed, a user can create as many Sybil images as
they desire. When the user uploads more Sybil images, theprotection success rate increases. Figure 11 shows that when
the user has uploaded 2 Sybil images per uncloaked image,the protection success rate increases by 5.5%.
Jointly Optimize Multiple Feature Extractors. The user
may not know the tracker’s exact feature extractor. However,given the small number of face feature extractors available
online, she is likely to know that the tracker would use one ofseveral candidate feature extractors. Thus, she could jointly
optimize the Sybil cloaks to simultaneously fool all the candidate feature extractors.
We test this in a simple experiment by jointly optimizing
Sybil cloaks on the four feature extractors from §5. We evaluate the cloak’s performance when the tracker uses one of
the four. Figure 12 shows the Sybil effectiveness averaged
across the 4 feature extractors. The average protection success rate remains above 65% when the ratio of the original

0.6

0.8

1

Ratio of Cloaked Images

Figure 12: Sybils jointly optimized onfour feature extractors have reasonably
high protection success for each individual extractor.

images owned by the tracker is less than 31%.

8 Countermeasures
In this section, we explore potential countermeasures a
tracker could employ to reduce the effectiveness of image
cloaking. We consider and (where possible) empirically validate methods to remove cloaks from images, as well as
techniques to detect the presence of cloak perturbations
on images. Our experiments make the strongest possible
assumption about the tracker: that they know the precisefeature extractor a user used to optimize cloaks. We test
our countermeasures on a tracker’s model trained on the
FaceScrub dataset. Cloaks were generated using the samerobust VGG2-Dense feature extractor from §5.3.
Inherent Limits on Cloaking Success. We acknowledge
that cloaking becomes less effective when an individual isan active target of a tracker. If a tracker strongly desires totrain a model that recognizes a certain individual, they cantake drastic measures that cloaking cannot withstand. For example, a tracker could learn their movements or invade theirprivacy (i.e. learn where they live) by following them physically.

8.1 Cloak Disruption
Without knowing which images in the dataset are cloaked,
the tracker may utilize the following techniques to disrupt
Fawkes’ protection performance, 1) transforming images or2) deploying an extremely robust model. We present and evaluate Fawkes’s performance against these two potential countermeasures.
Image Transformation. A simple technique to mitigate
the impact of small image perturbations is to transform images in the training dataset before using them for model training [8, 16]. These transformations include image augmentation, blurring, or adding noise. Additionally, images posted
online are frequently compressed before sharing (i.e. in the
upload process), which could impact cloak efficacy.
12

0.6

0.4

0.4

0.2

0.2

Normal Classification Accuracy
Protection Success Rate

0
3

9

0

15

21

1
0.8

0.6

0.6

0.4

0.4

0.2
0

Kernel Size of Gaussian Blur

0.2

Normal Classification Accuracy
Protection Success Rate

0

10

20

30

0
40

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2
5

Standard Deviation of Gaussian Noise

Figure 13: Normal classification accuracy decreases as input blurring increases but protection success rate remains high.

Figure 14: Normal classification accuracy decreases as Gaussian noise isadded to inputs but protection successrate remains high.

Original

ρ = 0.006

ρ = 0.008

ρ = 0.01

ρ = 0.012

Protection
Success Rate

51%

87%

100%

100%

Figure 16: When the user’s feature extractor is much less robust than the tracker’s feature extractor, the user can improvetheir protection success rate by increasing their DSSIM budget. (User: VGG2-Dense, Tracker: Web-Incept)
However, we find that none of these transformations defeat
our cloaks. The protection success rate remains 100% evenwhen data augmentation is applied to cloaked images 5 . Applying Gaussian blurring degrades normal accuracy by up to
18% (as kernel size increases) while cloak protection successrate remains > 98% (see Figure 13). Adding Gaussian noise
to images merely disrupts normal classification accuracy –the cloak protection success rate remains above 100% as
the standard deviation of the noise distribution increases (seeFigure 14). Even image compression cannot defeat our cloak.
We use progressive JPEG [57], reportedly used by Facebook
and Twitter, to compress the images in our dataset. The image quality, as standard by Independent JPEG Group [1],ranges from 5 to 95 (lower value = higher compression). As
shown in Figure 15, image compression decreases the protection success rate, but more significantly degrades normalclassification accuracy.
Robust Model.
As shown in §5, cloaks constructed on
robust feature extractors transfer well to trackers’ less robustfeature extractors. Thus, a natural countermeasure a tracker
could employ is training their model to be extremely robust.
Despite the theoretically proven trade-off between normal
accuracy and robustness [56], future work may find a way
5 Image augmentation parameters: rotation range=20o , horizontal
shift=15%, vertical shift=15%, zoom range=15%

0.2

Normal Classification Accuracy
Protection Success Rate

0

10

15

Classification Accuracy

0.6

1
0.8

Protection Success Rate

0.8

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao
Classification Accuracy

1

Protection Success Rate

1
0.8

Classification Accuracy

Protection Success Rate

Proc. of USENIX Security Symposium 2020

020

Image Quality

Figure 15: Protection success rate andnormal classification accuracy increase
as image quality increases using JPEG
compression.

to improve model robustness while minimizing the accompanying drop in accuracy. Thus, we evaluate cloaking success when the tracker’s model is much more robust than theuser’s feature extractor. In our simplified test, the user has
a robust VGG2-Dense feature extractor (adversarially trainedfor 3 epochs), while the tracker has an extremely robust
Web-Incept feature extractor (adversarially trained for 20
epochs). When the tracker’s model is this robust, the user’s
cloak only achieves a 64% protection success rate.
However, if the user is extremely privacy sensitive, she
could increase the visibility of her cloak perturbation toachieve a higher protection success rate. Figure 16 highlights the trade off between protection success and the input
DSSIM level. The cloak’s protection success rate increases
to 100% once the DSSIM perturbation is > 0.01.

8.2 Cloak Detection
We now propose techniques a tracker could employ to detectcloaked images in their dataset. We also discuss mitigations
the user could apply to avoid detection.
Existing Poison Attack Detection. Since cloaking is aform of data poisoning, prior work on detecting poisoning
attacks [11, 19, 40, 46, 49, 58] could be helpful. However, allprior works assume that poisoning only affects a small percentage of training images, making outlier detection useful.
Fawkes poisons an entire model class, rendering outlier detection useless by removing the correct baseline.
Anomaly Detection w/o Original Images. We first consider anomaly detection techniques in the scenario where thetracker does not have any original user images. If trackers
obtain both target and cloaked user images, they can detect
unusual closeness between cloaked images and target images
in model feature space. Empirically, the L2 feature space distance between the cloaked class centroid and the target class
centroid is 3 standard deviations smaller than the mean separation of other classes. Thus, user’s cloaked images can bedetected.
However, a user can trivially overcome this detection by13

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao

maintaining separation between cloaked and target images
during cloak optimization. To show this, we use the same experimental setup as in §5.2 but terminate the cloak optimization once a cloaked image is 20% of the original L2 distancefrom the target image. The cloak still achieves a 100% protection success rate, but the cloak/target separation remains
large enough to evade the previous detection method.
Anomaly Detection w/ Original Images. When the trackers have access to original training images (see §7), they
could use clustering to see if there are two distinct featureclusters associated with the user’s images (i.e. cloaked anduncloaked). Normal classes should have only one featurecluster. To do this, the tracker could run a 2-means clusteringon each class’s feature space, flagging classes with two distinct centroids as potentially cloaked. When we run this experiment, we find that the distance between the two centroidsof a protected user class is 3 standard deviations larger than
the average centroid separation in normal classes. In this way,
the tracker can use original images to detect the presence of
cloaked images.
To reduce the probability of detection by this method, theuser can choose a target class that does not create such a large
feature space separation. We empirically evaluate this mitigation strategy using the same experimental configuration
as in §5.2 but choose a target label with average (rather thanmaximal) distance from their class. The cloak generated withthis method still achieves a 100% protection success rate, but
L2 distance between the two cluster centroids is within 1 standard deviation of average.
The user can evade this anomaly detection strategy using
the maximum distance optimization strategy in §4. In practice, for any tracker model with a moderate number of labels (>30), cloaks generated with average or maximum difference optimization consistently achieves high cloaking success. Our experimental results show these two methods perform identically in protection success against both our local
models and the Face++ API.

impact on those using public images to build unauthorizedfacial recognition models and less so on agencies with legal
access to facial images such as federal agencies or law enforcement. We leave more detailed exploration of the tradeoff between user privacy and authorized use to future work.
Protecting content using cloaks faces the inherent challenge of being future-proof, since any technique we use to
cloak images today might be overcome by a workaround insome future date, which would render previously protected
images vulnerable. While we are under no illusion that this
proposed system is itself future-proof, we believe it is an important and necessary first step in the development of usercentric privacy tools to resist unauthorized machine learningmodels. We hope that followup work in this space will leadto long-term protection mechanisms that prevent the miningof personal content for user tracking and classification.

9 Discussion and Conclusion
In this paper, we present a first proposal to protect individuals from recognition by unauthorized and unaccountable facial recognition systems. Our approach applies small, carefully computed perturbations to cloak images, so that they are
shifted substantially in a recognition model’s feature representation space, all while avoiding visible changes. Our techniques work under a wide range of assumptions and provide
100% protection against widely used, state-of-the-art models
deployed by Microsoft, Amazon and Face++.
Like most privacy enhancing tools and technologies,
Fawkes can also be used by malicious bad actors. For example, criminals could use Fawkes to hide their identity fromagencies that rely on third-party facial recognition systems
like Clearview.ai. We believe Fawkes will have the biggest

Acknowledgments
We thank our shepherd David Evans and anonymous reviewers for their constructive feedback. This work is supported
in part by NSF grants CNS-1949650, CNS-1923778, CNS1705042, and by the DARPA GARD program. Any opinions,findings, and conclusions or recommendations expressed inthis material are those of the authors and do not necessarilyreflect the views of any funding agencies.

References[1] http://apodeline.free.fr/DOC/libjpeg/libjpeg-3.
html. Using the IJG JPEG library: Advanced features.
[2] https://aws.amazon.com/rekognition/.
Rekognition Face Verification API.

Amazon[3] https://azure.microsoft.com/en-us/services/cognitive-services/face/. Microsoft Azure Face API.
[4] https://www.faceplusplus.com/face-searching/.
Face++ Face Searching API.
[5] http://vision.seas.harvard.edu/pubfig83/.
PubFig83: A resource for studying face recognition in personal
photo collections.
[6] A BADI , M., C HU , A., G OODFELLOW, I., M C M AHAN ,H. B., M IRONOV, I., TALWAR , K., AND Z HANG , L. Deeplearning with differential privacy. In Proc. of CCS (2016).
[7] C AO , Q., S HEN , L., X IE , W., PARKHI , O. M., AND Z ISSER MAN , A. VGGFace2: A dataset for recognising faces acrosspose and age. In Proc. of IEEE FG (2018).
[8] C ARLINI , N., AND WAGNER , D. Adversarial examples are
not easily detected: Bypassing ten detection methods. In Proc.
of AISec (2017).
[9] C ARLINI , N., AND WAGNER , D. Towards evaluating the robustness of neural networks. In Proc. of IEEE S&P (2017).
[10] C ARLINI , N., AND WAGNER , D. Towards evaluating the robustness of neural networks. In Proc. of IEEE S&P (2017).

14

Proc. of USENIX Security Symposium 2020

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao[11] C HEN , B., C ARVALHO , W., BARACALDO , N., L UDWIG , H.,
E DWARDS , B., L EE , T., M OLLOY, I., AND S RIVASTAVA , B.
Detecting backdoor attacks on deep neural networks by activation clustering. arXiv:1811.03728 (2018).

[28] L I , Y., YANG , X., W U , B., AND LYU , S. Hiding faces inplain sight: Disrupting AI face synthesis with adversarial perturbations. arXiv:1906.09288 (2019).

[12] C HEN , Y., L I , H., T ENG , S.-Y., NAGELS , S., L I , Z., L OPES ,P., Z HAO , B. Y., AND Z HENG , H. Wearable microphone jamming. In Proc. of ACM CHI (April 2020).
[13] C ROSS , J. Valley attorney: Facebook facial recognition carries identity theft risk. KTAR News (September 2019).

[29] L IU , Y., C HEN , X., L IU , C., AND S ONG , D.
Delving
into transferable adversarial examples and black-box attacks.
arXiv:1611.02770 (2016).
[30] M ADRY, A., M AKELOV, A., S CHMIDT, L., T SIPRAS , D.,
AND V LADU , A. Towards deep learning models resistant toadversarial attacks. arXiv:1706.06083 (2017).

[14] D EMONTIS , A., M ELIS , M., P INTOR , M., JAGIELSKI , M.,
B IGGIO , B., O PREA , A., N ITA -ROTARU , C., AND ROLI , F.
Why do adversarial attacks transfer? explaining transferabilityof evasion and poisoning attacks. In Proc. of USENIX Security(2019), pp. 321–338.

[31] M ARI , A. Brazilian retailer quizzed over facial recognitiontech. ZDNet (March 2019).

[15] DWORK , C. Differential privacy: A survey of results. In Proc.
of TAMC (2008).

[33] M OZUR , P. Inside China’s dystopian dreams: A.I., shame andlots of cameras. The New York Times (July 2018).

[16] F EINMAN , R., C URTIN , R. R., S HINTRE , S., AND G ARD NER , A. B. Detecting adversarial samples from artifacts.
arXiv:1703.00410 (2017).

[34] N ECH , A., AND K EMELMACHER -S HLIZERMAN , I. Level
playing field for million scale face recognition. In Proc. of
CVPR (2017).

[17] F REDRIKSON , M., J HA , S., AND R ISTENPART, T. Model inversion attacks that exploit confidence information and basic
countermeasures. In Proc. of CCS (2015).

[35] N EWTON , E. M., S WEENEY, L., AND M ALIN , B. Preserving
privacy by de-identifying face images. IEEE transactions on
Knowledge and Data Engineering 17, 2 (2005), 232–243.

[18] G OODFELLOW, I. J., S HLENS , J., AND S ZEGEDY,C.
Explaining and harnessing adversarial examples.
arXiv:1412.6572 (2014).

[36] N G , H.-W., AND W INKLER , S. A data-driven approach tocleaning large face datasets. In Proc. of ICIP (2014).

[19] G UPTA , N., H UANG , W. R., F OWL , L., Z HU , C., F EIZI ,S., G OLDSTEIN , T., AND D ICKERSON , J. P.
Strongbaseline defenses against clean-label poisoning attacks.
arXiv:1909.13374 (2019).
[20] H ILL , K. The secretive company that might end privacy as
we know it. The New York Times (January 18 2020).
[21] H ILL , K., AND K ROLIK , A. How photos of your kids are
powering surveillance technology. The New York Times (October 11 2019).
[22] H UANG , G., L IU , Z., VAN D ER M AATEN , L., AND W EIN BERGER , K. Q. Densely connected convolutional networks.
In Proc. of CVPR (2017).
[23] JAN , S. T., M ESSOU , J., L IN , Y.-C., H UANG , J.-B., AND
WANG , G. Connecting the digital and physical world: Improving the robustness of adversarial attacks. In Proc. of AAAI
(2019).
[24] KOMKOV, S., AND P ETIUSHKO , A. Advhat: Real-world adversarial attack on arcface face id system. arXiv:1908.08705(2019).

[32] M ETZ , C., AND C OLLINS , K. How an A.I. ‘cat-and-mousegame’ generates believable fake photos. The New York Times(January 2018).

[37] N OCEDAL , J., AND W RIGHT, S. Numerical optimization, series in operations research and financial engineering. Springer,
New York, USA, 2006 (2006).
[38] O’F LAHERTY, K. Facial recognition at u.s. airports. should
you be concerned? Forbes (March 2019).
[39] PAPERNOT, N., M C DANIEL , P., AND G OODFELLOW,
I.
Transferability in machine learning: From phenomena to black-box attacks using adversarial samples.
arXiv:1605.07277 (2016).
[40] PAUDICE , A., M UÑOZ -G ONZÁLEZ , L., G YORGY, A., AND
L UPU , E. C.
Detection of adversarial training examples in poisoning attacks through anomaly detection.
arXiv:1802.03041 (2018).
[41] S ATARIANO , A. Police use of facial recognition is acceptedby British court. The New York Times (September 2019).
[42] S HAFAHI , A., H UANG , W. R., NAJIBI , M., S UCIU , O.,S TUDER , C., D UMITRAS , T., AND G OLDSTEIN , T. Poison
frogs! targeted clean-label poisoning attacks on neural networks. In Proc. of NeurIPS (2018).

[25] K URAKIN , A., G OODFELLOW, I., AND B ENGIO , S. Adversarial examples in the physical world. arXiv:1607.02533(2016).

[43] S HAN , S., W ENGER , E., WANG , B., L I , B., Z HENG , H.,
AND Z HAO , B. Y. Gotta catch ’em all: Using honeypots tocatch adversarial attacks on neural networks. In Proc. of CCS(Orlando, FL, November 2019). arXiv:1904.08554.

[26] L EE , N. Having multiple online identities is more normal than
you think. Engadget, March 2016. https://www.engadget.
com/2016/03/04/multiple-online-identities.

[44] S HARIF, M., B HAGAVATULA , S., BAUER , L., AND R EITER ,M. K. Accessorize to a crime: Real and stealthy attacks onstate-of-the-art face recognition. In Proc. of CCS (2016).

[27] L I , T., AND L IN , L.
Anonymousnet: Natural face deidentification with measurable privacy. In Proc. of CVPR
(2019).

[45] S HEN , J., Z HU , X., AND M A , D. Tensorclog: An imperceptible poisoning attack on deep neural network applications.
IEEE Access 7 (2019), 41498–41506.

15

Proc. of USENIX Security Symposium 2020[46] S HEN , S., T OPLE , S., AND S AXENA , P. Auror: Defending
against poisoning attacks in collaborative deep learning systems. In Proc. of ACSAC (2016).
[47] S HWAYDER , M. Clearview AI’s facial-recognition app is anightmare for stalking victims. Digital Trends (January 2020).
[48] S ONG , C., R ISTENPART, T., AND S HMATIKOV, V. Machinelearning models that remember too much. In Proc. of CCS(2017).
[49] S TEINHARDT, J., KOH , P. W. W., AND L IANG , P. S. Certified defenses for data poisoning attacks. In Proc. of NeurIPS(2017).
[50] S UCIU , O., M ĂRGINEAN , R., K AYA , Y., DAUMÉ III, H.,
AND D UMITRA Ş , T. When does machine learning fail? generalized transferability for evasion and poisoning attacks. In
Proc. of USENIX Security (2018).
[51] S UN , Q., M A , L., J OON O H , S., VAN G OOL , L., S CHIELE ,B., AND F RITZ , M. Natural and effective obfuscation by headinpainting. In Proc. of CVPR (2018).
[52] S UN , Q., T EWARI , A., X U , W., F RITZ , M., T HEOBALT, C.,
AND S CHIELE , B. A hybrid model for identity obfuscation by
face replacement. In Proc. of ECCV (2018).
[53] S ZEGEDY, C., I OFFE , S., VANHOUCKE , V., AND A LEMI ,A. A. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proc. of AAAI (2017).
[54] S ZEGEDY, C., Z AREMBA , W., S UTSKEVER , I., B RUNA , J.,
E RHAN , D., G OODFELLOW, I., AND F ERGUS , R. Intriguing
properties of neural networks. arXiv:1312.6199 (2013).
[55] T HYS , S., VAN R ANST, W., AND G OEDEMÉ , T. Fooling automated surveillance cameras: adversarial patches to attack
person detection. In Proc. of CVPR (workshop) (2019).
[56] T SIPRAS , D., S ANTURKAR , S., E NGSTROM , L., T URNER ,A., AND M ADRY, A. Robustness may be at odds with accuracy. arXiv:1805.12152 (2018).
[57] WALLACE , G. K. The JPEG still picture compression standard. IEEE Transactions on Consumer Electronics 38, 1(1992).
[58] WANG , B., YAO , Y., S HAN , S., L I , H., V ISWANATH , B.,Z HENG , H., AND Z HAO , B. Y. Neural cleanse: Identifying
and mitigating backdoor attacks in neural networks. In Proc.
of IEEE S&P (2019).
[59] WANG , B., YAO , Y., V ISWANATH , B., Z HENG , H., ANDZ HAO , B. Y. With great training comes great vulnerability:

S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng and B. Y. Zhao
Practical attacks against transfer learning. In Proc. of USENIX
Security (2018).
[60] WANG , G., KONOLIGE , T., W ILSON , C., WANG , X.,Z HENG , H., AND Z HAO , B. Y. You are how you click:
Clickstream analysis for sybil detection. In Proc. of USENIX
Security (2013), pp. 241–256.
[61] WANG , Z., B OVIK , A. C., S HEIKH , H. R., AND S IMON CELLI , E. P. Image quality assessment: From error visibility
to structural similarity. IEEE Trans. on Image Processing 13,
4 (2004), 600–612.
[62] WANG , Z., S IMONCELLI , E. P., AND B OVIK , A. C. Multiscale structural similarity for image quality assessment. In
Proc. of Asilomar Conference on Signals, Systems & Computers (2003), vol. 2, IEEE, pp. 1398–1402.
[63] W ONDRACEK , G., H OLZ , T., K IRDA , E., AND K RUEGEL , C.
A practical attack to de-anonymize social network users. In
Proc. of IEEE S&P (2010).
[64] W U , Y., YANG , F., AND L ING , H. Privacy-Protective-GAN
for face de-identification. arXiv:1806.08906 (2018).
[65] W U , Z., L IM , S.-N., DAVIS , L., AND G OLDSTEIN , T. Making an invisibility cloak: Real world adversarial attacks on object detectors. arXiv:1910.14667 (2019).
[66] YANG , C., W U , Q., L I , H., AND C HEN , Y.
Generative poisoning attack method against neural networks.
arXiv:1703.01340 (2017).
[67] YANG , Z., W ILSON , C., WANG , X., G AO , T., Z HAO , B. Y.,
AND DAI , Y.
Uncovering social network sybils in thewild. ACM Transactions on Knowledge Discovery from Data
(TKDD) 8, 1 (2014), 1–29.
[68] YANG , Z., Z HANG , J., C HANG , E.-C., AND L IANG , Z. Neural network inversion in adversarial setting via backgroundknowledge alignment. In Proc. of CCS (London, UK, November 2019).
[69] Y I , D., L EI , Z., L IAO , S., AND L I , S. Z. Learning face representation from scratch. arXiv:1411.7923 (2014).
[70] YOSINSKI , J., C LUNE , J., B ENGIO , Y., AND L IPSON , H.
How transferable are features in deep neural networks? In
Proc. of NeurIPS (2014).
[71] Z HU , C., H UANG , W. R., S HAFAHI , A., L I , H., TAYLOR ,G., S TUDER , C., AND G OLDSTEIN , T. Transferable cleanlabel poisoning attacks on deep neural nets. In Proc. of ICML
(2019).

16