Data Poisoning Won’t Save You From Facial Recognition
Authors: Evani Radiya-Dixit
Sanghyun Hong
Nicholas Carlini
Florian Tramer
Venue: ICLR 2022 Poster (2022)
Keywords: Poisoning attacks
adversarial examples
facial recognition
arms race
defenses

Abstract: Data poisoning has been proposed as a compelling defense against facial recognition models trained on Web-scraped pictures. Users can perturb images they post online, so that models will misclassify future (unperturbed) pictures.
  
 We demonstrate that this strategy provides a false sense of security, as it ignores an inherent asymmetry between the parties: users' pictures are perturbed once and for all before being published (at which point they are scraped) and must thereafter fool all future models---including models trained adaptively against the users' past attacks, or models that use new technologies discovered after the attack.
  
We evaluate two systems for poisoning attacks against large-scale facial recognition, Fawkes (500,000+ downloads) and LowKey. We demonstrate how an "oblivious" model trainer can simply wait for future developments in computer vision to nullify the protection of pictures collected in the past. We further show that an adversary with black-box access to the attack can (i) train a robust model that resists the perturbations of collected pictures and (ii) detect poisoned pictures uploaded online.
  
We caution that facial recognition poisoning will not admit an "arms race" between attackers and defenders. Once perturbed pictures are scraped, the attack cannot be changed so any future successful defense irrevocably undermines users' privacy.

Published as a conference paper at ICLR 2022
DATA POISONING WON'T SAVE YOU FROM FACIAL RECOGNITION
Evani Radiya-Dixit Stanford University
Sanghyun Hong Oregon State University
Nicholas Carlini Google
Florian Tramer
Stanford University, Google
ABSTRACT
Data poisoning has been proposed as a compelling defense against facial recogni- tion models trained on Web-scraped pictures. Users can perturb images they post online, so that models will misclassify future (unperturbed) pictures.
We demonstrate that this strategy provides a false sense of security, as it ignores an inherent asymmetry between the parties: users' pictures are perturbed once and for all before being published (at which point they are scraped) and must thereafter fool all future models-including models trained adaptively against the users' past attacks, or models that use technologies discovered after the attack.
We evaluate two systems for poisoning attacks against large-scale facial recognition, Fawkes (500,000+ downloads) and LowKey. We demonstrate how an "oblivious" model trainer can simply wait for future developments in computer vision nullify the protection of pictures collected in the past. We further show that an adversary with black-box access the attack can (i) train robust model that resists the perturbations of collected pictures and (ii) detect poisoned pictures uploaded online. We caution that facial recognition poisoning will not admit an "arms race" between attackers and defenders. Once perturbed pictures are scraped, the attack cannot be changed so any future successful defense irrevocably undermines users' privacy.
INTRODUCTION
Facial recognition systems pose a serious threat to individual privacy. Various companies routinely scrape the Web for users' pictures to train large-scale facial recognition systems (Hill, 2020a; Harwell, 2021), and then make these systems available to law enforcement agencies (Lipton, 2020) or private individuals (Harwell, 2021; Mozur & Krolik, 2019; Wong, 2019).
A growing body of work develops tools to allow users to fight back, using techniques from adversarial machine learning (Sharif al., 2016; Oh et al., 2017: Thys et al., 2019; Kulynych al., 2020; Shan et al., 2020; Evtimov et al.. 2020; Gao et al., 2020; Xu et al., 2020; Yang et al., 2020; Komkov & Petiushko, 2021: Cherepanova al., 2021a; Rajabi al., 2021: Browne al., 2020).
One approach taken by these tools lets users perturb any picture before they post it online, so that facial recognition models that train on these pictures will become poisoned. The objective that when an unperturbed image is fed into the poisoned model (e.g.. photo taken by stalker, security camera, or the police), the model misidentifies the user. This approach was popularized by Fawkes (Shan et al., 2020), an academic image-poisoning system with 500,000+ downloads and covered by the New York Times (Hill, 2020b), that promises "strong protection against unauthorized [facial recognition] models". Following Fawkes' success, similar systems have been proposed by academic (Cherepanova et al., 2021a; Evtimov et al., 2020) and commercial (Vincent, 2021) parties.
This paper shows that these systems (and, in fact, any poisoning strategy) cannot protect users' privacy. Worse, we argue that these systems offer a false sense of security. There exists a class of privacy-conscious users who might have otherwise never uploaded their photos to the internet; however who now might do so, under the false belief that data poisoning will protect their privacy. These users are now less private than they were before. Figure shows overview of our results.

---

Published as a conference paper at ICLR 2022
1) User Perturbs images 2) Images Are Scraped
Model Training
4) Model Evaluation
Protection Rate (%)
No Delensee
100
Attack
Brad Pitt
User perturbs images using public attack
Oblivious Delense
Walt year and frai new model on maggea acrapped a year ago
Oprah
Adaptive Defense
Perturb images of other users
Attack
Oprah
User posts perturbed images online
Model frainer scrapes the Web for images
Faw
LowKey
Augment for robust
Attacks and defenses for facial recognition poisoning. (1) Users perturb their pictures before posting them online. (2) A model trainer continuously scrapes the Web for pictures. (3-4) The model trainer builds a model from collected pictures and evaluates it unperturbed pictures. With no defense strategy, the poisoned model fails to recognize users whose online pictures were perturbed. An "oblivious" model trainer and retroactively train it on past pictures to resist poisoning. An adaptive model trainer with black-bor access the attack employed by users can immediately train robust model that resists poisoning. We show the effectiveness of these defenses against the Fawkes and LowKey poisoning attacks.
Figure I:
wait until better facial recognition model is discovered
The reason these systems are not currently private, and can never be private, comes down fundamental asymmetry between Web users and the trainers of facial recognition models. Once user commits to an attack and uploads a perturbed picture that gets scraped, this perturbation can no longer be changed. The model trainer, who acts second, then gets to choose their training strategy. As prior work lacks a formal security setup, we begin by defining a security game to capture this dynamic nature of poisoning attacks.
We then introduce two powerful defense strategies that completely break two state-of teffthh-art art poisoning attacks, Fawkes (Shan et al., 2020) and LowKey (Cherepanova et al., 2021a). In the first strategy, adapt the facial recognition training to work in the presence of poisoned images. Because image- perturbation systems made publicly accessible to cater large user base (Shan al., 2021; Cherepanova et al., 2021b), we must assume facial recognition trainers are aware of these attack techniques. Our adaptive models fully circumvent poisoning with just black-box access to the attack.
Worse, we find there exists an even simpler defensive strategy: model trainers can just wail for better facial recognition systems, which are no longer vulnerable these particular poisoning attacks. That is, because existing poisoning attacks were only designed to prevent current face recognition tools from working, there is no reason to believe that future tools will poisoned as well. Indeed, we show that the state-of-the-ard poisoning attacks are already broken by new training techniques that appeared less than a year later. For example, Fawkes (released in July 2020) ineffective if the model trainer switches to MagFace model (Meng et al., 2021) (released in March 2021), and LowKey (released January 2021) ineffective against a facial recognition model obtained by finetuning OpenAl's CLIP model (Radford et al., 2021) (also released January 2021).
We argue that poisoning attacks against facial recognition will not lead to an "arms race", where new attacks can continuously counteract new defenses. Since the perturbation applied to picture cannot be changed once the picture is scraped, a successful poisoning attack has to remain effective against all future models, even models trained adaptively against the attack, models that new techniques discovered only after the attack. In light of this, we argue that users' only hope is a push for legislation that restricts the use of privacy-invasive facial recognition systems (Singer, 2018; Weise & Singer, 2020; Winder, 2020).
2
DATA POISONING FOR FACIAL RECOGNITION
2.1 THREAT MODEL.
We consider a setting where a user uploads pictures of themselves to an online service such as a social media platform. The user attempts protect their pictures by adding perturbations that should
2

---

Published as a conference paper at ICLR 2022
be almost imperceptible to other people (Szegedy et al., 2013). The user's goal is that model trained on their perturbed pictures will achieve low accuracy when classifying unperturbed pictures of the user (Shan et al., 2020; Cherepamova et al., 2021a; Yang et al., 2020; Evtimov et al., 2020).
second party, the model trainer, scrapes the Web for pictures to train large-scale facial recognition model (capable of identifying large number of users). We assume that the data scraped by the trainer is labeled, i.e., all (possibly perturbed) images collected of a user can be assigned to the user's s identity. The trainer's goal is to build a model that correctly recognizes users in future images. The trainer is active, i.e., they continuously scrape new uploaded pictures at regular intervals.
This setting corresponds to that of training-only clean-label poisoning attacks (Shan et al., 2020; Cherepanova et al., 2021a; Goldblum et al., 2020; Evtimov et al.. 2020). Keeping with the terminology of the data poisoning literature (Goldblum al., 2020), we refer to the user as the attacker and the trainer as the defender (even though it is the trainer that aims to breach the user's 's privacy!).
2.2 POISONING ATTACK GAMES
We present a standard security game for training-only clcan-label poisoning attacks in Figure Sa. We argue that this game fails to properly capture the threat model of our facial recognition scenario.
In this game, the attacker first samples training data X, Y from a distribution D. The attacker then applies an attack to get the perturbed data Xadv- The defender gets the perturbed labeled data (Xavv. Y) and trains a model f. The model f is evaluated on unperturbed inputs I from the distribution D. For a given test input z, the attacker wins the game if the perturbation of the training data is small (as measured by an oracle O(X, Xadv) + (0, 1)), and if the model misclassifies x.
The poisoning game in Figure Sa fails capture an important facet of the facial recognition problem. The problem is not static: users continuously upload new pictures, and the model trainer actively scrapes them to update their model. Below, we introduce dynamic version of the poisoning game, and show how a model trainer can use a retroactive defense strategy to win the game. In turn, we discuss how users and model trainers may adapt their strategies based on the other party's actions.
Dynamic poisoning attacks. To capture the dynamic nature of the facial recognition game, we define a generalized game for clean-label poisoning attacks in Figure 8b. The game now operates in rounds indexed by i 2 1. In each round, the attacker perturbs new pictures and sends them to the defender. The strategies of the attacker and defender may change from one round to the next.
The game in Figure 8b allows for the data distribution D; change across rounds. Indeed, new users might begin uploading pictures, and users' faces may change over time. Yet, our thesis is that the main challenge faced by the user is precisely that the distribution of pictures of their own face changes little over time. For example, a facial recognition model trained on pictures of user at 20 years old can reliably recognize pictures of the same user at 30 years old (Ling et al., 2010). Thus, in each round the defender can reuse training data (Kadv, Y) collected in prior rounds. If the defender scrapes a user's 's images, the perturbations applied to these images cannot later be changed.
Retroactive defenses. The observation above places a high burden on the attacker. Suppose that in round i, the defender discovers a training technique t ra in; that is resilient to past poisoning attacks Attack, for j < i. Then, the defender can train their model solely on data (Xadv. 3)) collected up to round j. From there on, the defender can trivially win the game by simply ignoring future training data (until they find a defense against newer attacks as well). Thus, the attacker's 's perturbations have to work against all future defenses, even those applied retroactively, for long as the user's facial features do not naturally change. By design, this retroactive defense does not lead to an "arms race" with future attacks. The defender applies newly discovered defenses to past pictures only.
As we will show, this retroactive defense can even be instantiated by a fully oblivious model trainer, with knowledge of users' attacks. The model trainer simply waits for better facial recognition model to be developed, and then applies the model to pictures scraped before the new model was published. This oblivious strategy demonstrates the futility of preventing facial recognition with data poisoning, so long as progress in facial recognition models is expected to continue in the future.
Adaptive defenses. A model trainer that does not want to wait for progress in facial recognition can exploit another source of asymmetry over users: adaptivit. In our setting, it is easier for the
3

---

Published as a conference paper at ICLR 2022
defender to adapt the attacker, than vice-versa. Indeed, users must perturb their pictures before the model trainer scrapes them and feeds them to a secret training algorithm. As the trainer's 's model f will likely be inaccessible to users, users will have no idea if their attack actually succeeded not.
In contrast, the users' attack strategy is likely public (at least as a black-box) to support users with minimal technical background. For example, Fawkes offers open-source software to perturb images (Shan et al., 2021), and LowKey (Cherepanova et al., 2021b) and DoNotPay (Vincent, 2021) offer a Web API. The defender can thus assemble dataset of perturbed images and use them train a model. We call such defender adaptive.'
note on evasion and obfuscation attacks. The security games in Figure assume that the training data is "clean label" (i.e., the user still be identified in their pictures by other human users) and that the evaluation data is unperturbed. This the setting considered by Fawkes (Shan al., 2020) and LowKey (Cherepanova et al., 2021a), where the user cannot control the pictures that are fed to the facial recognition model (e.g., pictures taken by stalker, security camera, or law enforcement).
shares their pictures online, but
The game dynamics change if the user evades the model with adversarial examples, by modifying their facial appearance at test time (Szegedy al., 2013; Sharif al., 2016; Thys et al., 2019; Gao et al.. 2020; Cilloni et al., 2020; Rajabi et al., 2021; Oh et al., 2017; Deb et al., 2019; Browne et al., 2020; Deb et al., 2020). Evasion attacks favor the attacker: the defender must commit to a defense and the attacker can adapt their strategy accordingly (Tramer et al., 2020).
Our setting and security game also do not capture face obfuscation or anonymization techniques (New- ton et al., 2005; Sun et al., 201 8a;b; Sam et al., 2020; Cao et al., 2021; Maximov et al., 2020; Gafni et al., 2019). These attacks remove or synthetically replace a user's face, and thus fall outside of our threat model of clean-label poisoning attacks (i.e., the aim of these works is to remove identifying features from uploaded pictures, so that even a human user would fail to identify the user).
EXPERIMENTS
We evaluate facial recognition poisoning tools, Fawkes (Shan al., 2020) and LowKey (Cherepanova et al., 2021a), against various adaptive and oblivious defenses. We show that:
An adaptive model trainer with black-box access Fawkes and LowKey can train robust model that resists poisoning attacks and correctly identifies all users with high accuracy.
An adaptive model trainer can also detect perturbed pictures with near-perfed accuracy.
Fawkes and LowKey already broken by newer facial recognition models that appeared less than a year after the attacks were introduced.
Achieving robustness against poisoning attacks need not come at cost in clean accuracy (in contrast to existing defenses against adversarial examples (Tsipras et al., 2018)).
ramer/ Code to reproduce our experiments is available at: htttps::/githuu..com / FaceCure.
3.1 ATTACKS
We evaluate three distinct poisoning attacks:
Fawkes v0.3: this the attack originally released by Fawkes (Shan et al., 2020) in July 2020. received 500,000 downloads by April 2021 (Shan et al., 2021).
Fawkes vł.0: this is major update to the Fawkes attack from April 2021 (Shan et al., 2021). LowKey: this attack was published in January 2021 with an acommpanying Web applica- tion (Cherepanova et al., 2021a;b).
'A black-bor adaptive defense might be preventable with an attack that uses secref per-user randommess to ensure that robustness to attack from user does not generalize other users. Existing attacks fail do this, and designing such an attack open problem. Moreover, such an attack would remain vulnerable oblivious strategy.
Unless noted otherwise, for our experiments with Fawkes, we use the most recent version 1.0 of the tool in its strongest protection mode ("high").
4

---

Published as a conference paper at ICLR 2022
These three attacks rely on the same underlying principle. Each attack perturbs user's online pictures with adversarial examples (Szegedy et al., 2013) so as to poison the training set of a facial recognition model. The attack's goal is that the facial recognition model learns to associate user with spurious features that are not present in unperturbed pictures. Since the user does not know the specifies of the model trainer's facial recognition pipeline, the above attacks craft adversarial examples against set of known facial recognition models (so-called surrogate models), in hopes that these adversarial perturbations will then transfer other models (Papernot al., 2016).
3.2 EXPERIMENTAL SETUP
The experiments in this section are performed with the FaceScrub dataset (Ng & Winkler, 2014), which contains over 50,000 images of 530 celebrities. Each user's pictures are aligned (to extract the face) and split into training set (pictures that posted online and scraped) and test set, at 70%-30% split. Additional details on the setup for each experiment can be found in Appendix B. We replicate main results with different dataset, PubFig (Kumar et al., 2009) in Appendix C.2.
Attacker setup. A user (one of FaceSrub's identities) perturbs all of their training data (i.e., their pictures uploaded online). We use Fawkes and Lowkey's official attack code in their strongest setting.
Model trainer setup. We consider a standard approach for facial recognition wherein the model trainer uses fixed pre-trained feature extractor g(x)) to convert pictures into embeddings. Evaluation done using a 1-Nearest Neighbor rule. Given a test image x, we find the training example x that minimizes | (z) g g<x')l ll and return the identity y' associated with x'.
In Appendix C.I, we evaluate an alternative setup considered by Fawkes (Shan et al., 2020), where the model trainer converts the feature extractor g(x) into supervised classifier (by adding linear layer on top of g) and fine-tunes the classifier on labeled pictures. This setting is less representative of large-scale facial recognition pipelines where the set of labels is not explicitly known.
Feature extractors. We consider a variety of pre-trained feature extractors that can be used by model trainer to build facial recognition system. We order the extractors chronologically by the date at which all training components necessary to replicate the model (model architecture, training data and loss function) were published.
FaceNet: An Inception ResNet model pre-trained on VGG-Face2 (Schroff et al., 2015).
WebFace: An Inception ResNet model pre-trained on CASIA-WebFace (Yi et al., 2014). This feature extractor is used as a surrogate model in the Fawkes v0.3 attack.
VGG-Face: A VGG-16 16 model pre- pre-trained on VGG-Face2 (Cao et al., 2018).
CelebIM: A ResNet trained on MS-Celeb-IM (Guo et al., 2016) with the ArcFace loss (Deng et al., 2018). This feature extractor used as a surrogate model in the Fawkes v1.0 attack.
ArcFace: ResNet trained on CASiA-Webface with the ArcFace loss (Deng et al., 2018).
MagFace: A ResNet trained on MS- Mes-Celeb-IM 1M with the MagFace loss (Meng et al., 2021).
CLIP: OpenAl's CLIP (Radford et al., 2021) vision transformer model, which fine-tuned on CASIA- Weeeeee and VGG-Face2. Details on this model are in Appendix B.3. We are not yet aware of facial recognition system built upon CLIP. Yet, due the the model's strong performance in transfer-learing and out-of-distributin generalization, it is good candidate to test how existing attacks fare against facial recognition systems based on novel techniques (e.g.. vision transformers and contrastive learning).
Evaluation metric. We evaluate the effectiveness of Fawkes and LowKey by the (top-1) error rate (a.k.a. protection rate) of the facial recognition classifier when evaluated on the unperturbed test images of the chosen user. We repeat each of our experiments 20 times with different user in the position of the attacker, and report the average error rate across all 20 users.
3.3 ADAPTIVE DEFENSES
In this section, we assume that users perturb pictures using a public service (e.g.. a Web application), to which the model trainer has black-box access. This assumption is realistic for Fawkes and LowKey, since both attacks offer a publicly accessible application. We show how the model trainer can adaptively train a feature extractor that resists these attacks.
5

---

Published as a conference paper at ICLR 2022
Error Rate (%)
100
No Attack
77
Fawkes v0.3
75
Fawkes v1.0
53
LowKey
50
25
7
WebFace
Sabbacc-Adaptivy
Feature Extractor
Figure 2: Adaptive defenses break facial poisoning attacks. Existing attacks break standard WebFace model, but fail against model explicity trained on these attacks' perturbations.
Training a robust feature extractor. The model trainer begins by collecting a public dataset of unperturbed labeled faces Xpublic Y +- I (e.g., canonical dataset of celebrity faces), and calls the attack (as a black box) to obtain perturbed samples: adv Attack pXpublic).
As the model trainer has access to both unperturhed images and their corresponding perturbed versions for set of users, they can teach model to produce similar embeddings for unperturbed and perturbed pictures of the same user-bhereby encouraging the model to learn robust features. The hope then is that this robustness will generalize the perturbations applied other users' pictures.
We use the images of half of the FaceScrub users as the public labeled data (Xpublic Ypublic,, and use the Fawkes and LowKey attacks a black-box to obtain perturbed samples X-aavicc We robustly fine- tune the pre-trained WebFace feature extractor by adding a linear classifier head and then fine-tuning the entire model to minimize the cross-entropy loss on (Xpubli, Y and public Y After fine-tuning, the classifier head is discarded.
adv
The model trainer's adaptive strategy entails performing "robust data augmentation" for some users in the training set. We remark that this could also happen without explicit intervention from the model trainer. Indeed, some users are likely to have both perturbed and unperturbed pictures of themselves on the Web. (e.g., because they forgot perturb some pictures, or because another user uploaded them). Feature extractors trained on these pictures would then be encouraged to learn robust features.
This robust training approach differs from adversarial training (Szegedy et al., 2013: Madry et al., 2017). Adversarial training makes a model robust against an attack that depends on the model. In our case, the attack fixed (since the user to commit to it), so the model trainer's goal is much easier.
Results. As baseline, we first evaluate all attacks against a non-robust WebFace model. Figure shows that the attacks are effective in this setting. For users who poisoned their online pictures, the model's s error rate is 55-77% (as compared only 8% error for unprotected users).
We now evaluate the performance of our robustly fine-tuned feature extractor. We use the extractor obtain embeddings for the entire FaceScrub dataset, including the attacker's perturbed pictures (note that by keeping the users from X in the evaluation set, we favor the attacker as this only make the model's 's task harder). As shown Figure all attacks are rendered ineffective. The robust model's is error rate for users who attempted to attack the system is no higher than for the average user.
In Appendix C.2, we show that our feature extractor retains its robustness for an alternative facial recognition benchmark. Thus, our robust training did not somehow overfit to the FaceScrub dataset.
Attack detection. With the knowledge of users' attack tool, the model trainer can also build model detecr whether given image has been perturbed. accurate detector can be used filter out perturbed images, and retain only unperturbed images of a user (if such pictures exist online). Moreover, detecting an attack attempt could itself be a privacy concern (e.g., law enforcement might actively target users whose attack attempts are detected).
For each attack, we fine-tune a standard pre-trained ImageNet model to distinguish between the perturbed and clean (unperturbed) images of 25 random users. We then evaluate the model on the perturbed and clean images of 20 other users not seen during training. As we show in Appendix B.4, perturbed pictures the Fawkes and LowKey attacks. Note that the cost false positive is low, as this simply causes the model trainer discard some user's unperturbed image.
be detected with near near-perfect precision (99.8%) and recall (99.8%) for both

---

Published as a conference paper at ICLR 2022
3.4 OBLIVIOUS DEFENSES: TIME IS ALL YOU NEED
We now evaluate an even simpler "oblivious" defense strategy. wherein the model trainer waits for better facial recognition systems to be developed, and then retroactively applies such a system to pictures scraped in the past. To bypass this oblivious defense strategy, poisoning attack must fool not only today's 's models, but also all future models. Asymmetrically. these new models do not have to be robust to all attacks; they just have to resist the specific attack that was used on prior pictures.
Adversarial examples do not transfer across time. Recall that attacks such as Fakwes and LowKey aim to build adver- sarial examples that transfer to the facial recognition model. The question of whether adversarial examples transfer to future models has received little attention. We demonstrate that they do not. We briefly depart from facial recognition to consider standard vision task: ImageNet. The availability of a large num- ber of pre-trained ImageNet models allows to easily test how well adversarial examples created a few years ago transfer to today's models. Suppose that in 2015, user took an ensemble of state-of- oft models (at the time) (mme>-GooLeNNe VGG-16, Inception-v3 and ResNet500 and generated adversarial exam- ples against this ensemble using PGD (Madry et al., 2017). This setup mimics the attack used by LowKey (Cherepanova et al., 2021a). Figure 3 shows that the attack transfers well to contemporary models, but becomes near-ineffective for later models. Details on this experiment are in Appendix B.5.
ImageNet Top5 Error
100
Transfer Attack
80-
No Attack
40
20
2015
2016
2017 2018 2019 2020 2021 Year
Figure 3: Transferability of ad- versarial examples over time. Each point is a model evaluated the clean test set (red) and per- turbed set (blue).
Coming back to data poisoning, our thesis is that atlucks designed to transfer to today's 's models will inevitubly decline as better models are developed. Below, we provide evidence for this thesis.
Results. Figure 4 shows the performance of the Fawkes attack against variety of feature extractors (each model was trained without knowledge of the attack). We order the models chronologically by the time at which all components required to train each model were publicly available.
We first observe that the original Fawkes v0.3 attack is completely ineffective. The only model for which it substantially increases the error rate is the WebFace model, which is the surrogate model that the attack explicitly targets. Thus, this attack simply fails at transferring to other feature extractors.
Fawkes' attack was updated in version 1.0 (Shan et al., 2021) to target the more recent C feature extractor. The new version of Fawkes works perfectly against this specific feature extractor (error rate of 100%), and transfers to other canonical feature extractors such as VGG-Face, FaceNet and ArcFace. However, the Fawkes v1.0 attack fails against more recent extractors, such as MagFace and our fine-tuned CLIP model- mheel-thereby giving credence to our thesis.
Interestingly, we find that even if a model trainer uses a model that is vulnerable to Fawkes' new v1.0 attack, the 500,000 users who downloaded the original v0.3 attack cannot "regain" their privacy by switching to the updated attack. To illustrate, we show that half a user's online pictures were originally poisoned with Fawkes v0.3, and half are later poisoned with Fawkes v1.0, the attack fails to break the CeleblM M model (6% error rate). Thus, once the model trainer adopts a model that resists past attacks, the protection for pictures perturbed in the past lost regardless of future attacks.
Figure 5 evaluates LowKey against the same set of feature extractors. LowKey fairs better than Fawkes and transfers to all canonical facial recognition models including MagFace. However, LowKey fails to break our fine-tuned CLIP model. While CLIP was not trained for facial recognition. it can extract rich facial features (Goh et al., 2021) and is remarkably robust to image perturbations (Radford et al., 2021). By fine-tuning CLIP on facial data², we achieve clean accuracy comparable to facial recognition models such as WebFace or VGG-Face, but with much higher robustness to attacks.
This experiment shows how developments in computer vision can break poisoning attacks that were developed before those techniques were discovered. Critically, note that CLIP is still vulnerable to adversarial examples. Thus, one could develop a new poisoning attack that explicitly targets CLIP,
Following (Wotsmaan et al., 2021). we improved our fine-tuned model's 's robustness by linearly interpolating the weights of the fine-tuned model with the weights of the original CLIP model. Details are in Appendix B.5.
7

---

Published as a conference paper at ICLR 2022
Error Rate (%)
100
100
No Attack
Fawkes v0.3
75
73
Fawkes v1.0
63
53
50% v0.3, 50% v1.0
49
50
26
25
13
17
8
8
17
3
6
11
16
FaceNet
WebFace
VGG-Tace
CelebIM
ArcFace
MagFace
CLIP
Feature Extractor
Figure 4: Oblivious defenses break Fawkes. Fawkes v0.3 does not transfer to any facial recognition model that it does not explicitly target. Fawkes v v1.0 fares better, but fails against new models such MagFace CLIP. Moreover, user that perturbs half their pictures with the original weak v0.3 attack and then switches to the stronger v1.0 attack cannot reclaim their privacy.
Error Rate (%)
100
90
97
96
No Attack LowKey
74
77
75
50
25
8 16
5
S
3
FaceNet
Weblaace
VGG-Tace
Celeb1M
ArcTaaee
MagiTaee
CLIP
Feature Extractor
Figure Oblivious defenses can break LowKey. The attack transfers well to canonical facial recognition models, but fails to transfer to our fine-tuned CL.IP model.
and transfers to all models we consider. Yet, this attack will in turn fail when yet newer models and techniques discovered. There is thus arms-race here: the user always has commit to attack, and the model trainer later gets to apply newer and better models retroactively.
3.5 INCREASING ROBUSTNESS WITHOUT DEGRADING UTILITY
have shown how model trainer can adaptively train a robust feature extractor to resist known attacks, wait for new facial recognition models that robust to past attacks.
potential caveat of these approaches is that increased robustness may come at a cost accu- racy (Tsipras et al., 2018). For example, our CLIP model is much more robust than other facial recognition models, but its clean accuracy slightly below the best models. A model trainer might thus be reluctant deploy a more robust model if only small minority of users trying attack the system. We now show how a model trainer can combine a highly accurate model with highly robust model obtain the best of both worlds. We consider potential approaches:
Top2: If a facial recognition system's results are processed by a human (e.g., the police searching for match on suspect), then the system could simply run both models and return two candidate labels. The system could also run the robust model only when the more accurate model fails find match (which the human operator can check by visual inspection).
Confidence thresholding: To automate the above process, the system can first run the most accurate model, and check the model's confidence (the embedding similarity between the target picture and its nearest match). If the confidence is below a threshold, the system runs the robust model instead. We set the threshold that 2% of clean images
through both models.*
In Figure 6 we evaluate these two approaches for a facial recognition system that combines MagFace and more robust model (either adaptive feature extractor, the system's 's clean accuracy matches or exceeds that of MagFace, while retaining high robustness. Note that the MagFace model alone achieves 96.6% top-1 accuracy and 96.8% top-2 accuracy. Thus, remarkably, the top-2 accuracy obtained by outputting MagFace's top prediction and the robust model's 's top prediction is much better than outputting MagFace's top two predictions. This shows that the more robust models make very different mistakes than a standard facial recognition model.
fine-tumed CLIP). In both cases,
*Confidence thresholding does lead short arms-race, that the attacker nevertheless loses (see Appendix D).
8

---

Published as a conference paper at ICLR 2022
Error Rate (%)
100-
LowKey Adaptive+
Lowkey MagFace+CLIP
Error Rate (%)
100
Fawkes + CLIP
Fawkes + MagFace
LowKey CLIP
75
75-
* LowKey - MagFace
50
50
25
16
17
25
7
7
Top2
Confidence Thresholding (clean error rate=3.3%)
(clean error rate=0.3%%
Unprotected Images
Figure 6: A facial recognition system els to achieve state-of-the-an accuracy and robustness. (Left) A system that runs MagFace and robust model in parallel has high top-2 accuracy and robustness; (Right) A system that runs the robust model if MagFace makes non- confident prediction has high top-1 accuracy and robustness.
two mod-
Figure 7: Poisoning attacks fail when few images are unpro- tected. Uploading single unpro- tected image online can significantly reduce user's protection rate.
4 DISCUSSION
Formal security games. The motivation for systems such as Fawkes and LowKey is that adversarial examples can be used for poisoning and that these attacks transfer different models. Yet, the unwritten assumption that these attacks also transfer to future (or adaptive) models does not hold. As beautifully argued by Gilmer et al. (2018), adversarial machine learning research often ignores important questions about the order in which parties interact, whether they can adapt, and how the game dynamies evolve over time. Defining formal security games, as we did in Section 2.2, is a useful way to reason about these questions, and we encourage future work to adopt this practice.
Limitations. Our evaluation uses curated datasets and a single attacking user. This may facilitate the model trainer's task compared to real applications. Yet, we also favor the attacker by measuring model's s top-1 accuracy, while a top-k match may work in some settings (e.g.. a list of candidates may suffice for the police to identify suspect). Moreover, since all images in experiments are pre- aligned and fixed size, the attack does not have to transfer to an unknown pre-pri pre-processing pipeline. Our main takeaway- that the defender has the upper hand-is oblivious these experimental details.
The game is already lost. Taking a step back, we argue that even a "perfect" poisoning attack that works for all future models cannot save users' privacy. Indeed, many users already have unperturbed pictures online, which can be matched against new pictures for many years into the future.
Our experiments conducted in the attacker-favorable setting where all of user's training pictures are perturbed. The presence of unperturbed training pictures significantly weakens the efficacy of poisoning attacks (Shan al., 2020; Evtimov al., 2020). As we show in Figure 7, user that uploads a single unperturbed picture (with all other pictures perturbed) already breaks current attacks.
Prior work recognized this issue and proposed collaborative poisoning attacks sure (Shan et al., 2020; Evtimov et al., 2020). However, such attacks are futile for users whose unperturbed pictures are already online. There is simple retroactive defense strategy for the model trainer: collect only pictures that were posted to the Web before the first face face-poisoning attacks were released and match future pictures against these. Thus, for most users, the point time where even a perfect poisoning attack would have stood chance of saving their privacy is long gone.
countermea-
5 CONCLUSION
Our work has demonstrated that poisoning attacks will not save users from large-scale facial recog- nition models trained on Web-scraped pictures. The initial motivation for these attacks is based on the premise that poisoning attacks can give rise to an "arms race", where better attacks can counteract improved defenses. have shown that no such arms race can exist, as the model trainer can retroactively apply new models (obtained obliviously or adaptively) to pictures produced by past attacks. To at least counteract an oblivious model trainer, users would have to presume that no significant change will be made facial recognition models in the coming years. Given the current pace of progress in the field, this assumption unlikely to hold. Thus, we argue that legislative rather than technological solutions needed to counteract privacy-invasive facial recognition systems.

---

Published as a conference paper at ICLR 2022
Ethics Statement. Our work uncovers critical limitations of popular systems (Hill, 2020b; Marks, 2020; Heaven, 2021) that users turn to in the hope of protecting their privacy. In the spirit of responsi- ble disclosure, we have informed the authors of Fawkes (Shan et al., 2020) and LowKey (Cherepanova et al., 2021a) a) of our results. We also mitigate the harm caused by our work by not targeting any real users of these systems.
As our experiments with oblivious defenses show, model trainers will obtain the upper hand regardless of the existence of our paper, merely by adopting more advanced facial recognition models. We thus view our work as a net positive to users' privacy, as it highlights the false sense of security that existing attacks provide.
Ultimately, we believe that the only viable course of action for posting pictures online, or support policy and legislation that restrict the use of facial recogni- tion (Singer, 2018; Weise & Singer, 2020; Winder, 2020).
users is to avoid
Reproducibility Statement. Our experiments rely on publicly available datasets and models, which are clearly referenced the main text and described in more detail in Appendix B. We will release source code to reproduce all our experiments.
REFERENCES
Kieran Browne, Ben Swift, and Terhi Nurmikko-Fuller. Camera adversaria. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-9,2020 2020.
Jingyi Cao, Bo Liu, Yunqian Wen, Rong Xie, and Li Song. Personalized and invertible face de- identification by disentangled identity information manipulation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 3334-3342, 2021.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 67-74. IEEE, 2018.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin. Shuicheng Yan, and Jiashi Feng. Dual path networks. arXiv preprint arXiv: 1707.01629, 2017.
Valeriaa Cherepanova, Micah Goldblum, Harrison Foley, Shiyuan Duan, John Dickerson, Gavin Taylor, and Tom Goldstein. LowKey: Leveraging adversarial attacks to protect social media users from facial recognition. In International Conference Learning Representations (ICLR), 2021a.
Valeriia Cherepanova, Micah Goldblum, Harrison Foley, Shiyuan Duan, John Dickerson, Gavin Taylor, and Tom Goldstein. LowKey: Prevent your images from being used to track you. https://owkkyyuumicss www.yy..maccccceiie...... 2021b. Accessed: 2021-05-15.
Thomas Cilloni, Wei Wang, Charles Walter, and Charles Fleming. Preventing personal data theft in images with adversarial ML. arXiv preprint arXiv:2010.1022 2020.
Debayan Deb, Jianbang Zhang, and Anil K Jain. Advfaces: Adversarial face synthesis. In 2020 IEEE International Joint Conference on Biometrics (IJC)), pp- 1-10. IEEE, 2019.
Debayan Deb. Xiaoming Liu, and Anil K Jain. FaceGuard: A self- defense against adversarial face images. arXiv preprint arX1v:2011.14218. 2020.
J Deng, J Guo, and Zafeiriou. Areface: Additive angular margin loss for deep face recognition.
сorr. arXiv preprini arXiv: 1801.0768. 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
Ivan Evtimov, Pascal Sturmfels, and Tadayoshi Kohno. FoggySight: scheme for facial lookup privacy. arXiv preprint wrXiv:20/2.08588 2020.
10

---

Published as a conference paper at ICLR 2022
Oran Gafni, Lior Wolf, and Yaniv Taigman. Live face de-identification in video. In Proceedings of the IEEE/CVF Interaational Conference on Computer Vision, pp. 9378-9387, 2019.
Chuhan Gao, Varun Chandrasekaran, Kassem Fawaz, and Somesh Jha. Face-off: Adversarial face obfuscation, arXiv preprint arXiv:2003.08861, 2020.
Justin Gilmer, Ryan P Adams, lan Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research. arXiv preprint TFXXvv8800006733 2018.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.239 aediistil1.00330 https://distill.puu22222 /multimodal-neurmeem
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Data security for machine learning: Data poisoning. backdoor attacks, and defenses. arXiv preprini arXiv:2012. 10544, 2020.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-lm: A dataset and benchmark for large-scale face recognition. In European conference on computer vision, pp. 87-102. Springer, 2016.
Drew Harwell. This facial recognition website turn anyone into cop stalker. The Washington Post, May 2021. URL /Bw.tttpp:://w.wwaaaaittttenoee 2021/05/ 14 htap://///aaaaeee-aaaaaaoooooooo
com. http:///ppppsc.cocttccccoooo//// technolog// secreec.....
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Will Douglas Heaven. How stop Al from recognizing your face in selfies. MIT Technology Review, May 2021. URL
srtorpaa--ccgggiiiooooooo--eeeee recognition-clearvie.
Kashmir Hill. The secretive company that might end privacy we know it. New York Times, Jan
http::/www...aaa./////////////aa 2020a. URL wtw.--rrooooooiioittt...... privaggy-accaaa--eemooii.t......
faciall
Kashmir Hill. This tool could protect your photos from facial recognition. The New York Times,
Aug 2020b. URL /ttp///p///////o///sssssss////// http://ww...aaaccoo///////////// fawkes- tool-protects-phooee
from- -
Jie Hu, Li Shen, and Gang Sun. Sgueezz-and-excitaee networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, July 2021. URL https: http://diooooo///////
Stepan Komkov and Aleksandr Petiushko. AdvHat: Real-world adversarial attack on ArcFace face ID system. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 819-826.
IEEE, 2021.
Bogdan Kulynych, Rebekah Overdorf, Carmela Troncoso, and Seda Gürses. POTs: protective optimization technologics. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 177-188. 2020.
Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pp. 365-372. IEFE, 2009.
11

---

Published as a conference paper at ICLR 2022
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 510-519, 2019.
Haibin Ling, Stefano Soatto, Narayanan Ramanathan, and David W. Jacobs. Face verification across age progression using discriminative methods. IEEE Transactions on Information Forensics and Security, 5(1):82-91, 2010. doi: 10.1 119911FF.2009999 2009220877.
Beryl Lipton. Records Clearview AI reveal new info on police use. Jan 2020. URL /ttch//we//000//////////cceeeee facial- www.reeenoooerroooooooooo httpp/ocw..ooooeeeeeerrcee//////
https::/ww..mukk:rt http://wwwwuuccccccoooooaaaa/a
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rXiv:1706.0000 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 181-196, 2018.
Paul Marks. Blocking facial recognition. Communications of the ACM, Jun 2020. URL https: / ///cacm.acm.org//eewee.. /25/rr/24444544 401-8844-bloci-gg-- facial racoigg-gtacil---rrtttttotoooooo fulltext.
Maxim Maximov, Ismail Elezi, and Laura Leal-Taixé. CIAGAN: Conditional identity anonymization
generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5447-5456, 2020.
Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. MagFace: A universal representation for face recognition and quality assessment. In CVPR, 2021.
Paul Mozur and Aaron Krolik. surveillance net blankets China's cities, giving police vast powers. New York Times, Dec 2019. URL http:a/s/www.nytineesscoo/////// wttpll/gg//chiannsueeeelelle
Elaine M Newton, Latanya Sweeney, and Bradley Malin. Preserving privacy by de-identifying face images. IEEE transactions on Knowledge and Data Engineering, 17(2):232-244 2005.
Hong- Wei Ng and Stefan Winkler. data-driven approach to cleaning large face datasets. In 2014
IEEE international conference image processing (ICIP), pp. 343-347, IEEE, 2014.
Seong Joon Oh, Mario Fritz, and Bert Schiele. Adversarial image perturbation for privacy protection game theory perspective. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 1491-1500. IEEE, 2017.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: 07277. 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 103.00003000020. 2021.
Arezoo Rajabi, Rakesh B Bobba, Mike Rosulek, Charles Wright, and Wu-chi Feng. On the (im) practicality of adversarial perturbation for image privacy. Proceedings on Privacy Enhancing Technologies, 2021.
Deepak Babu Sam, Skand Vishwanath Peri, Mukuntha Narayanan Sundararaman, Amogh Kamath, and Venkatesh Babu. Locate, size and count: Accurately resolving people in dense crowds via detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chieh Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4510 2018
12

---

Published as a conference paper at ICLR 2022
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recogmition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. Fawkes: Protecting privacy against unauthorized deep learning models. In 29th {USENIX} Security Symposium ( (UUSENIX) Security 20), pp. 1589-1604, 2020.
Shawn Shan, Emily Wenger, Jiayun Zhang. Huiying Li, Haitao Zheng, and Ben Y Zhao. Image 2021. Accessed: 2021-
"cloaking" for personal privacy. http://awwlabbb...cccccc....../a 05-15.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks state-of-thee offttheart face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communications security, pp. 1528-1540, 2016.
Natasha Singer. Microsoft urges congress to regulate of facial recognition. New York Times, Jul 2018. URL https:/// www.wnytimes.co...
07 /1/07//1///0 1/3//t/chooogyyyy /m/ ott.ooooooo//mmm.oooofo.ttt. crosoft-
Qianru Sun, Liqian Ma. Scong Joon Oh, Luc Van Gool, Bernt Schiele, and Mario Fritz. Natural and effective obfuscation head inpainting. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pp. 5050-5059, 2018a.
Qianru Sun, Ayush Tewari, Weipeng Xu, Mario Fritz, Christian Theobalt, and Bernt Schiele. A hybrid model for identity obfuscation by face replacement. In Proceedings of the European Conference Computer Vision (ECCV), pp. 553 553-569, 2018b.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6/99 2013.
Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras: adversarial patches to attack person detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.08.: 083447 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be at odds with accuracy. arXiv preprint arXiv: 1215,, 2018
James Vincent. Legal chatbot firm DoNotPay adds anti- recognitionn filters its suite of handy tools. Verge, Apr 2021. URL http://///w..ttrrrrrree/./////22 filter.
Q Wang. B Wu, P Zhu, Li, W Zuo, and Q Hu. ECA-Net: Efficient channel attention for deep convolutional neural networks, 2020 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.
Karen Weise and Natasha Singer. Amazon pauses police use its facial recognition software. New http://wwy..i..ccooom/////////// 08/ haa.ac--fcccalreeaeooooooo-aaaae -
York Times, June 2020. URL httptt:tts...ttttt omazon-facian
Davey Winder. Police facial recognition use unlawful-UKKK court of appeal makes landmark ruling. Forbes, Aug 2020. URL
-aaaaakkkkkllnnnn ruling.
Queenie Wong. Facebook built facial recognition app for employees. CNET, Nov 2019.
hattps::/www.cnet.com//// news///ww////ee facebook-built- hatp-tt-a-aaiill-eeoniiiic..... agp-ffor-eplloyeess.
13

---

Published as a conference paper at ICLR 2022
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning zero-shot models, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen, Pin- Yu Chen, Yanzhi Wang, and Xue Lin. Adversarial T-shirt! evading person detectors physical world. In European Conference Computer Vision, pp. 665-681. Springer, 2020.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv: 00546, 2019.
Xiao Yang, Yinpeng Dong, Tianyu Pang, Jun Zhu, and Hang Su. Towards privacy protection by generating adversarial identity masks. arXiv preprint 17Xiv:2003.06814, 2020.
Dong Yi, Zhen Lei, Shengeai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv: 141 1. 7923, 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 165 2016.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, al. ResNeSt: Split-attention networks. arXiv preprint 2004 2020.
14

---

Published as a conference paper at ICLR 2022
A
SECURITY GAMES
Attacker (User)
Defender (Model Trainer)
Áttacker (User)
Defender (Model Trainer)
Nadv 8,3
Training.
Training()).
X. Y
X,
Xadv X
add X to N
Xadv. Y
adv +- (x)
f t- Erain(XKaa:
Xudvi
add Xadv to Nadv Y y
train, (aud 3)
Evaluation
Evaluation
2,y +
13;
g f(z)
5 f(x)
Attacker wins if # and O(X, X.ev)= =1
Attacker wins in round if # and O Knav) =
attacker creates clean-labeled poisoned training set (Xadv. Y) and the defender trains a model f. which is evaluated on unperturbed inputs The attacker wins if misclas- sifies and the poisoned data Xadv "close" the original data (according to an oracle O).
(a) Game 1: Static game.
(b) Game Dynamic game. In each round 1, the attacker sends new poisoned data to the defender. The defender may train on all the training data (Nadv, )) it collected over prior rounds. The strategics of the attacker and defender may change between rounds.
Figure 8: Security games for training-only clean-label poisoning attacks.
B EXPERIMENTAL DETAILS
B. GENERATION OF PERTURBED PICTURES.
For the experiments in Section we generate perturbed images for random FaceSerub (Ng & Winkler, 2014 4) users using Fawkes (either version 0.3 in "high" mode or the most recent version 1.0 in "high" m) and LowKey.?
We use the official pre-all rre-aliged and extracted faces from the FaceScrub dataset, and thus disable the automatic face-detection routines in both Fawkes and LowKey. For LowKey, we additionally resize all images to 112 112 pixels as we found the attack to perform best in this regime.
B.2 ATTACK AND MODEL TRAINING SETUP
In each of our experiments, randomly choose attacker. We perturb 100% of the training pictures of that user (70% of all pictures) with the chosen attack (Fawkes vO.3, Fawkes v1.0, LowKey). The training set for the model trainer contains these perturbed pictures, as well as the training pictures of all other 529 FaceScrub users.
from the 530 FaceSerub identities to be the
For evaluation, we extract features using fixed pre-trained feature extractor and assign test points the same class as the closest training point in feature space (under the Euclidean distance).
The FaceNet. VGG-Face and ArcFace models are taken from the DeepFace library. The WebFace and CelebIM models are taken from the released Fawkes tool (Shan et al., 2021). The MagFace model taken from the official repository." 9 The CLIP model taken from OpenAl's official repository and fine-tuned using the procedure described in Appendix B.3.
htt.ss//ggttbbbcco//srrgg/////ee
"httess//githbbbccc//gnee/////
15

---

Published as a conference paper at ICLR 2022
To report the protection rate conferred by attack (a.k.a. the model's error rate), compute the model's error rate on the chosen user's 's unprotected test pictures. We then average these error rates across 20 experiments, each with different random attacking user.
B.3 FINE-TUNING CLIP FOR FACIAL RECOGNITION
OpenAT's pre-trained CLIP model achieves moderate accuracy as facial recognition feature extractor. With the pre-trained ViT-32 model, nearest neighbor classifier on extracted face embeddings achieves 83% clean accuracy using the FaceScrub dataset. While this accuracy is far below that of state-of- the-art models such as MagFace, CLIP's unique robustness to various image perturbations (Radford al., 2021) acts strong defense against data poisoning: the model's 's error rate under the LowKey attack is only 29%.
To improve CLIP's performance for facial recognition, fine-tune the model recognition datasets, CASIA-WebFace (Yi et al., 2014) and VGG-Face2 (Schroff et al., 2015). For simplicity, fine-tune CLIP using the same contrastive image-text loss used for standard CLIP training. That is, we associate each face image with a text label "A photograph of user #X.", where X is a unique integer corresponding each user in the dataset. (Alternatively, fine-tuning CLIP using loss function tailored to facial recognition such as ArcFace or MagFace. Here, we were interested in evaluating a training procedure that is sufficiently different from existing facial recognition pipelines while still achieving strong accuracy.)
canonical facial
could consider
We fine-tune CLIP's pre-trained ViT-32 model on CASIA- WebFFac and VGG-Face2 for 50 epochs using an open source implementation of CLIP training (Ilharco et al., 2021). The resulting model achieves high accuracy on FaceScrub (>95%), but loses most of CLIP's robustness against poisoning attacks. Similar behavior has been observed when fine-tuning CLIP for other tasks (Wortsman et al., 2021). Remarkably, Wortsman et al. (2021) recently showed that by interpolating the weights of the original CLIP model and the fine-tuned model, it is often possible to achieve high accuracy on the fine-tuned task while preserving CLIP's strong robustness properties. Concretely, given the pre-trained CLIP model with weights fCLIP, and the fine-tuned model with weights Ouned, we build model with weights
a Gruned + (1-a) a) OCLIP
where [0, is interpolation parameter. We find that with 0.6, obtain model that achieves both high accuracy on FaceScrub (92%) and high robustness against poisoning attacks (error rate of 16% against LowKey-half half the initial error rate of the baseline CLIP model).
B.4 ADAPTIVE DEFENSES
Data generation using public attacks. To train a robust feature extractor, we first generate per- turbed pictures for many FaceScrub users using different attacks:11
Table 1: Number of FaceScrub users whose images are perturbed for each attack. Both the perturbed and unperturbed images of these users are used during robust training.
Attack
Number of users
Fawkes v0.3
265
Fawkes v1.0
50
LowKey
150
Note that this corresponds 265 users in total (i.e., the for the Fawkes v1.0 and LowKey attacks are subset of the users for the Fawkes v0.3 attack). The public dataset public consists of the original pictures of these 265 users, and the perturbed dataset public consists of all the perturbed pictures (across all attacks) of these users.
adv
We started this project by experimenting with Fawkes v0.3 and thus have generated many more perturbed pictures for that attack than for the newer attacks.
16

---

Published as a conference paper at ICLR 2022
Robust model training setup. For the model trainer, Fawkes that the original authors adversarially trained on a dataset different than FaceScrub (Shan et al., 2020). We first fine-tune this feature extractor on the data from the 265 chosen public users. That is, we add a 265-class linear layer on top of the feature extractor, and fine-tune the entire model end-to-end for 500 steps with batch size 32. To evaluate this robust feature extractor, we pick an attacking user at random (not one of the 265 public users), and build a training set consisting of the perturbed pictures of that user, and the unperturbed pictures of all other 529 users. We then extract features from this training set using the robust model and perform nearest neighbors classification.
the WebFace feature extractor from
Attack detection. To evaluate the detectability of perturbed pictures. choose 45 users and generate perturbations using Fawkes v1.0 (in "low", "mid" and "high" protection modes) and LowKey. We use 25 users during training and 20 users during evaluation. For LowKey, we build training dataset containing all unperturbed and perturbed pictures of the 25 users. For Fawkes, we do the same but split user's perturbed pictures equally among its three attack strengths ("low", "mid" and "high"). Higher attack strengths introduce larger perturbations that provide more protection.
We then fine-tune pre-trained MobileNetv2 model (Sandler al., 2018) on the binary classification task of predicting whether a picture is perturbed. We fine-tune the model for 3 epochs using Adam with learning rate 1) = 5 10 5, The model is evaluated by its accuracy on the unperturbed and perturbed pictures of the 20 test users (each user has an equal number of perturbed and unperturbed pictures since we evaluate the Fawkes modes separately).
In Table 2 below, we report the detection accuracy, as well as precision, recall and AUC scores.
Table 2:
Performance of a model trained to detect perturbed images. Detection performance very high across all attacks even when smaller perturbations are used (i.e. Fawkes "low" and "mid").
Attack
Detection Aceuracy
Precision
Recall
AUC
Fawkes high
99.8%
99.8%
99.8%
99.99%
Fawkes mid
99.6%
99.8%
99.4%
99.91%
Fawkes low
99.1 1%.1%
99.8%
98.4%
99.72%
LowKey
99.8%
99.8%
99.8%
99.97%
Finally, we show that a detector that trained on system (i.e., Fawkes LowKey) transfers to the other. That is, we take the detector model that was trained on 20 users perturbed with one attack, and evaluate whether this detector also succeeds in detecting the perturbations from the other attack.
Table 3: Performance of model trained to detect perturbed images one attack (source) when evaluated on another attack (destination).
Source > Destination
Detection Accuracy
Precision
Recall
AUC
Fawkes LowKey
99.4%
99.0% 100%
99.8% 43.9%
99.59% 98.3%
LowKey - Fawkes high
71.9%
B.5 OBLIVIOUS DEFENSES
To generate Figure 4, we perturb one user's training pictures using either the Fawkes v0.3 attack, the Fawkes v1.0 attack, or a joint attack that perturbs half the user's training pictures with either of the two Fawkes versions. We then use each pre-trained feature extractor extract embeddings for the entire training set of 530 users, and evaluate the performance of I -nearesst neighbor classifier on the user's unprotected test images. To generate Figure 5, we repeat the same process with the LowKey attack.
Transferability of adversarial examples across time. For the experiment in Figure 3 in Sec- tion 3.4, we evaluate the transferability of adversarial examples crafted using an ensemble of ImageNet models from 2015 to future ImageNet models.
17

---

Published as a conference paper at ICLR 2022
We create adversarial examples for ensemble of GoogLeNet, VGG-16, Inception-v3 and ResNet- 50 models using the PGD attack of Madry et al. (2017) with a perturbation budget of = 16/255. The attack lowers the ensemble's top-5 accuracy from 93% to 0%.
We then transfer these adversarial examples to a variety models. All pre-trained models are taken from either pytorch/visiie For each model, we report the year the model
more recent ImageNet or
wi wightman http://tottttteeeoooeeeoee was originally proposed, and the top-1/top- accuracy ImageNet and on the transferred adversarial examples in Table 4
Table 4: Transferability of adversarial examples from ensemble of four models- GoogLeNet, VGG-16, Inception-v3, ResNet-500 future ImageNet models. Numbers in bold show the models that most robust the attack given point time.
Top-1 Acc
Top-5 Ace
Model
Year
Clean
Adv
Clean
Adv
ResNet-101 (He et al., 2016)
2015-12
76%
8%
93%
%
Wide-ResNet-100 (Zagoruyko & Komodakis, 2016)
2016-05
78%
14% %
95%
43%
Densenet-121 (Huang et al., 2017)
2016-08
77%
11 11%
94%
39%
ResNeXt-101_32x66 (Xie al., 2017)
2016-11
79%
15%
95%
44%
Dual Path Networks-107 (Chen et al., 2017)
2017-07
80%
25%
95%
58%
SE-ResNeX:101 32x4d (Hu al., 2018)
2017-09
80%
23%
95%
54%
IG-ResNeXt-101_333 (Mahajan et al.. 2018)
2018-05
85%
38%
97%
70%
WSI.-ResNext-001 10x3xxx8d 33xx48d (Mahajan al., 2018)
2018-05
85%
49%
98%
79 %
SK-ResNeXt-50__3e 32x4d (Li al., 2019)
2019-03
79%
29%
94%
61 %
SSL-ResNeXt--011 132x16d (Yalniz et al., 2019)
2019-05
82%
26%
97%
61%
1WSL-RResNeX(-0011112 16d (Yalniz et al., 2019)
2019-05
82%
44%
96%
76%
ECA-ResNet-101s (Wang et al., 2020)
2019-10
81%
34%
96%
66%
ResNeSt-101 (Zhang al., 2020)
2020-04
79%
38%
96%
70%
ViT-16 (Dosovitskiy et al., 2020)
2020-10
83%
57% %
97%
86%
C ADDITIONAL EXPERIMENTS
C.I SUPERVISED FACTAL. RECOGNITION CLASSIFIERS
In Section 3, we evaluated a standard facial recognition pipeline based on nearest neighbor search on top of facial embeddings. For completeness, we evaluate two additional facial recognition setups considered by Fawkes (Shan et al., 2020), where a face classifier is trained a supervised fashion on a labeled dataset of users' photos.
Specifically, given a pre-trained feature extractor g(x), we add a linear classifier head on top of and then train the classifier to minimize the cross-entropy loss the full FaceScrub dataset (i.e., the training data of all 530 users). We consider two approaches:
Linear: the weights of the feature extractor g are frozen and only the linear classification
head is tuned;
End end: the feature extractor and the linear classifier are jointly tuned on the training dataset.
Baseline. We first evaluate the baseline performance of the Fawkes (v1.0) and LowKey attacks for linear fine-tuning and end to end tuning. We also reproduce the results for nearest neighbor search for comparison. As shown in Figure 9a, the poisoning attacks more effective when the model trainer fine-tunes a linear classifier on top of fixed feature extractor (error rate 2 93%) instead of fine-tuning the entire classifier end-to-end, or performing nearest neighbor search in feature space
Pnttps:///i phtr:es://tttmbbcco//rrrrr///// Phttps:// http::/:ittttbccocoootttttttttta
18

---

Published as a conference paper at ICLR 2022
(error rates of 73-77%). That is, linear classifiers are unsurprisingly easier to poison given their lower capacity.
also replicate our experiments with an adaptive model from Section 3.3. When the model trainer uses linear fine-tuning, they the robust feature extractor described in Section 3.3 and fine-tune a linear classifier on top. When the model trainer fine-tunes a model end-to-end, we add pairs of public unperturbed and perturbed faces (X public Ypublic to the model's training set, and tune the entire classifier end to end.
Robust training.
adv
As shown Figure 9b, each of the three facial recognition approaches we consider can be made robust. In all cases, the user's protection rate (the test error rate on unperturbed pictures) similar the classifier's s average eπror rate for unprotected users.
Attack
Lowkey
No Attack
Fawkes
LowKey
100
94
93
73 74
73
Error Rate
50
25
10
8
7
7
Linear
End to end
NN
Linear
End to end
KN
(a) No Defense.
(h) Robust Training.
Figure 9: Adaptive defenses against Fawkes and LowKey. We report (a) the baseline performance (i.e. when no defense is used) for three training modes (Linear, End to end, Nearest neighbors); (b) the attack performance after robust training.
C.2 EXPERIMENTS ON PUBFIG
In this section, we replicate the results from Section on a different facial recognition dataset. We use curated subset of the PubFig dataset (Kumar et al., 2009), with over 11,000 pictures of 150 celebrities.
Figure 10 and Figure 11 show the protection rates of the Fawkes v0.3, Fawkes v1.0 and LowKey attacks against the various feature extractors we consider. Note that the adaptive feature extractor used here the same one in Section 3.3, which was trained without any data from PubFig.
Error Rate (%)
100
No Attack
7679
Fawkes v0.3
75
Fawkes v1.0
57
49
50% v0.3, 50% v1.0
50
25
15
18
19
12
21
26
14
16
8 121312
911
0
FaceNet
WebFace
VCG-Face
CelebIM
ArcFace
MagFace
CLIP
WebFace -Adaptive
Feature Extractor
Figure 10: Adaptive and oblivious defenses break Fawkes PubFig. As on FaceScrub in Figure 2 and Figure 4, the Fawkes v0.3 attack fails to transfer, and the Fawkes v1.0 attack fails against new models such as MagFace or CLIP as well as against adaptively trained model.
The results on PubFig are qualitatively similar as on FaceScrub:
The Fawkes v0.3 attack fails to transfer to models other than the WebFace model that the attack explicitly targets.
The Fawkes v1.0 attack transfers reasonably well older facial recognition models, but is ineffective against MagFace and CLIP.
The LowKey attack works well against all "traditional" facial recognition models, but fails against our fine-tuned CLIP model.
19

---

Published as a conference paper at ICLR 2022
Error Rate (%)
100
88
No Attack LowKey
78
75
67
63
72
50
19
18
12
10
8
8
9
9
1
FaceNet
WebFace
VGG-Face
CelebIM
ArcFace
MagFace
CLIP
WebFace +Adaptive
Feature Extractor
Figure 11: Adaptive and oblivious defenses break LowKey PubFig. As on FaceScrub in Figure 2 and Figure 5, the Low Key attacks transfers well to canonical pre-trained facial feature extractors, but fails against our CLIP model and against adaptively trained model.
All attacks are ineffective against the robust feature extractor.
In Figure 12, further replicate the experiment from Section 3.5 on building facial recognition system that combines state-of-the-ar accuracy and robustness. As on FaceScrub, a system that only returns confident predictions from the accurate model, and otherwise diverts more robust model, achieves strong accuracy and robustness. Note that system that solely uses the MagFace model achieves a clean error of 8.1 (top-1) 1) and 6.2% (top-2).
Error Rate (%)
100
LowKey MaLFaceCCLIP Lowkey - AddppicAA
75-
25
18
9
20
0-
Top2
Confidence Thresholding (clean error rate-3.4%)
(clean error rate- 1.0%)
Figure 12: Replication of Figure 6 on PubFig. (Left) A system that runs both MagFace and a robust model (either CLIP our adaptive model) in parallel achieves high top-2 accuracy and robustness; (Right) system that only runs the robust model when MagFace fails to make confident prediction has high top-1 accuracy and robustness.
D AN ARMS-RACE ON CONFIDENCE THRESHOLDING
In Section 3.5, we showed how build facial recognition system that combines a highly accurate model and a highly robust model. The system first runs the most accurate model. If the model's prediction has low confidence, the system instead runs the robust model.
The reason that confidence thresholding works is that Fawkes and LowKey are untargeted attacks. That is, each of the user's training pictures is perturbed to produce embeddings that are far from the clean embeddings. These perturbed embeddings will typically be far from all facial embeddings, and a model will thus not find a close match when evaluated on an unperturbed picture of the user.
A user could thus aim to circumvent the confidence thresholding system by switching to targeted attack. This actually requires that users collude: user A would perturb their pictures so as to match the clean embeddings of user B, so that unperturbed picture of user B would get mislabeled as user A with high confidence.
The model trainer can further counteract such attack. The model trainer first runs the target image through the most accurate model. If the model returns a confident match, the system further checks whether the returned match is perturbed image (by using attack detector described in Section 3.3 and Appendix B.4). this is the case, the system ignores the accurate model's prediction and runs the target image through the more robust model instead.
20

---

Published as a conference paper at ICLR 2022
To circumvent this adapted system, users would have not-only collude to create targeted attacks, but also ensure that these attacks fool the model trainer's 's detector. But since users have to commit to an attack before the model trainer decides defense strategy, users will always be on the losing end of this cat-and-mous game.
21


Reviews:
[{"review_id": "u78tEG0tZOS", "reviewer": "Reviewer_6oYM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nThis is an interesting paper. The evaluation criteria and the game setup was well conducted. A large number of feature extractors were evaluated showing the problem is consistent across multiple. It is especially interesting to see how even \"oblivious\" systems can just wait and see to get better. Figure 5 is shows how Fawkes deteriorates over time with new extractors causing it to fail. The paper is well-written and easy to read. Ethical concerns are well documented in ethics statement.\n\nWeaknesses\nThis is a strong statement \"Ultimately, we believe that the only viable course of action for privacy-conscious users is to avoid posting pictures online, or to support policy and legislation that restrict the use of facial recognition (Singer, 2018; Weise & Singer, 2020; Winder, 2020).\" and may not be a feasible solution to the problem, although supporting policy is certainly feasible. A more concrete solution to the problem would have strengthened the paper.\n\nMinor point: The paper argues that it is not an arms-race, however, it can still be considered an arms race with the constraint that the defender is in a better position to win. This is arguably just a semantic difference, and a minor point.", "summary_of_the_paper": "Paper presents an analysis of two systems for poisoning attacks. The paper shows that perturbing facial images does not offer long term security; future systems can still recognize the once perturbed image(s).", "summary_of_the_review": "The paper does a good job analyzing poisoning attacks. It shows that the methods fail over time and they do not generalize to future attacks. It is in interesting paper and has value for the community to consider longer term security measures for facial recognition privacy concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZrN0PJFTTL", "reviewer": "Reviewer_8c4m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n1. The paper is well written and easy to follow.\n2. The topic discussed in this paper is of significant importance as the rapid development of deep learning techniques nowadays also poses great threats to people's privacy, especially for the face data.\n3. Extensive experiments have been conducted to statistically ascertain the authors' claims.\n\n### Weaknesses:\nFrom technical point of view, there is less innovation in this paper. It is widely known that adversarial attacks, either obtained in white-box or black-box manners, can be effectively guarded by robust training or by adopting a different algorithm. The experimental results are within expectation and little technical insight is gained.\n", "summary_of_the_paper": "This paper points out that current data poisoning techniques cannot effiectively protect users privacy, i.e., face data, on the Internet. The authors have examined several strategies to enable modern face recognition models to defense attacks from widely used data poisoning methods. Experimental results suggest that those data poisoning attacks can be easily defensed by adaptively tuning the face recognition models or using more advanced algorithms which would be developed in the future. The main conclusion is that people should not rely on technical solutions to protect users privacy and legislation actions are what is actually needed.", "summary_of_the_review": "This paper has addressed an important problem that, data poisoning techniques are insufficient to protect people's privacy from modern development of face recognition models and it is of great importance to draw people's attention to the discussed topic, to potentially push any legislation actions instead of searching for technical solutions to protect users privacy. However, I think the technical contribution from this paper is limited as little new insight is provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OcPhDM_SSF4", "reviewer": "Reviewer_6s7m", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The insight of this paper is very useful for the proctecter of user face.\n- It is not clear how the oblivious trainer works. \n- There is no clear presentation about the poisoning details of Flawkes and LowKeys.\n- The technical novelty is very limited. Rather than fancying legislative alternative, a research paper needs to propose technical solution.\n", "summary_of_the_paper": "This paper studies the effect of data poisoning in face recognition and the relation to the defense techniques. Conventionally, the poisoned data will fail the face recognition models who is trained without defense strategy. Two solutions of defense are given: oblivious trainer and adaptive trainer. The claim is that, any existing poisoning methods cannot protect the privacy of users in the face images.", "summary_of_the_review": "Overall, this is a facial privacy analysis with insightful claims, but the presentation and the discussion is very confusing. Thus, we are not able to agree this argument whether is reasonable and solid. The final rating will depend on the authors’ feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8aYDqiVO3fY", "reviewer": "Reviewer_3JiV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- This paper defines the dynamic game scenario, which is more practical than the static game scenario used in the previous poison-based identity protection works\n- It proposes two defenses that can defeat the poison-based identity protection methods\n- The authors propose a strategy for the model trainer to defense from poisoning attacks without sacrifying accuracy\n- The authors give a good discussion on the flaws of the poison-based identity protection approach. The game is already lost, and users should not rely on the methods in that category to have a false sense of security.\n- This paper has a good amount of references\n\n### Weaknesses\n- In the adaptive defense, the authors assume access to the poisoning function, which may be unrealistic. The poisoning function should be secret for a better protection,\n- Obvious defense can break LowKey only once (Figure 6). There is no guarantee that robust models, like CLIP, will be popular in the future facial recognition models.\n\n### Not a weakness of the paper itself\nI agree with the section \"The game is already lost\": the poison-based identity protection approach is very unrealistic and has a low value.", "summary_of_the_paper": "Recent works propose to protect users from facial recognition by poisoning their images before uploading them to the Internet (called poisoning \"attacks\"). This paper reveals the flaws of these methods by designing two effective methods (called \"defenses\") to defeat that protection mechanism. The first approach is adaptive defense, in which the model trainer assumes to have access to the poisoning function as a black-box. He then can collect a clean facial image dataset, create perturbed and unperturbed versions, and finetune the poisoned face recognition model to learn robust features. The second approach, called obvious defense, relies on the fact that poisoned examples do not transfer well over time to newer models. Hence, the model trainer can wait to obtain a better face recognition model and properly safely finetune it on the perturbed images. Both methods successfully defeat two poisoning attack baselines, raising awareness on the inefficiency of the poisoning-based identity protection mechanism.", "summary_of_the_review": "This paper reveals the flaws of previous works that use poison attacks to protect user identity from face recognition systems. It has some good discussions and provides two methods to break those protections. There are still some concerns about the experiments and the significance of the previous works the paper is targeting. Hence, I give it a Borderline, slightly towards Acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper reveals that popular data poisoning systems, Fawkes and LowKey, fail to effectively protect user privacy in facial recognition. The methods to defend against poisoning attacks are quite simple---you can either adaptively tune the face recognition models or just wait for more advanced facial recognition systems. Given these “disappointed” findings from the technical solution side, this paper further argues that legislation may be the only viable solution to prevent abuses of facial recognition.

Overall, all the reviewers highly appreciate the comprehensive and rigorous evaluations provided in this paper and enjoy reading it. The biggest concern is raised by the Reviewer 6s7m, given this work fails to discuss/compare to previous works on Facial identity anonymizing and the technical contribution is incremental. During the discussion period, all other reviewers reach a consensus that 1) facial identity anonymizing is not relevant; and 2) this work make enough contributions and is worthy to be heard by the general community; the Reviewer 6s7m still hold the opposite opinion, but is okay for accepting this paper anyway. 

In the final version, the authors should include all the clarification provided in the discussion period.
Decision: Accept (Poster)