Data Poisoning Won’t Save You From Facial Recognition

1 INTRODUCTION . Facial recognition systems pose a serious threat to individual privacy . Various companies routinely scrape the Web for users ’ pictures to train large-scale facial recognition systems ( Hill , 2020a ; Harwell , 2021 ) , and then make these systems available to law enforcement agencies ( Lipton , 2020 ) or private individuals ( Harwell , 2021 ; Mozur & Krolik , 2019 ; Wong , 2019 ) . A growing body of work develops tools to allow users to fight back , using techniques from adversarial machine learning ( Sharif et al. , 2016 ; Oh et al. , 2017 ; Thys et al. , 2019 ; Kulynych et al. , 2020 ; Shan et al. , 2020 ; Evtimov et al. , 2020 ; Gao et al. , 2020 ; Xu et al. , 2020 ; Yang et al. , 2020 ; Komkov & Petiushko , 2021 ; Cherepanova et al. , 2021a ; Rajabi et al. , 2021 ; Browne et al. , 2020 ) . One approach taken by these tools allow users to perturb any picture before they post it online , so that a facial recognition model that trains on these pictures will become poisoned . The objective of these perturbed images is that when any unperturbed image is fed into the facial recognition model ( e.g. , a photo taken by a stalker , a security camera , or by the police ) , the model misidentifies the user . This research direction was popularized by Fawkes ( Shan et al. , 2020 ) , an academic image-poisoning system with 500,000+ downloads and covered by the New York Times ( Hill , 2020b ) , that promises “ strong protection against unauthorized [ facial recognition ] models ” . Following the success of Fawkes , similar systems have been proposed by academic ( Cherepanova et al. , 2021a ; Evtimov et al. , 2020 ) and commercial ( Vincent , 2021 ) parties . This paper shows that these systems ( and , in fact , any poisoning strategy ) can not protect users ’ privacy . Worse , we argue that these systems offer a false sense of security . There exists a class of privacy-conscious users who might have otherwise never uploaded their photos to the internet ; however who now might do so , under the false belief that data poisoning will protect their privacy . These users are now less private than they were before . Figure 1 shows an overview of our results . The reason these systems are not currently private , and can never be private , comes down to a fundamental asymmetry between Web users and the trainers of facial recognition models . Once a user commits to an attack and uploads a perturbed picture that gets scraped , this perturbation can no longer be changed . The model trainer , who acts second , then gets to choose their training strategy . As prior work lacks a formal security setup , we begin by defining a security game to capture this dynamic nature of poisoning attacks . We then introduce two powerful defense strategies that completely break two state-of-the-art poisoning attacks—Fawkes ( Shan et al. , 2020 ) and LowKey ( Cherepanova et al. , 2021a ) . In the first strategy , we show how to adapt the facial recognition training to work in the presence of poisoned images . Because image-perturbation systems are made publicly accessible to cater to a large user base ( Shan et al. , 2021 ; Cherepanova et al. , 2021b ) , we must assume facial recognition trainers are aware of these attack strategies . Our adaptive models fully circumvent these poisoning tools with just black-box access to the attack system . Worse , we find there exists an even simpler defensive strategy : model trainers can just wait for better facial recognition systems , which are no longer vulnerable to these particular poisoning attacks . That is , because existing poisoning attacks were only designed to prevent current face recognition tools from working , there is no reason to believe that future tools will be poisoned as well . Indeed , we show that the state-of-the-art poisoning attacks are already broken by new training techniques that appeared less than a year later . For example , Fawkes ( released in July 2020 ) is ineffective if the model trainer switches to a MagFace model ( Meng et al. , 2021 ) ( released in March 2021 ) , and LowKey ( released January 2021 ) is ineffective against a facial recognition model obtained by finetuning OpenAI ’ s CLIP model ( Radford et al. , 2021 ) ( also released January 2021 ) . We argue that poisoning attacks against facial recognition will not lead to an “ arms race ” , where new attacks can continuously counteract new defenses . Since the perturbation applied to a picture can not be changed once the picture is scraped , a successful poisoning attack has to remain effective against all future models , even models trained adaptively against the attack , or models that use new techniques discovered only after the attack . In light of this , we argue that users ’ only hope is a push for legislation that restricts the use of privacy-invasive facial recognition systems ( Singer , 2018 ; Weise & Singer , 2020 ; Winder , 2020 ) . 2 DATA POISONING FOR FACIAL RECOGNITION . 2.1 THREAT MODEL . We consider a setting where a user uploads pictures of themselves to an online service such as a social media platform . The user attempts to protect their pictures by adding perturbations that should be almost imperceptible to other people ( Szegedy et al. , 2013 ) . The user ’ s goal is that a model trained on their perturbed pictures will achieve low accuracy when classifying unperturbed pictures of the user ( Shan et al. , 2020 ; Cherepanova et al. , 2021a ; Yang et al. , 2020 ; Evtimov et al. , 2020 ) . A second party , the model trainer , scrapes the Web for pictures to train a large-scale facial recognition model ( capable of identifying a large number of users ) . We assume that the data scraped by the trainer is labeled , i.e. , all ( possibly perturbed ) images collected of a user can be assigned to the user ’ s identity . The trainer ’ s goal is to build a model that correctly recognizes users in future images . The trainer is active , i.e. , they continuously scrape new uploaded pictures at regular intervals . This setting corresponds to that of training-only clean-label poisoning attacks ( Shan et al. , 2020 ; Cherepanova et al. , 2021a ; Goldblum et al. , 2020 ; Evtimov et al. , 2020 ) . Keeping with the terminology of the data poisoning literature ( Goldblum et al. , 2020 ) , we refer to the user as the attacker and the trainer as the defender ( even though it is the trainer that aims to breach the user ’ s privacy ! ) . 2.2 POISONING ATTACK GAMES . We present a standard security game for training-only clean-label poisoning attacks in Figure 2a . We argue that this game fails to properly capture the threat model of our facial recognition scenario . In this game , the attacker first samples training data X , Y from a distribution D. In the setting of facial recognition , the data consists of facial images of users X , along with their identities Y . The attacker then applies an attack to get the perturbed data Xadv . The defender gets the perturbed labeled data ( Xadv , Y ) and trains a model f . The model f is evaluated on unperturbed inputs x from the distribution D. For a given test input x , the attacker wins the game if the perturbation of the training data is small ( as measured by an oracle O ( X , Xadv ) 7→ { 0 , 1 } ) , and if the model misclassifies x . The poisoning game in Figure 2a fails to capture an important facet of the facial recognition problem . The problem is not static : users continuously upload new pictures , and the model trainer actively scrapes them to update their model . Below , we introduce a dynamic version of the poisoning game , and show how a model trainer can use a retroactive defense strategy to win the game . In turn , we discuss how users and model trainers may adapt their strategies based on the other party ’ s actions . Dynamic poisoning attacks . To capture the dynamic nature of the facial recognition game , we define a generalized game for clean-label poisoning attacks in Figure 2b . The game now operates in rounds indexed by i ≥ 1 . In each round , the attacker perturbs new pictures and sends them to the defender . The strategies of the attacker and defender may change from one round to the next . The game in Figure 2b allows for the data distribution Di to change across rounds . Indeed , new users might begin uploading pictures , and users ’ faces may change over time . Yet , our thesis is that the main challenge faced by the user is precisely that the distribution of pictures of their own face changes little over time . For example , a facial recognition model trained on pictures of a user at 20 years old can reliably recognize pictures of the same user at 30 years old ( Ling et al. , 2010 ) . Thus , in each round the defender can reuse training data ( Xadv , Y ) collected in prior rounds . If the defender scrapes a user ’ s images , the perturbations applied to these images can not later be changed . Retroactive defenses . The observation above places a high burden on the attacker . Suppose that in round i , the defender discovers a training technique traini that is resilient to past poisoning attacks Attackj for j < i . Then , the defender can train their model solely on data ( Xadv , Y ) collected up to round j . From there on , the defender can trivially win the game by simply ignoring future training data ( until they find a defense against newer attacks as well ) . Thus , the attacker ’ s perturbations have to work against all future defenses , even those applied retroactively , for as long as the user ’ s facial features do not naturally change . By design , this retroactive defense does not lead to an “ arms race ” with future attacks . The defender applies newly discovered defenses to past pictures only . As we will show , this retroactive defense can even be instantiated by a fully oblivious model trainer , with no knowledge of users ’ attacks . The model trainer simply waits for a better facial recognition model to be developed , and then applies the model to pictures scraped before the new model was published . This oblivious strategy demonstrates the futility of preventing facial recognition with data poisoning , so long as progress in facial recognition models is expected to continue in the future . Adaptive defenses . A model trainer that does not want to wait for progress in facial recognition can exploit another source of asymmetry over users : adaptivity . In our setting , it is easier for the defender to adapt to the attacker , than vice-versa . Indeed , users must perturb their pictures before the model trainer scrapes them and feeds them to a secret training algorithm . As the trainer ’ s model f will likely be inaccessible to users , users will have no idea if their attack actually succeeded or not . In contrast , the users ’ attack strategy is likely public ( at least as a black-box ) to support users with minimal technical background . For example , Fawkes offers open-source software to perturb images ( Shan et al. , 2021 ) , and LowKey ( Cherepanova et al. , 2021b ) and DoNotPay ( Vincent , 2021 ) offer a Web API . The defender can thus assemble a dataset of perturbed images and use them to train a model . We call such a defender adaptive . A note on evasion and obfuscation attacks . The security games in Figure 2 assume that the training data is “ clean label ” ( i.e. , the user can still be identified in their pictures by other human users ) and that the evaluation data is unperturbed . This is the setting considered by Fawkes ( Shan et al. , 2020 ) and LowKey ( Cherepanova et al. , 2021a ) , where a user shares their pictures online , but the user can not control the pictures that are fed to the facial recognition model ( e.g. , pictures taken by a stalker , a security camera , or law enforcement ) . The game dynamics change if the user evades the model with adversarial examples , by modifying their facial appearance at test time ( Szegedy et al. , 2013 ; Sharif et al. , 2016 ; Thys et al. , 2019 ; Gao et al. , 2020 ; Cilloni et al. , 2020 ; Rajabi et al. , 2021 ; Oh et al. , 2017 ; Deb et al. , 2019 ; Browne et al. , 2020 ; Deb et al. , 2020 ) . Evasion attacks favor the attacker : the defender must commit to a defense and the attacker can adapt their strategy accordingly ( Tramer et al. , 2020 ) . Our setting and security game also do not capture face obfuscation or anonymization techniques ( Newton et al. , 2005 ; Sun et al. , 2018a ; b ; Sam et al. , 2020 ; Cao et al. , 2021 ; Maximov et al. , 2020 ; Gafni et al. , 2019 ) . These attacks remove or synthetically replace a user ’ s face , and thus fall outside of our threat model of clean-label poisoning attacks ( i.e. , the aim of these works is to remove identifying features from uploaded pictures , so that even a human user would fail to identify the user ) .

Summary: Recent works propose to protect users from facial recognition by poisoning their images before uploading them to the Internet (called poisoning "attacks"). This paper reveals the flaws of these methods by designing two effective methods (called "defenses") to defeat that protection mechanism. The first approach is adaptive defense, in which the model trainer assumes to have access to the poisoning function as a black-box. He then can collect a clean facial image dataset, create perturbed and unperturbed versions, and finetune the poisoned face recognition model to learn robust features. The second approach, called obvious defense, relies on the fact that poisoned examples do not transfer well over time to newer models. Hence, the model trainer can wait to obtain a better face recognition model and properly safely finetune it on the perturbed images. Both methods successfully defeat two poisoning attack baselines, raising awareness on the inefficiency of the poisoning-based identity protection mechanism.