Optimal Strategies Against Generative Attacks

1 INTRODUCTION . Generative models have attracted considerable attention since the introduction of Generative Adversarial Networks ( Goodfellow et al. , 2014a ) . Empirically , GANs have been shown to generate novel data instances that resemble those in the true distribution of the data . The success of GANs also comes with the risk that generative models will be used for attacking sensor-based security systems . One example is identity authentication systems , where an individual is identified via her images , and GANs might be able to generate such images to gain access ( Thies et al. , 2016 ) . Another is anomaly detection systems protecting critical infrastructure . As demonstrated by recent cyber-attacks ( notably the Stuxnet attack ) sensors of these systems can be hijacked , so that GANs can be used to generate “ normal ” looking activity while the actual system is being tampered with . The latter is , in fact , a new form of a man-in-the-middle attack . Our goal here is to construct a theoretical framework for studying the security risk arising from generative models and explore its practical implications . We begin with a simple key insight . If the attacker ( i.e. , the generative model ) has unlimited observations of the source it is trying to imitate , it will be able to fool any authenticator . On the other hand , if the attacker has access to fewer sensor observations than the number of fake observations it needs to generate , it seems intuitively clear that it can not always succeed ( as we indeed prove in Sec . 4 ) . Therefore , the optimal defense and attack strategies depend crucially on the amount of information available to the attacker and authenticator . Motivated by the above insight , we cast the authentication setting as a two-player maximin game ( authenticator vs. attacker ) where all observations are finite . Specifically , there are three key obser- vation sets to consider : those available to the attacker , those that the attacker needs to generate , and those available to the authenticator when designing the system . Our goal is to understand how these three information sources determine the optimal strategies for both players . Under the realistic assumption that cyber attackers are sophisticated enough to play optimal or close to optimal strategies , a characterization of the maximin authentication strategy can be of significant value . We prove several theoretical results characterizing the optimal strategy for both players . These results highlight the role played by the available observations as well as the functional form for an optimal attacker and authenticator . We refer to the setting above as “ GAN in The Middle ” ( GIM ) due to its similarity to “ man in the middle ” attacks . After describing our theoretical results , we show how to learn both authenticator and attacker policies in practice , where both are based on neural architectures . Our GIM method can be applied to multiple practical problems . The first is building an authentication system that is robust to impersonation attacks . The second is building a data generating mechanism that can generate novel data instances . Finally , we evaluate the method empirically , showing that it outperforms existing methods in terms of resilience to generative attacks , and that it can be used effectively for data-augmentation in the few-shot learning setting . 2 PROBLEM STATEMENT . We begin by motivating our problem and formulating it as a two-player zero-sum game . As a simple illustrative example , consider a face authentication security system whose goal is to maximize authentication accuracy . The system is initialized by registering k images of an individual θ ( the “ source ” ) , whose identity is to be authenticated . At test-time , each entity claiming to be θ is required to present to the system n of its images , and the authentication system decides whether the entity is θ or an impersonator . We let m denote the maximum number of “ leaked ” images any attacker obtained . We observe that if an attacker obtainedm ě n images of θ , it can present n of those images . Thus the observations generated by the attacker are indistinguishable from ones generated by θ , leading to failure of the authentication system ( see Sec . 4.2 below ) . Hence , the number of images of θ that the attacker obtains and the size of the authentication sample are of key importance . We now turn to formally stating the problem . Notation . The set of possible observations is denoted by X . Let H denote the known set of possible sources θ , where each source θ P H is defined by a probability density fθ , and an observation of a source θ P H is an X -valued random variable with density fθ . We assume that subsequent observations of the source are IID , so that n sequential observations have density f pnq θ px1 , . . . , xnq : “ śn i “ 1 fθpxiq . We allow θ to be sampled from a known distribution Q on H and denote the corresponding H-valued random variable by Θ . In what follows we will denote the number of observations leaked to the attacker by m , the number of “ registration ” observations available to the authenticator by k , and the number of observations required at authentication by n ( these may be generated by either the attacker or the true source ) . The Authentication Game . The game begins with a random source Θ being drawn from H according to Q . The authenticator first receives information about the drawn source , and then chooses a decision rule for deciding whether a given test sequence x P Xn is an authentic sequence of observations sampled from f pnqθ or a fake sequence generated by the attacker . Formally , the authenticator learns about the source by seeing k IID “ registration ” observations A “ A1 , . . . , Ak „ f pkqθ . The set of all possible decision rules is then D : X k ˆ Xn Ñ t0 , 1u ( where a decision of 1 corresponds to the true source and 0 to an attacker ) . After the authenticator fixes its strategy , the attacker can seek the best attack strategy . We assume that the attacker has access to m “ leaked ” IID observations Y “ Y1 , . . . , Ym „ f pmqθ as information about the source θ . Then it generates an attack sequence X P Xn and presents it to the authenticator , which uses its decision rule to decide whether X is an authentic sequence of observations sampled from f pnqθ or a fake sequence generated by the attacker . Formally , the strategy set of the attacker is all functions G : Xm Ñ ∆pXnq , where ∆pXnq is the set of probability distributions over Xn , and gX|Y is the associated conditional probability density . We note that the set H , the parametric family fθ , and the prior probability Q are known to both players . Also , note that the leaked sample Y revealed to the attacker is not available to the authenticator , and the “ registration ” sample A is not available to the attacker . The goal of the authenticator is to maximize its expected accuracy , and the goal of the attacker is to minimize it ( or equivalently maximize its success probability ) . We define the utility ( payoff ) of the game as the expected prediction accuracy of the authenticator . To define expected accuracy we consider the case of equal priors for attack and real samples.1 Formally , for a pair of strategies pD , Gq and a specific source θ , the expected accuracy of the authenticator is then : V pθ , D , Gq “ 1 2 E A „ fpkqθ E Y „ fpmqθ ” E X „ fpnqθ rDpA , Xqs ` EX „ GpY qr1´DpA , Xqs ı ( 2.1 ) Since this utility only depends on G in the second term , minimizing it is equivalent to G maximizing its success probability . To obtain the overall utility for the authenticator , we take the expected value w.r.t Θ and define V pD , Gq “ EΘ „ QV pΘ , D , Gq . Finally , we arrive at the following maximin game : Vgame “ max DPD min GPG V pD , Gq ( 2.2 ) where D , G are the sets of all possible authenticator and attacker strategies , respectively . In Sec . 4 we show that this game has a Nash equilibrium , we characterize the optimal strategies and game value in general , and find them in closed form for the case of Multivariate Gaussian sources . 3 RELATED WORK . Adversarial hypothesis testing ( AHT ) : Hypothesis testing ( HT ) is a rich field in statistics that studies how one can detect whether a sample was generated by one of two sets of distributions . A variant of HT that is related to our work but distinct from it is AHT ( e.g. , see Brandão et al. , 2014 ; Barni & Tondi , 2013b ; a ; 2014 ; Bao et al. , 2011 ; Zhou et al. , 2019 ; Brückner & Scheffer , 2011 ; Brückner et al. , 2012 ) . These works describe an HT setting where the sample is generated by one of two hypotheses classes , but is then modified by an adversary in some restricted way . E.g. , in Barni & Tondi ( 2013b ; a ; 2014 ) the adversary can change the sample of one class up to a fixed distance ( e.g. , Hamming ) . Given the quality of current generative models and the rapid pace of progress , when considering an impersonation attack , it seems that the only relevant restriction one can assume on an attacker is on the information it has . This is not captured by prior work since it assumes that the adversary has a restricted strategy set . In contrast , our work considers a novel problem setting where both players are not limited in their strategy set in any way . This leads to a novel analysis that focuses on the dependence on the finite information available to each player ( m , n , k ) . Adversarial Examples : It has been observed ( Goodfellow et al. , 2014b ) that deep learning models can be very sensitive to small changes in their input . Such “ misleading ” inputs are known as adversarial examples and much recent work has analyzed the phenomenon ( Ilyas et al. , 2019 ; Shamir et al. , 2019 ; Zhang et al. , 2019 ) , addressed the problem of robustness to these ( Moosavi-Dezfooli et al. , 2016 ; Papernot et al. , 2017 ; Yuan et al. , 2017 ) , and studied when robusntess can be certified ( Raghunathan et al. , 2018 ; Wong et al. , 2018 ) . The setting of robust classification in the presence of adversarial examples can also be thought of as a specific case of AHT ( see above ) , where a classifier is required to predict the class of an observation that could have been perturbed by a restricted adversary ( Wong et al. , 2018 ; 2019 ) or generated by an adversary limited to generating examples that will be classified correctly by humans ( Song et al. , 2018 ) . In contrast , in our setting the attacker is not limited in any way , nor does it have another utility in addition to impersonating the source . Furthermore , in adversarial examples , there is no notion of limited information for the adversary , whereas our work focuses on the dependence of the game on the information available to the players ( sample sizes n , m , k ) . GAN : The GAN model is a game between a generator and a discriminator . While our concept of generative attacks is inspired by GAN , it is very different from it : a successful GAN generator is not necessarily a successful attacker in our setting , and vice-versa ( e.g. , given sufficiently expressive generators and discriminators , GANs can “ memorize ” the training data , and thus the discriminator will perform at chance level.2 Such a discriminator will not be useful as a defense against generative attacks ) . Unlike GANs , in our setting , sample sizes are of key importance . Thus , our attacker will not memorize the data it sees , as this will be detected when generating n ą m examples . Conditional GANs : In conditional GANs ( Mirza & Osindero , 2014 ) the generator uses side information for generating new samples . The attacker in our approach ( analogous to GAN generator ) has input , but this input is not available to the authenticator ( analogous to GAN discriminator ) . Thus , 1We give equal prior probability to attack and real samples for simplicity and clarity , and since this is common in GAN formulations . All results can be trivially generalized to any prior probability for attack . 2If the generator is not expressive enough , the learned GAN may have small support ( Arora et al. , 2017 ) . the objective of the learning process is fundamentally different . Few-shot learning and generation : Our work relates to few-shot learning ( Snell et al. , 2017 ; Vinyals et al. , 2016 ; Finn et al. , 2017 ; Lake et al. , 2011 ; Koch et al. , 2015 ) and few-shot generative models ( Rezende et al. , 2016 ; Zakharov et al. , 2019 ; Lake et al. , 2015 ; Edwards & Storkey , 2017 ; Hewitt et al. , 2018 ) in the sense that both authenticator and attacker need to learn from a limited set of observations . However , in our setting the authenticator is required to predict whether a sample came from the true source or an attacker impersonating the source while taking into consideration the amount of information both players have . These are notions that are not part of the general few-shot learning setup . Also , in prior work on few-shot generation , the generator is either measured through human evaluation ( Lake et al. , 2015 ) or trained to maximize the likelihood of its generated sample ( Rezende et al. , 2016 ; Edwards & Storkey , 2017 ; Hewitt et al. , 2018 ) . In contrast , in our setting the attacker ’ s objective is to maximize the probability that its generated sample will be labeled as real by an authenticator . To this end , we show that the attacker must consider the sample sizes m , n , k , which the generative models in prior work do not account for . Furthermore , we show in Sec . F.4 , and in Figures 1c,6 , that the maximum likelihood ( ML ) solution is indeed sub-optimal in our setting . Image to image translation : Several GAN models have been introduced for mapping between two domains ( Zhu et al. , 2017 ; Huang et al. , 2018 ; Isola et al. , 2017 ; Wang et al. , 2017 ; Park et al. , 2019 ) . This relates to our work since the attacker also needs to learn to map the leaked sample to an attack sample . However , in our setting the mapping is not to a different domain but rather to other images from the same distribution , which results in a different objective . Data Augmentation : Generative models have also been used for augmenting data in supervised learning , and in particular few-shot learning ( Koch et al. , 2015 ; Snell et al. , 2017 ; Vinyals et al. , 2016 ; Lake et al. , 2011 ) . One such approach is Data Augmentation GAN ( DAGAN ) ( Antoniou et al. , 2018 ) , which takes as input an image and generates a new image . It relates to our framework in the limited case of m “ 1 , n “ 2 , k “ 1 , in the sense that the generator ’ s objective is to map one image to two . However , in DAGAN the only goal of the discriminator is to improve the generator , and the generator is limited to the functional form of adding a new image to the existing one , which is a sub-optimal attack strategy , as can be seen from the Gaussian case of our problem .

Summary: This paper proposes a new threat model for generative impersonation attacks: The attacker has access to several leaked images of a person; the authenticator knows several registration images per person and decides a person's identify by comparing some newly-sampled images from that person with corresponding registration images. The authors formulate this threat model as a minimax game and analyzed its Nash equilibrium. In the simplified case that observations are multivariate Gaussian, the authors are able to characterize the optimal strategies of the attacker and authenticator explicitly, which gives a nice intuition on how the theoretical optimum changes with respect to data dimension, number of leaked images, etc. Additionally, the authors implemented this attack (named Gan-in-the-middle attack) with an objective similar to GANs, empirically verified the theoretical results, and demonstrated the success of their approach on VoxCeleb2 and