Adversarial Privacy Preservation under Attribute Inference Attack

1 INTRODUCTION . With the growing demand for machine learning systems provided as services , a massive amount of data containing sensitive information , such as race , income level , age , etc. , are generated and collected from local users . This poses a substantial privacy challenge and it has become an imperative object of study in machine learning ( Abadi et al. , 2016 ; Gilad-Bachrach et al. , 2016 ) , computer vision ( Chou et al. , 2018 ; Wu et al. , 2018 ) , healthcare ( Beaulieu-Jones et al. , 2018b ; a ) , security ( Shokri et al. , 2017 ) , and many other domains . In this paper , we consider a practical scenario where the prediction vendor requests crowdsourced data for a target task , e.g , scientific modeling . The data owner agrees on the data usage for the target task while she does not want her other private information ( e.g. , age , race ) to be leaked . The goal of privacy-preserving in this context is then to protect private attributes of the sanitized data released by data owner from potential attribute inference attacks of a malicious adversary . For example , in an online advertising scenario , while the user ( data owner ) may agree to share her historical purchasing events , she also wants to protect her age information so that no malicious adversary can infer her age range from the shared data . Note that simply removing age attribute from the shared data is insufficient for this purpose , due to the redundant encoding in data , i.e. , other attributes may have a high correlation with age . Among many other techniques , differential privacy ( DP ) has been proposed and extensively investigated to protect the privacy of collected data ( Dwork & Nissim , 2004 ; Dwork et al. , 2006 ) . DP embraces formal guarantees for privacy problems such as defending against the membership query attacks ( Abadi et al. , 2016 ; Papernot et al. , 2016 ) , or ensures the distribution of any two data records statistically indistinguishable ( Erlingsson et al. , 2014 ; Duchi et al. , 2013 ; Bassily & Smith , 2015 ) . However , DP still suffers from attribute inference attacks ( Fredrikson et al. , 2015 ; Cormode , 2011 ; Gong & Liu , 2016 ) , as it only prevents an adversary from gaining additional knowledge by inclusion/exclusion of a subject , not from gaining knowledge from the data itself ( Dwork et al. , 2014 ) . As a result , an adversary can still accurately infer sensitive attributes of data owners from differentially-private datasets . Such a gap between theory and practice calls for an important and appealing challenge : Can we find a representation of the raw data to remove private information related to a sensitive attribute while still preserving our utility of the target task ? If no , what is the fundamental trade-off between privacy preservation and utility maximization ? Clearly , under the setting of attribute inference attacks , the notion of privacy preservation should be attribute-specific : the goal is to protect specific attributes from being inferred by malicious adversaries . Note that this is in sharp contrast with differential privacy , where mechanisms are usually designed to resist worst-case membership query among all the data owners . From this perspective , our relaxed definition of privacy also allows for a more flexible design of algorithms with better utility . Our Contributions In this paper , we first formally define the notion of utility and privacy . We justify why our definitions are particularly suited under the setting of attribute inference attacks . Through the lens of representation learning , we then formulate the problem of utility maximization with privacy constraint as a minimax optimization problem that can be effectively and practically implemented . To provide a formal guarantee on privacy preservation , we prove an informationtheoretic lower bound on the inference error of the protected attribute under attacks from arbitrary adversaries . To investigate the relationship between privacy preservation and utility maximization , we also provide a theoretical result to formally characterize the inherent trade-off between these two concepts . Empirically , we extensively conduct experiments to corroborate our privacy guarantee and validate the inherent trade-offs in different privacy preservation algorithms . From our empirical results , we conclude that the adversarial representation learning approach achieves the best trade-off in terms of privacy preservation and utility maximization , among various state-of-the-art privacy preservation algorithms . 2 PRELIMINARY . We first introduce our problem setting , the notations used throughout the paper and formally define the notions of utility and privacy discussed in this paper . 2.1 PROBLEM SETUP AND NOTATION . Problem Setup We focus on the setting where the goal of the adversary is to perform attribute inference . This setting is ubiquitous in sever-client paradigm where machine learning is provided as a service ( MLaaS , Ribeiro et al . ( 2015 ) ) . Formally , there are two parties in the system , namely the prediction vendor and the data owner . We consider the practical scenarios where users agree to contribute their data for training a machine learning model for specific purposes but do not want others to infer their private attributes in the data , such as health information , race , gender , etc . The prediction vendor will not collect raw user data but processed user data and the target attribute for the target task . In our setting , we assume the adversary can not get other auxiliary information than the processed user data . In this case , the adversary can be anyone who can get access to the processed user data to some extent and wants to infer other private information . For example , malicious machine learning service providers are motivated to infer more information from users to do user profiling and targeted advertisements . The goal of the data owner is to provide as much information as possible to the prediction vendor to maximize the vendor ’ s own utility , but under the constraint that the data owner should also protect the private information of the data source . Notation We use X , Y andA to denote the input , output and adversary ’ s output space , respectively . Accordingly , we use X , Y , A to denote the random variables which take values in X , Y and A . We note that in our framework the input space X may or may not contain the private attribute A . For two random variables X and Y , I ( X ; Y ) denotes the mutual information between X and Y . We use H ( X ) to mean the Shannon entropy of random variable X . Similarly , we use H ( X | Y ) to denote the conditional entropy of X given Y . We assume there is a joint distribution D over X × Y × A from which the data are sampled . To make our notation consistent , we use DX , DY and DA to denote the marginal distribution of D over X , Y and A . Given a feature map function f : X → Z that maps instances from the input space X to feature space Z , we define Df : = D◦f−1 to be the induced ( pushforward ) distribution ofD under f , i.e. , for any event E′ ⊆ Z , PrDf ( E ′ ) : = PrD ( f −1 ( E′ ) ) = PrD ( { x ∈ X | f ( x ) ∈ E′ } ) . 2.2 UTILITY AND PRIVACY . To simplify the exposition , we mainly discuss the attribute inference setting where X ⊆ Rd , Y = A = { 0 , 1 } , but the underlying theory and methodology could easily be extended to the categorical case as well . In what follows , we shall first formally define both the utility of the prediction vendor and the privacy of the data owner . It is worth pointing out that our definition of privacy is attributespecific , and this is in contrast with the classic framework of differential privacy where the goal is to preserve privacy in the general and worst-case query scenario . In particular , we seek to keep the utility of the data while being robust to an adversary on protecting specific information from attack . A hypothesis is a function h : X → Y . The error of a hypothesis h under the distribution D over X × Y is defined as : Err ( h ) : = ED [ |Y − h ( X ) | ] . Similarly , we use Êrr ( h ) to denote the empirical error of h on a sample from D. For binary classification problem , when h ( x ) ∈ { 0 , 1 } , the above loss also reduces to the error rate of classification . LetH be the Hilbert space of hypotheses . In the context of binary classification , we define the utility of a hypothesis h ∈ H as the opposite of error : Definition 2.1 ( Utility ) . The utility of a hypothesis h ∈ H is Util ( h ) : = 1− ED [ |Y − h ( X ) | ] . For binary classification , we always have 0 ≤ Util ( h ) ≤ 1 , ∀h ∈ H. Now we proceed to define a measure of privacy in our framework : Definition 2.2 ( Privacy ) . The privacy w.r.t . task A under attacks fromH is defined as PrivA ( H ) : = minh∈H 1− ∣∣PrD ( h ( X ) = 1 | A = 1 ) − PrD ( h ( X ) = 1 | A = 0 ) ∣∣ . Again , it is straightforward to verify that 0 ≤ PrivA ( H ) ≤ 1 . Based on our definition , PrivA ( H ) then measures the privacy of data under possible attacks from adversaries in H. We can also refine the above definition to a particular hypothesis h : X → { 0 , 1 } to measure its ability to steal information about A : PrivA ( h ) = 1− ∣∣PrD ( h ( X ) = 1 | A = 1 ) − PrD ( h ( X ) = 1 | A = 0 ) ∣∣ . Proposition 2.1 . Let h : X → { 0 , 1 } be a hypothesis , then PrivA ( h ) = 1 iff I ( h ( X ) ; A ) = 0 and PrivA ( h ) = 0 iff h ( X ) = A almost surely or h ( X ) = 1−A almost surely . Proposition 2.1 justifies our definition of privacy : when PrivA ( h ) = 1 , it means that h ( X ) contains no information about the sensitive attribute A . On the other hand , if PrivA ( h ) = 0 , then h ( X ) fully predicts A ( or equivalently , 1 − A ) from input X . In the latter case h ( X ) also contains perfect information of A in the sense that I ( h ( X ) ; A ) = H ( A ) , i.e. , the Shannon entropy of A . It is worth pointing out that our definition of privacy is insensitive to the marginal distribution of A , and hence is more robust than other definitions such as the error rate of predicting A . In that case , if A is extremely imbalanced , even a naive predictor can attain small prediction error by simply outputting constant . We call a hypothesis space H symmetric if ∀h ∈ H , 1 − h ∈ H as well . Interestingly , when H is symmetric , we can also relate the privacy PrivA ( H ) to a binary classification problem : Proposition 2.2 . If H is symmetric , then PrivA ( H ) = minh∈H Pr ( h ( X ) = 0 | A = 1 ) + Pr ( h ( X ) = 1 | A = 0 ) . Remark Consider the following confusion matrix between the actual private attribute A and its predicted variable h ( X ) in Table 1 . The false positive rate ( eqv . Type-I error ) is defined as FPR = FP / ( FP + TN ) and the false negative rate ( eqv . Type-II error ) is similarly defined as FNR = FN / ( FN + TP ) . Using the terminology of confusion matrix , it is then clear that Pr ( h ( X ) = 0 | A = 1 ) = FNR and Pr ( h ( X ) = 1 | A = 0 ) = FPR . In other words , Proposition 2.2 says that ifH is symmetric , then the privacy of a hypothesis spaceH corresponds to the minimum sum of Type-I and Type-II error that is achievable under attacks fromH .

Summary: This paper proposes a definition of privacy against attribute inference attack and formulates it as a minimax optimization problem that balances accuracy and privacy, it then provides an information-theoretic lower bound for the inference error. It also provides an analysis of the intrinsic privacy-utility tradeoff. Finally, it shows experimental evaluation in terms of inference error and accuracy of several representation-learning based approaches for protecting against attribute inference attack and compares with the lower bounded proved in the paper. By comparing utility with a differentially private algorithm, it shows that the other algorithm achieves higher utility under similar privacy.