Adversarial Privacy Preservation under Attribute Inference Attack
Authors: Han Zhao
Jianfeng Chi
Yuan Tian
Geoffrey J. Gordon
Venue:  (2020)

Abstract: With the prevalence of machine learning services, crowdsourced data containing sensitive information poses substantial privacy challenges. Existing work focusing on protecting against membership inference attacks under the rigorous framework of differential privacy are vulnerable to attribute inference attacks. In light of the current gap between theory and practice, we develop a novel theoretical framework for privacy-preservation under the attack of attribute inference. Under our framework, we propose a minimax optimization formulation to protect the given attribute and analyze its privacy guarantees against arbitrary adversaries. On the other hand, it is clear that privacy constraint may cripple utility when the protected attribute is correlated with the target variable. To this end, we also prove an information-theoretic lower bound to precisely characterize the fundamental trade-off between utility and privacy. Empirically, we extensively conduct experiments to corroborate our privacy guarantee and validate the inherent trade-offs in different privacy preservation algorithms. Our experimental results indicate that the adversarial representation learning approaches achieve the best trade-off in terms of privacy preservation and utility maximization.

Under review as a conference paper at ICLR 2020
ADVERSARIAL PRIVACY PRESERVATION UNDER AT- TRIBUTE INFERENCE ATTACK
Anonymous authors Paper under double-blind review
ABSTRACT
With the prevalence of machine learning services, crowdsourced data containing sensitive information poses substantial privacy challenges. Existing work focusing on protecting against membership inference attacks under the rigorous framework of differential privacy are vulnerable to attribute inference attacks. In light of the current gap between theory and practice, we develop a novel theoretical framework for privacy-preservaate
under the attack of attribute inference. Under our frame- work, we propose a minimax optimization formulation to protect the given attribute and analyze its privacy guarantees against arbitrary adversaries. On the other hand, it is clear that privacy constraint may cripple utility when the protected attribute is correlated with the target variable. To this end, also prove information- theoretic lower bound to precisely characterize the fundamental trade-off between utility and privacy. Empirically, rate our privacy guarantee and validate the inherent trade-off' in different privacy preservation algorithms. Our experimental results indicate that the adversarial representation learning approaches achieve the best trade-off in terms of privacy preservation and utility maximization.
extensively conduct experiments to corrobo-
INTRODUCTION
With the growing demand for machine learning systems provided as services, massive amount of data containing sensitive information, such as race, income level, age, etc., are generated and collected from local users. This poses substantial privacy challenge and it has become an imperative object of study in machine learning (Abadi al. 2016, -achhahh et al. 2016), computer vision (Chou et al.) 2018: Wu et al.] 2018), healthcare (Beaulieu-Jonee et al.) 2018b a). security (Shokri et al. 2017). and many other domains. In this paper, we consider a practical scenario where the prediction vendor requests crowdsourced data for target task, e.g, scientific modeling. The data owner agrees on the data usage for the target task while she does not want her other private information (e.g., age, in this context is then to protect private attributes
race) to be leaked. The goal of trivacy-preservee of the sanitized data released by data owner from potential attribute inference attacks a malicious adversary. For example, in online advertising scenario, while the user (data owner) may agree to share her historical purchasing events, she also wants to protect her age information so that no malicious adversary can infer her age range from the shared data. Note that simply removing age attribute from the shared data is insufficient for this purpose, due to the redundant encoding in data, i.e., other attributes may have a high correlation with age.
Among many other techniques, differential privacy (DP) has been proposed and extensively in- vestigated to protect the privacy of collected data (Dwork & Nissim 2004; Dwork et al., 2006). DP embraces formal guarantees for privacy problems such as defending against the membership query attacks (Abadi al., 2016 PPapernot et al. 2016), or ensures the distribution of any two data records statistically indistinguishable (Erlingsson et al. 2014 Duchi et al.] 2013) Bassily & Smith 2015). However, DP still suffers from attribute inference attacks (Fredrikson et al. 2015 Cormode
2011 Gong & Liu| 2016), as it only prevents an adversary from gaining additional knowledge by inclusion/exclusion of subject, not from gaining knowledge from the data itself (Dwork et al. 2014). As result, adversary can still accurately infer sensitive attributes of data owners from differentially-prine datasets. Such a gap between theory and practice calls for an important and appealing challenge:

---

Under review as a conference paper at ICLR 2020
Can we find representation of the raw data remove private information related to a sensitive attribute while still preserving our utility of the target task? If no, what is the fundamental trade-off between privacy preservation and utility maximization?
Clearly, under the setting of attribute inference attacks, the notion of privacy preservation should be attribute-specific: the goal is to protect specific attributes from being inferred by malicious adversaries. Note that this is in sharp contrast with differential privacy, where mechanisms are usually designed to resist worst-case membership query among all the data owners. From this perspective, our relaxed definition of privacy also allows for a more flexible design of algorithms with better utility.
Our Contributions In this paper, we first formally define the notion of utility and privacy. We justify why our definitions are particularly suited under the setting of attribute inference attacks. Through the lens of representation learning, we then formulate the problem of utility maximization with privacy constraint implemented. To provide a formal guarantee on privacy preservation, we prove an information- theoretic lower bound the inference error of the protected attribute under attacks from arbitrary adversaries. To investigate the relationship between privacy preservation and utility maximization, we also provide a theoretical result to formally characterize the inherent trade-off between these two concepts. Empirically, we extensively conduct experiments to corroborate our privacy guarantee and validate the inherent trade-offs in different privacy preservation algorithms. From our empirical results, we conclude that the adversarial representation learning approach achieves the best trade-off in terms of privacy preservation and utility maximization, among various tfate-of-the-ar privacy preservation algorithms.
minimax optimization problem that can be effectively and practically
2 PRELIMINARY
We first introduce our problem setting, the notations used throughout the paper and formally define the notions of utility and privacy discussed this paper.
2.1 PROBLEM SETUP AND NOTATION
Problem Setup We focus the setting where the goal of the adversary is to perform attribute inference. This setting is ubiquitous in sever-client paradigm where machine learning is provided as service (MLaaS, (MLaaS,|Ribeire Ribeiro et al. (2015).. Formally, there two parties in the system, namely the prediction vendor and the data owner. We consider the practical scenarios where users agree to contribute their data for training a machine learning model for specific purposes but do not want others to infer their private attributes in the data, such as health information, race, gender, etc. The prediction vendor will not collect raw user data but processed user data and the target attribute for the target task. In our setting, we assume the adversary cannot get other auxiliary information than the processed user data. In this case, the adversary can be anyone who can get access to the processed user data to some extent and wants to infer other private information. For example, malicious machine learning service providers are motivated to infer more information from users to do user profiling and targeted advertisements. The goal of the data owner is provide much information possible to the prediction vendor to maximize the vendor's 's own utility, but under the constraint that the data owner should also protect the private information of the data source.
Notation We use X', y and A to denote the input, output and adversary's output space, respectively, Accordingly, we use X, Y.A A denote the random variables which take values X' and A. We note that in our framework the input space X' may or may not contain the private attribute A. For two random variables X and Y. Y(; denotes the mutual information between X and Y. We use (XX to mean the Shannon entropy of random variable X. Similarly, we use H(X Y) to denote the conditional entropy of X given Y. We assume there is a joint distribution D over X' y A from which the data are sampled. To make our notation consistent, we use Dx. Dy and DA to denote the marginal distribution of D over X. y and A. Given a feature map function : X + Z that maps instances from the input space X' to feature space Z, define DI := Dof f-1 to be the induced (pushforward) distribution D under f, i.c., for any event E' C Z, Prps (F')= := Pr Pro (F))) Pro(( E X f(x) E F))).
2

---

Under review as a conference paper at ICLR 2020
2.2 UTILITY AND PRIVACY
To simplify the exposition, we mainly discuss the attribute inference setting where X' C , = [(0, 13. but the underlying theory and methodology could easily be extended the categorical case as well. In what follows, we shall first formally define both the utility of the prediction vendor and the privacy of the data owner. It is worth pointing out that our definition of privacy is attribute- specific, and this is in contrast with the classic framework of differential privacy where the goal is to preserve privacy in the general and worst-case query scenario. In particular, we seek to keep the utility of the data while being robust to an adversary on protecting specific information from attack.
hypothesis a function X + y. The error of a hypothesis h under the distribution over X' x y is defined as: Err(h) == Ep Y h(X)])- Similarly, we use Err(h) to denote the empirical error of h on a sample from D. For binary classification problem, when h(x) E {0, 1), the above loss also reduces to the error rate of classification. Let H be the Hilbert space of hypotheses. In the context of binary classification, we define the utility of a hypothesis h E H. as the opposite of error: Definition 2.1 (Utility). The utility of a hypothesis / E H. is (h) == 1 Ep E Y h(X)]]
For binary classification, we always have 0 S Util(h) S 1, vh E H. Now we proceed to define a measure of privacy in our framework:
Definition 2.2 (Privacy). The privacy w.r.t. task A under attacks from H is defined as Priv =
mingeH 1 - (r)(h(X)= = A = - (r)()(X) = 1 A= A)
Again, it is straightforward verify that 0 S Priv (?H S 1. Based our definition, Priv A (H) then measures the privacy of data under possible attacks from adversaries in H. We can also refine the above definition to a particular hypothesis h X' {0.1} to measure its ability steal information (r)()(X) 1 1) Pr
about A: Priv (h)
A == 0)/-
Proposition 2.1. Let h: : X - {0, 13 be a hypothesis, then Phiva(h) = 1 iff [(h(X); A) = 0 and almost surely.
Priv iff 1: (X) = almost surely or (x)
Proposition 2.1 justifies our definition of privacy: when (h) = 1. it means that ((X) contains no information about the sensitive attribute A. On the other hand, if Priv a (h) O. then h(X) fully predicts A (or equivalently, 1 - A) from input X. In the latter case h(X) also contains perfect information of in the sense that 7 (X); A) = (A), i.e., the Shannon entropy of A. It is worth pointing out that our definition of privacy is insensitive to the marginal distribution of A, and hence is more robust than other definitions such as the error rate of predicting A. In that case, if A is extremely imbalanced, even naive predictor can attain small prediction error by simply outputting constant. We call a hypothesis space H. symmetric if Vh E H, symmetric, can also relate the privacy Priva(?() to binary classification problem:
E 7 as well. Interestingly, when H is
Proposition 2.2. If H. is symmetric, then Priv (iva(H)) (H) = minneN Pr(h(X) = A = 1) Pr(h(X)= = A 4==00 0).
Remark Consider the following confusion matrix be- tween the actual private attribute A and its predicted vari- able (xX) in Table I The false positive rate (equ. Type-I error) is defined FPR FP/ (FP + TN) and the false neg- ative rate (eqv. Type-II error) is similarly defined as FNR FN/ / (FN + TP). Using the terminology of confusion ma- trix, it is then clear that (r
Table 1: Confusion matrix between A and h(X).
M(X) = h(X)=
A= 0
TN
FP
0 A = 1) FNR 0) FPR. In other words, Proposition 2ays says that if H. is symmetric, then the privacy of a hypothesis space H corresponds to the minimum sum of Type-I and Type-Il error that is achievable under attacks from H.
A=
FN
TP
and Pr(h(X)
MINIMAX OPTIMIZATION AGAINST ATTRIBUTE INFERENCE ATTACKS
3.1 MINIMAX FORMULATION
Given a set of samples S = {(xx, yi. a;) |/ drawn i.i.d. from the joint distribution D, how can the data owner keeps the utility of the data while keeping the sensitive attribute A private under potential
3

---

Under review as a conference paper at ICLR 2020
attacks from malicious adversary? Through the lens of representation learning, we seek to find (non-linean feature representation f X + Z from input space X' to feature space 2 such that f still preserves relevant information w.r.t. the target task of inferring Y while hiding sensitive attribute A. Specifically, we can solve the following unconstrained regularized problem with À > 0:
min max Err(h - f) 1 ((Pr(M - 0 - 1) + Pr(N'((()) = A - 0)) (1) KEH.S WEH
It is worth pointing out that the optimization formulation in ①1 admits interesting game-theoretic interpretation, where two agents If and h play game whose score is defined by the objective function in (T). Intuitively, h seeks minimize the sum of Type-I and Type-Il error while plays against h'' by learning transformation to removing information about the sensitive attribute A. Algorithmically, for the data owner to achieve the goal of hiding information about the sensitive attribute A from malicious adversary. it suffices to learn representation that is independent of A. Formally:
Proposition 3.1. Let f : X + Z be a deterministic function and H. C 2 be a hypothesis class over 2. For any joint distribution D over A. if ((f(X); A) = 0, then Priv (H f) 1.
Note that in this sequential game, f is the first-mover and h is the second. Hence without explicit constraint f possesses first-mover advantage so that f can dominate the game by simply mapping all the input X to a constant or uniformly random noise. To avoid these degenerate cases, the first term in the objective function of (T) acts as an incentive to encourage to preserve task-related information. But will this incentive compromise our privacy? As an extreme case if the target variable y and the sensitive attribute A are perfectly correlated, then it should be clear that there a trade-off in achieving utility and preserving privacy. In Sec. 4 we shall provide information-thhoree bound to precisely characterize such inherent trade-off. Furthermore, although the formulation in (I) only works for defense over a single attribute, it is straightforward extend the formulation that it can protect attacks over multiple attributes. Due to space limit, we defer the discussion of this extension to Section in appendix.
3.2 PRIVACY GUARANTEES ON ATTRIBUTE INFERENCE ATTACKS
In the last section we propose the unconstrained minimax formulation (T) to optimize both our utility and the defined privacy measure. Clearly, the hyperparameter À measures the trade-off between utility and our privacy. On one hand, if A + 0, we barely care about the privacy and devote all the focus to maximize our utility. On the other extreme, X - oc, we are only interested in protecting the privacy. In what follows we analyze the true error that an optimal adversary has to incur in the limit when both the task classifier and the adversary have unlimited capacity, i.c., they can be any randomized functions from Z to {0, 1). To study the true error, we hence use the population loss rather than the empirical loss in our objective function. Furthermore, since the binary classification error in (1 is NP-hard to optimize even for hypothesis class of linear predictors, in practice we consider the pross-entropp loss function as a convex surrogate loss. The cross-entropy loss CEy(h) of probabilistic hypothesis h X + [0, 1] w.r.t. Y on distribution D is defined follows:
CFy (h) == - ED [T(Y= = 0) log(1 - h(X)) + T(Y = l)lo(h((X))
(2)
With slight abuse of notation, w.r.t. A. Using the same notation, the optimization formulation with cross-entropy loss becomes:
( to mean the cross-entropy loss of the adversary h/
min max AE N'EH
CBr f) XCE f)
Given a feature map f: X + Z, assume that H contains all the possible randomized classifiers from the feature space 2 to (0, 1}. For example. randomized classifier can be constructed by first defining probabilistic function h: Z + [0. 1] followed by random coin flipping to determine the output label, where the probability of the coin being is given by h ZZ). Under such assumptions, the following lemma shows that the optimal target classifier under f is given by the conditional distribution h" (Z) := Pr(Y- = 1 Z).
Lemma 3.1. For any feature map f X Z, assume that H contains all the randomized binary classifiers, then o and he (Z) := arg min CEy o f) = Pr(Y = 1 2 = f(X)).
also see that the worst-case (optimal) adversary under is the conditional distribution hi" h²((Z)= Pr(A 2) and minweH CEA o f) H(A) Z). Hence
By a symmetric argument,
4

---

Under review as a conference paper at ICLR 2020
can further simplify the optimization formulation optimization variable is the feature map f:
to the following form where the only
min H(Y Z = f(x)) XY (A 2 = f()))
Since Z - -X) is deterministic feature map, it follows from the basic properties of Shannon entropy that
H(Y X) H Z= f(x)) s H(Y). H(A ( H(A z f(X)) S H(),,
which means that HY X) - (A) is a lower bound of the optimum of the objective function in 4 However, such lower bound is not necessarily achievable. To see this, consider the simple case where Y = A almost surely. In this case there exists no deterministic feature map 2 = f(X) that both a sufficient statistics of X w.r.t. Y while simultaneously filters out all the information w.r.f. A except in the degenerate case where A is constant. On the other hand, to show that solving the optimization problem in (4) helps to protect our privacy, the following theorem gives a bound of privacy in terms of the error that has to be incurred by the optimal adversary:
Theorem 3.1. Let f be the optimal feature map of (4 and define H2 := H(A Z fx)))
Then for any adversary Ā Z - {0. 1} such that Á A Z. PTDS" (Ā A) 2 H /21g(//HH)
Remark Theorem 3.1 shows that whenever the conditional entropy H" 2 f()) large, then the inference error of the protected attribute incurred by any (randomized) adversary has to be at least Q2H"" / log(1/H"*)) the second term in our objective function, whose optimal value could further be flexibly adjusted by tuning the trade-off parameter X. As a final note, Theorem 3.1 also shows that representation learning helps to protect the privacy about A since we always have ((A) Z = f(X)) 2 H(A | X) for any deterministic feature map f so that the lower bound of inference error by any adversary is larger after learning the representation Z f(X).
As we have already shown above, the conditional entropy
essentially corresponds
4 INHERENT TRADE-OFF BETWEEN UTILITY AND PRIVACY
As we briefly mentioned in Sec. 3.1 when the protected sensitive attribute A and the target variable Y are perfectly correlated, it is impossible to simultaneously achieve the goal of privacy-preserving and utility-aximizingg
But what the exact trade-off between utility and privacy when they are correlated? In this section we shall provide an informatiin--heeortiee bound to quantitatively characterize the inherent trade-off between privacy and utility. due to the discrepancy between the conditional distributions of the target variable given the sensitive attribute. Our result is algorithm- independent, hence it applies to a general setting where there is a need to preserve both utility and privacy. To the best of our knowledge, this is the first informationn-hhorettii result to precisely quantify such trade-off. Due to space limit, we defer all the proofs to appendix.
Before we proceed, we first define several information-theorel concepts that will be used in our analysis. For two distributions D and D', the Jensen-Shannon (JS) divergence Dis(D) D') is:
1 Dis (D. D') = DK. (D) PM) + 1 DKL (D' P|
where (- -I is the Kullback-Leible (KL) divergence and DM := + ( The JS divergence can be viewed as symmetrized and smoothed version of the KL divergence, and it is upper bounded by the L1 distance (total variation) between two distributions through Lin's Lemma:
Lemma 4.1 Lin 1991)). Let DD and D' be two distributions, then Dis (D, D') < 21I - DDIlh-
Unlike the KL divergence, the IS divergence is bounded: 0 < DD, < 1. Additionally, from the IS divergence, we can define a distance metric between two distributions as well, known as the IS distance (Endres Schindelin 2003): (D, DD) :=
With respect to the JS distance, for any feature space 2 and any deterministic mapping X + 2, we can prove the following lemma via the celebrated data processing inequality:
Lemma 4.2. Let Do and D2 be two distributions over X' and let Dí and Df be the induced distribu- tions of Do and 'D over 2 by function then dis dis(D/ D() < dis(Do, Dı).
5

---

Under review as a conference paper at ICLR 2020
Without loss of generality, any method aiming to predict the target variable Y defines Markov chain
as X -2 z - Y, where Y is the predicted target variable given by hypothesis h and Z is the intermediate representation defined by the feature mapping f. Hence for any distribution Pa(Di) 1) of X, this Markov chain also induces a distribution D (Dfee) of Y and distribution D/ (Df) of Z. Now let DJ (DF) be the underlying true conditional distribution of Y given A = 0(A = 1). Realize that the JS distance is metric, the following chain of triangular inequalities holds:
dis (D DY) S dis (D' D + dis (Db D + dis(D) ,
Combining the above inequality with Lemma 4.2 to show
dss(D)of, D I dis(D/, D(),
we immediately have:
dis dis D S dis(D)', D + dis(P),, D() + dis ,DY).
Intuitively, dis (D F) and dDs , measure the distance between the predicted and the true target distribution on A = =0/1 cases, respectively. Formally, let Err, (h o f) be the prediction error of function h o conditioned on A a. With the help of Lemma 4.1 the following result establishes a relationship between dis(D. Dhof) and the utility of h o f:
Lemma 4.3. Let y h(f(X)) E 10. be the predictor, then for (0, 1), d s
VErra(h f).
Combine Lemma 4.2 and Lemma 4.3 we get the following key lemma that is the backbone for proving the main results in this section:
Lemma 4.4 (Key lemma). Let Do, Di be two distributions over X' x ) conditioned on A = 0 and
A = 1 respectively. Assume the Markov chain X م z b Y holds, then wh E H:
dis (D) D I VErro(h o f) + dis (D/, D()= + (Err,(h f).
(5)
We emphasize that for E {0, 1}, the term Erra (ho measures the conditional error of the predicted variable Y by composite function h f over Da. Similarly, can define the conditional utility for a E (0. 1} Util, (h f) := 1 - Err, (h a f). The following main theorem then characterizes a
fundamental trade-off between utility and privacy:
Theorem 4.1. Let H C 2 contains all the classifiers from 2 to (0, 13. Given the conditions in Lemma 4.4. 4h H. Utilo o f)+ Util, (h f) + Priv (H o f) S 3 (D) D))
A few remarks follow. First, note that the maximal value achievable by the sum of the three terms on the I.H.S. In light of this, the upper bound given in Theorem 441 shows that when the marginal distribution of the target variable Y differ between two cases A = 0 or A = 1, then it is impossible to perfectly maximize utility and privacy. Furthermore, the trade-off due to the difference in marginal distributions is precisely given by the JS divergence Das (D) DY ). Note that in Theorem 4.1 the upper bound holds for any hypothesis in the richest hypothesis class H that contains all the possible binary classifiers. Put it another way, if we would like to maximally preserve privacy w.r.t. sensitive attribute A, then we have to incur a large joint error:
Theorem 4.2. Assume the conditions in Theorem 4.1 hold. If Priv A (H o f) 2 1 DIs D
then Vh E H. Erro(h f) + Err (h o f) 2 1/2 (dss(2 (D) D - P
Remark The above lower bound characterizes fundamental trade-off between privacy and joint error. In particular, up to certain level 1 - D1s Dis(D)) (D) DV ), the larger the privacy, the larger the joint error. light of Proposition 3.1 this means that although the data-owner, or the first-mover f. could try to maximally preserve the privacy via constructing f such that f(X) is independent of A, such construction will also inevitably compromise the joint utility of the prediction vendor. is also worth pointing out that our results in both Theorem 4.1 and Theorem 4.2 attribute-independee sense that neither of the bounds depends on the marginal distribution of A. Instead, all the terms in results only depend on the conditional distributions given A 0 and A = 1. This is often more desirable than bounds involving mutual information, e.g., [AA, Y). since I (A, Y) is close to 0 if the marginal distribution of A is highly imbalanced.
in the

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS
Our theoretical results on the privacy guarantee of attribute inference attacks imply that the in- ference error of the protected attribute incurred by any (randomized) adversary has to be least In this section we extensively conduct experiments on two real-world bench- mark datasets, the UCI Adult dataset (Dua & Grafi 2017) and the UTKFace dataset (Zhang et al.) 2017) to verify 1). Our guarantee on privacy can be used as a certificate for different privacy preservation methods. 2). Inherent trade-offs exist between privacy and utility exist in all methods. 3). Among all the privacy preservation algorithms, including differential privacy, the adversarial representation learning approach achieves the best trade-off in terms of privacy preservation and utility maximization.
5.1 DATASETS AND SETUP
Datasets 1). Adult dataset: The Adult dataset is a benchmark dataset for privacy-preservation The task is predict whether individual's income is greater or less than 50K/year based on census data. The attributes in the dataset includes gender, education, occupation, age, etc. In this experiment we set the target task as income prediction and the private task as inferring gender, age and education, respectively. 2). The UTKFace dataset is a large-scale face dataset containing more than 20,000 images with annotations of age, gender, and ethnicity. It one of the benchmark datasets for age estimation, gender and race classifications. In this experiment, we set our target task as gender classification and we use the age and ethnicity as the protected attributes. We refer readers to Section D in the appendix for detailed descriptions about the data pre-processing pipeline.
Methods We conduct extensive experiments with the following methods to verify our theoretical
results and provide thorough practical comparison among these methods. 1). Privacy Partial Least
Squares (PPLS) Enev et al. 2012). 2). Privacy Linear Discriminant Analysis (PLDA) (Whitehill &
Movellan 2012). 3). Minimax filter with alternative update (ALT-UP) (Hamm 2017), 4) Maximum
Entropy Adversarial Representation Learning (MAX-ENT) (Roy & Boddeti 2019) 5). Gradient
Reversal Layer (GRL) (Ganin et al., (0166) 6). Principal Component Analysis (PCA) No defense
(NO-DEF), 8) Local Differential Privacy (LDP) with Laplacian mechanism, 9). differentially private
SGD (DPSGD) (Abadi et 2016
Among the first seven methods, the first five are state-of ohethherrt minimax methods for protecting against attribute inference attacks while the latter two are non-private baselines for comprehensive comparison. Although DP is not tailored to attribute inference attack, we can still add two DP baselines to examine the utility and privacy trade-off for comparison. Our goal here is to provide a thorough comparison in terms of utility-prine utility-privacy trade-off by using methods from both representation learning and differential privacy.
To make sure the comparison is fair among different methods, we conduct controlled experiment by using the same network structure as the baseline hypothesis among all the methods for each dataset. For each experiment on the Adult dataset and UTKFaace dataset, repeat the experiments for 10 times to report both the average performance and their standard deviations. Details on network structures and implementation of the above algorithms are provided in the appendix.
Note that in practice due to the non-convexity nature of optimizing deep neural nets, we cannot guarantee to find the global optimal conditional entropy H* Hence in order compute the privacy guarantee given by our lower bound in Theorem 3. we use the cross-entropy loss of the optimal adversary found by our algorithm on inferring the sensitive attribute A. Furthermore, since our analysis only applies to representation learning based approaches, we do not have privacy guarantee for De-related methods in our context.
5.2 RESULTS AND ANALYSIS
We visualize the performances of the aforementioned algorithms on privacy preservation and utility maximization in Figure T and Figure 2 respectively. First, from Figure I we can see that among all the methods, both LDP, PLDA, ALT-UP, MAX-ENT and GRL are effective in privacy preservation by forcing the optimal adversary to incur a large inference error. On the other hand, PCA and NO-DEF the least effective ones. This is expected as either NO-DEF nor PCA tries to filter information in
7

---

Under review as a conference paper at ICLR 2020
ALT-UPA-10
5.0
large
0.5
PrtÅ A)
0.3
0.2
0.1
0.0
Adult-Gender
Adull-Age
Adult-Education
UTKFace-Race
UTKFace-Age
Figure 1: Performance on privacy preservation of different methods (the larger the better). The horizontal lines across the bars indicate the corresponding privacy guarantees given by our lower bound in Theorem 3.1
NO.0EF PALS
ALTAPA--03
MAK-INT D.
1.0
LSP 1
DPECD small
10
GRLA-50
DPSG0
1.0
Emg Em
0.6
0.4
0.2
0.0
Adult-Gender
Adult-Age
Adult-Education
UTKFace-Raca
UTKFaace-Ag
Figure 2: The joint conditional error (Erry + ErT1, the smaller the better) of different methods.
data about the sensitive attribute A. We also that with larger trade-off value X, ALT-UP, MAX-ENT and GRL achieve better privacy preservation.
Second, from Figure _
also see sharp contrast between ba-based methods and other methods
in terms of the joint conditional error on the target task: both LDP and DPSGD incur significant utility loss compared with other methods. Combining this one with our previous observation from Figure [ we can see that DP either makes data private by adding large amount of noise to effectively filter out all the information available in the data, including both target-related and sensitive information, or add insufficient amount of noise so that both target-related and sensitive information is well preserved. As comparison, representation learning based approaches leads to much better trade-off. Among all the representation learning methods, PLDA, ALT-UP, MAX-ENT and GRL perform the best in privacy preservation. Compared to ALT-UP and GRL, MAX-ENT and PLDA is more effective in privacy preservation in some cases, but at the cost of a significant drop in utility. It is also worth to note that different adversarial representation learning methods have different sensitivity on A: a large À for MAX-ENT can lead unstable model training process and result in large utility loss. In contrast, GRL often more stable, which is consistent to the results shown in (Daskalakis & Panageas 2018
8

---

Under review as a conference paper at ICLR 2020
6 RELATED WORK
The attribute inference attack problem has close connections to both differential privacy and algorith- mie fairness. In this section mainly focus on discussing the connections and differences between these problems. As summary, we visualize their relationships in the diagram shown in Figure 3
Differential Privacy DP has been proposed to bound the difference of algorithmic output between any two "neighboring" datasets from the released data (Dwork & Nissim 2004; Dwork et al. 2006 Erlingsson et al. 2014) and was used in the training of deep neural network recently (Abadi et al. al. 2016)] |han et al., 2017). Our definition of privacy is different from (local) differential privacy since the goal of DP tries to make any two neighboring datasets have close probabilities to produce the same output. In the setting of learning algorithms, this means that the models trained from two neighboring datasets should be close to each other. However, this does not necessarily imply that the learned model itself is free from attribute inference attacks, As comparison, our goal of defending attribute inference attacks is to learn a representation such that the protected attributes cannot be accurately inferred. Put it in another way, given a dataset matrix, the goal of DP is to ensure that it is hard to infer about row in the matrix while our privacy definition seeks to ensure that is hard to infer about a specific column the data matrix. From this perspective, DP is closely related to the well-known membership inference attack (Shokri al. 2017) instead. It is observed (work et al. 2012 that the notion individual fairness may be viewed as a generalization of DP.
2016:
Algorithmic Fairness The privacy defined in this work is related to the notion of group fairness in the literature of algorithmic fairness (Dwork al. 2012; Edwards & Storkey 2015). In particular, adversarial learning methods have been used as tool in both fields to achieve the corresponding goals. However, the motivations and goals significantly differ between these two fields. Specifically, the widely adopted notion of group fairness, namely equalized odds (Hardt et al. (016)), requires cqualized false positive and false negative rates across different demographic subgroups. As a comparison, in applications where privacy is concern, mainly want ensure that adversaries cannot steal sensitive information from the data. Hence our goal is to give a worst case guarantee on the inference error that any adversary has least incur. To the best of our knowledge, our results in Theorem 3.1 is the first one to analyze the performance of privacy preservation such scenarios. Furthermore, prior theoretical results exist on discussing the trade-off between privacy and utility on defending attribute inference attacks. Our proof techniques developed in this work could also be used derive
lower bounds in related problems as well (Zhao et al. 2019
Zhao & Gordon 2019).
Individual Fairness
Group Fairness
(Dwork et al., 2012)
(Zenel et al., 2013)
Membership Inference Attack
Attribute Inference Attack (Fredrikson al., 2015)
(Shokri et al., 2017)
Figure 3: Relationships between different notions of fairness and privacy.
7 CONCLUSION
We develop a theoretical framework for privacy preservation under the setting of attribute inference attacks. Under this setting, we propose theoretical framework that suggests using adversarial learning techniques to protect the private attribute. We further analyze the privacy guarantee of the defense method in the limit of worst-case adversaries and prove an information-theone theoretic lower bound quantify the inherent trade-off between utility and privacy. Following formulation, we conduct

---

Under review as a conference paper at ICLR 2020
extensive experiments to corroborate theoretical results and to empirically compare different privacy preservation algorithms. Experimental results show that the adversarial very effective in defending attribute inference attacks and
representation learning approaches often achieve the best trade-off in terms of privacy preservation and utility maxinization. We believe work takes important step towards better understanding the privacy-utility trade-off, and it also helps to stimulate the future design of privacy-preservation algorithm with adversarial learning techniques.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMAhan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308-318. ACM, 2016.
Raef Bassily and Adam Smith. Local, private, efficient protocols for succinct histograms. In Proceedings of the forty-seventh anmual ACM symposium on Theory of computing, pp. 127-135.
ACM, 2015.
Brett K Beaulieu-Jones. Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey S Greene. Privacy-preserving generative deep neural networks support clinical data sharing. BioRxiv, pp. 159756, 2018a.
Brett K Beaulieu-Jones, William Yuan, Samuel G Finlayson, and Zhiwei Steven Wu. Privacy- preserving distributed deep learning for clinical data. arXiv preprint arXiv: 2018b.
Chris Calabro. The exponential complexity of satisfiability problems. PhD thesis, UC San Diego, 2009.
Edward Chou, Josh Beal, Daniel Levy, Serena Yeung, Albert Haque, and Li Fei-Fei. Faster cryptonets: Leveraging sparsity for real-world encrypted inference. arXiv preprint 1rXiv:1811.0995. 2018.
Graham Cormode. Personal privacy vs population privacy: learning to attack anonymization. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1253-1261 ACM, 2011.
Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. In Advances in Neural Information Processing Systems, pp. 9236-9246, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htth:http///hhhhaee
ics. acci, eedummll
John C Duchi, Michael Jordan, and Martin J Wainwright. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pp. 429-438. IEEE, 2013.
Cynthia Dwork and Kobbi Nissim. Privacy-preserviee preservieeeenggeee data mining on vertically partitioned databases. In Annual International Cryptology Conference, pp. 528-544. Springer, 2004.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284, Springer, 2006.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226. ACM, 2012.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4)21114009 1-407, 2014.
Harrison Edwards and Amos Storkey. Censoring representations with adversary. arXiv preprint arXiv:1511.0588 2015.
Dominik Maria Endres and Johannes E Schindelin, A new metric for probability distributions. IEEE Transactions Information theory, 2003.
10

---

Under review as a conference paper at ICLR 2020
Miro Enev, Jaeyeon Jung, Liefeng Bo, Xiaofeng Ren, and Tadayoshi Kohno. Sensorsift: balancing sensor data privacy and utility in automated face understanding, In Proceedings of the 28th Anmual Computer Security Applications Conference, pp. 149.-18. ACM, 2012.
Ülfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. Rappor: Randomized aggregatable privacy- preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference computer and communications security, pp. 1054-1067. ACM, 2014.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference Computer and Communications Security, pp. 1322-1333, ACM, 2015.
Yaroslav Ganin, Bvgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learming Research, 17(1):209-20330, 2016.
Ran
Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International Conference on Machine Learning. pp. 201-210, 2016.
Neil Zhenqiang Gong and Bin Liu. You are who you know and how you behave: Attribute inference attacks via users' social friends and behaviors. In 25th {USENIX) Security Symposium (USENIXX Security 16), pp. 979-995, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural informa- fion processing systems, pp. 2672-2680, 2014.
Jihun Hamm. Minimax filter: Learning to preserve privacy from inference attacks. The Journal of Machine Learning Research, (((1):470447344 2017.
Moritz Hardt, Eric Price, Nati Srebro, et al. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3315-3323, 2016.
Jianhua Lin. Divergence measures based the Shannon entropy. IEEE Transactions on Information Theory, 37(1)1445-15,, 1991.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv: 2016
HhatHai Phan, Xintao Wu, Han Hu, and Dejing Dou. Adaptive laplace mechanism: Differential privacy preservation in deep learning. In 2017 IEEE International Conference on Data Mining (ICDM), pp. 385-394. IEEE, 2017.
Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. Mlaas: In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. IEEE, 2015.
Machine learning as a service.
Proteek Chandan Roy and Vishnu Naresh Boddeti. Mitigating information leakage in image repre- sentations: maximum entropy approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Jacob Whitehill and Javier Movellan. Discriminately decreasing discriminability with learned image filters. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2488-2499 IEEE, 2012.
Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. Towards preservin visual recognition via adversarial training: A pilot study. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 606-624, 2018.

---

Under review as a conference paper at ICLR 2020
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 165.v:1005.0714. 2016.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/eeressionn by conditional adversarial autoencoder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5810-5818, 2017.
Han Zhao and Geoffrey J Gordon. Inherent tradeoffs in learning fair representations. In Advances in neural information processing systems, 2019.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference Machine Learning, pp- 7523-7532, 2019.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
In this appendix we provide the missing proofs of theorems and claims our main paper. We also describe detailed experimental settings here.
A TECHNICAL TOOLS
In this section we list the lemmas and theorems used during our proof.
Lemma A.1 (Theorem 2.2, (Calabro 2009)). Let Hz (s) be the inverse binary entropy function for E [0, 1), then Hs) 2
(s)
s/2lg(6/s)
Theorem A.1 (Data processing inequality). Let X T Z, then (X; I(X;Y)
B MISSING PROOFS
Proposition 2.1. Let h X - {0, be a hypothesis, then Priv (h) 1 iff T(h(X); A) 0 and almost surely.
(h) = iff ((X)= = almost surely or ((X)
Proof. We first prove the first part of the proposition. By definition, Priv, iff Pro (h (r = A = 0), which is also equivalent to h(X) 1 A. It then follows that M(X + (() A) =
1
= -
For the second part of the proposition, again, by definition of Priv A (h), it is clear to see that we either have Pr = 1=
and Pro(h(X) = A 0) = O, or (r A 1) 0 and (h(() = A 0) 1. Hence we discuss by these two cases. For ease of notation, we omit the subscript D from Prp when is obvious from the context which probability distribution we referring to.
If Pr()(X) = = 1) and Pr(M(X) = A 0) = 0, then we know that:
Pr(h(X) A) Pr(A 0) Pr(h A 0) Pr(A= Pr * A=
= Pr(A
=Pr(A 0) - -+ 1)d = 0 A = 1)
=.
2. IfPr(h(X)= = A- 1) = 0 and Pr(h(X) 1 1,, similarly, we have:
Pr(h(X) -1 Pr(A (A- 0) 1 - Pr 1 -1
Pr(A = 0)
=1 4=1)
Pr(A = 0) 0 Pr(A=
0.
Combining the above two parts completes the proof.
Proposition 2.2. is symmetric, then x minkeN Pr(M(X) A 1)
Pr(h(X)= A 0).
Proof. By definition, we have:
(H) == min Priv (h) kEH
min Pr((h(X) = A Pr(n(X) = 1 A 0)
min 1 (Pr(h(X)= = 1| 14 1) Pr(M(X) = A 0))
KEW
min Pr(h(X) 0 1 1) + Pr(h(X) 1 A 0), kER
where the third equality holds due to the fact that max Pr(h(X) = 1 1) Pr(h(X)= = Pr(h(X) 1 A 0)). To this, for any
1 A 0)| max ( (x)(h(X) specific h such that the term inside the absolute value is negative, we can find / E H. such that
becomes positive, due to the assumption that H is symmetric.
13

---

Under review as a conference paper at ICLR 2020
Proposition 3.1. Let f X' 2 be deterministicc function and H. C 2 be hypothesis class over 2. For any joint distribution D over X, I(f(X):AA = 0, then Priv
Proof. First, by the celebrated data-processing inequality, Vh E H:
S [((((((X)): A) s T(((X);; A) =0 By Proposition 2.1 this means that v/ H. Priv a (h) 1, which further implies that Priv A (Hof) f)
1 by definition.
Lemma 3.1. For any feature map f ; X + Z, assume that H. contains all the randomized binary classifiers, then o and h" (Z) == arg min CEy (h f) Pr(Y
f(X)).
Proof. Let D' be the induced (pushforward) distribution of 'D under the map f : X + Z. By the
definition of cross-entropy loss, we have:
Chy f) ED [T(Y 0) log(1 ((f(x))) + = -1) log
- ED [I(Y = 0) log(1 h(2)) + +I[(Y = 1) [og[h(Z))))
EyyEyiz [I(Y 0) log(11 1os(1-1/42 -(Z)) + =I( l0loollo()()
--Ez (Pr(Y Z) - - -Pr(Y =1
Z) log(h(Z)))
Ez [Dx Z) [[2)] + HY ()
H(Y Z).
It is also clear from the above proof that the minimum value of the cross-entropy loss is achieved when h(Z) equals the conditional probability Pr(Y = 1 2). i...e.. ht' (2) Pr(Y = Z = f(X)).
Theorem 3.1. Let f be the optimal feature map of (4) and define H := H(A | Z = (x)).
Then for any adversary Ā: Z + {0, 1) such that Á - A Z, pr (Ā + A) 2 H* / lgg(///H*)
Proof. To prove this theorem, E be the binary random variable that takes value iff + Z, i.e.
E = T(A * A). Now consider the joint entropy of A, Å and E. On one hand, we have:
((A,A, = + H(F A, A) H(A, A) + 0 (AA A) H(A)
Note that the second equation holds because E is deterministic function of A and A, that is, once A and Å are known, E is also known, hence (EE) A) 0. On the other hand, we can also decompose H(A, A, F.) follows:
H(A, A, E) = H(E)+ H E) + ((A A,E).
Combining the above two equalities yields
H(E) (AA) E) H(Á (AAAA H(A Å) (()).
Furthermore, since conditioning cannot increase entropy, we have (Á A) S H(A), which further implies
+ H(A) E). H(A A) < (
Now consider H(A) F). Since A {0, 1}. by definition of the conditional entropy, we have:
((A) E) Pr(E- = (A E = 1) + Pr(E 0) ((A) E 0
To lower bound (AA A), realize that
[(A; A) + H(A A) = H(A) = [(A;2) + ((A Z).
Since À randomized function of Z such that À Z, due to the celebrated data-processing inequality, we have [AA; A) & [(A; which implies
((A A) 2 H(A Z).
14

---

Under review as a conference paper at ICLR 2020
Combine everything above, have the following chain of inequalities hold:
H(A) 2) < (AA) A) H(B) - H(A E) = H()),
which implies
Pr (A + A) D
Pr (E= 1) 2 (H(A) 2)); Dr
where (-) is the inverse function of the binary entropy [[( := -t logt (1 t) (1 -ogg111) t) when & E [0, 1]. To conclude the proof, we apply Lemma A.1 to further lower bound the inverse
binary entropy function by
(H 2)) 2 H(A 20/2lg(6/H(4 2)),
completing the proof.
Lemma 4.2. Let Do and Dj be two distributions over X' and let D and Df be the induced distribu- tions of Do and D1 over by function f. then dDs(D)) D S (D, D1).
Proof. Let B be uniform random variable taking value in (0. }} and let the random variable ZB with distribution Dí (resp. X with distribution Dn) be the mixture of Dí and D' (resp. Do and Di) according to B. It is easy to see that Dn (Do + /2, and we have:
I(B; xB) = (X H(x
B)
Σ DB log DB
Σ lo Do + Σ D2 Dog
+
1 Σ Do log DB
1
Σ Dr log DB
Σ Do log Do + Σ D1 log log
2
2
2
Σ Do log
Do
Σ
Dj log DB
D1
2
DB
2
DxI. (D))
+
(Ri DT)
2
Dis(Do. Di).
Similarly, we have:
Dis(D), D/) I(B;
Since D' (resp. D is induced by f from Do (resp. D1), by linearity, D/g is also induced by f from
DB. Hence Z f(XB) and the following Markov chain holds:
B Xn - Zn.
Apply the data processing inequality, have
Das (Do D1) T(B; Xn) 2 [[:; Rn) = Dis (D/ D(). Taking square root on both sides of the above inequality completes the proof.
Lemma 4.3. Let Ý M((()) E {0, be the predictor, then for a E (0, 1}, dis(D) Dhor) S
/ /EEI, f).
Proof. For a E (0,1), by definition of the JS distance:
dss Dhof) = Dis(D)' DDorf
< IID) - -Dhot 1l2/2
(Lemma 4.1
((Pr(Y
A a) Pr(k([(X)) - a)
+ +|rr(Y = A P
= A a))) /2
Pr(Y =1|A= a) ||
= EE [A=a] (c]EE(b(f()) AA
S LE[JY - h(f(x))) A=
= Err, (h o f),
where the expectation is taken over the joint distribution of X, Y. Taking square root at both sides then completes the proof.
15

---

Under review as a conference paper at ICLR 2020
Theorem 4.1. Let H C 2 contains all the classifiers from 2 to {0, 1). Given the conditions in Lemma 4.4. vh E H. Utilg(h o f) (hi f) + (H f) I 3 ( D.
Proof. Before we delve into the details, we first give a high-level sketch of the main idea. The proof could be basically partitioned into two parts. In the first part, we will show that when contains all the measurable prediction functions,
Priv a (H f' could used to upper bound (is(D) D.). The second part combines Lemma 4.3 and Lemma 4.2 to complete the proof.
In this part we first show that Djs (D, D() = Priv a f):
Dis(D), D/) - =/110// Dflli
= =dv((D/, D
sup Dó AE
(A) (A) (A)).
where d denotes the total variation distance and H is the sigma algebra that contains all the measurable subsets of Z. On the other hand, when contains all the measurable functions in 2 we have:
1-Priv (H min (1 (P)(h( 1 0) - NEW
max
Pr(h(2) = A A= - Pr(h(2)
NEW.
max [Do(h- -ihh))))
NEW
sup [Do(A) - (A))),,
AE3
where the last equality follows from the fact that H. is complete and contains all the measurable functions. Combine the above two parts we immediately have Dis (P/ D A (H o f).
Now using the key lemma, we have:
dss (D) . DY) s dis dis(D) Doo0) + dDs(D//,)))) Df) + dis dD D
v Erro(h f) v1 A f) + Err,(ho
= Utilg f) + v - Priv , o f) + v (h
V 3(1 Utilg + Util,
(H f))
3(3 (Utilo(h o + Util, (h f) + Priv f)))-
Taking square at both sides and then rearrange the terms then completes the proof.
Theorem 4.2. Assume the conditions in Theorem 4.1 hold. If Priv A (H o f) 2 1- Dis(D) D then Vh H, Erro(h f) + Err, (h o f) 2 (dss D D - /I Priv (H f))²
Proof. Similarly, using the key lemma, we have:
dis(D) Df s dis(D) D + dis D1) + dis (D 1 )
/EEro (h f)+ + /I Priv f) + EErz,
Under the assumption that Priv (H f) 2 1 Dis(D) DY), we have dis(D) DY) 2 VI o f), hence by AM-GM inequality:
2 (EErooh + Erri (h ())
v Erro(h f) + Em Erri f) dis(D)) DK) V1 P f).
Taking square both sides then completes the proof.
16

---

Under review as a conference paper at ICLR 2020
C MULTI-ATTRIBUTE DEFENSE
Although our discussion in the paper only focuses on the case where there is only one sensitive attribute that the data vendor would like protect, our optimization framework is flexible enough to extend to the setting where multiple sensitive attributes need to be preserved simultancously. For instance, in online advertising, the data vendor often needs to keep the personal information about specific users secret, e.g., age range, demographic group, income level, etc. To this end, let {A, be K sensitive attributes that the data vendor would like protect.
Define E; := Prs (M)()(X)) = 0 A; = 1) Prs (x(())(())) 1 A, = 0) to simplify the notation. Similar to the optimization formulation in ① the following general optimization formulation handles the setting of multi-attribun defense:
Hard version:
Err(h f) + À max min max NEH. MgE
=E[K]
Problem O6 is still a minimax optimization problem. we initialize all the functions, including using deep neural networks, then (6) turns into a nonconvex minimax optimization problem. Inspired by Ganin et al. (20T6), we can use the gradient reversal layer to effectively implement (6) by backpropagation. Essentially, with the gradient reversal layer, we use the Gradient Descent/Ascent (GDA) (Daskalakis Panageas 2018) algorithm to optimize all the model parame- ters, opposed the alternative gradient algorithm, which is known to be unstable in the nonconvex setting (Goodfellow et al. (014).
f. and ( ==
One notable drawback of (6) is that in each iteration, the forward evaluation phase requires computa- tion over all the K adversaries, while in the backward propagation phase only one of them is being utilized due the hard max operator. This is rather data-inefficient and can waste computational resources in the forward evaluation phase. To avoid this problem, we propose a smoothed formulation of (6) using the fact that y log EiE(K] exp(-7e.) + |ax (-E:) as 7 0o:
Smooth version:
min KE
max Mg EH
Err(h o f) + log KE[K]
exp(- --E))
We call the one in (6) as hard version multi-attribute defense and 7 as the smooth version multi- attribute defense. Let A denote the model parameters of f. Take the derivative w.r.t. 0 we have:
1 log Σ 00 7
exp(->e))= - Σ exp(-7Es) Oe,
xE[K] Σ ]jE[K] exp(-nE)) 00
[E[K]
Compared with the hard version, the smooth version not only avoids the data-inefficiency problem, but also provides an adaptive way to combine the feedback from all the K adversaries by convex combination. Intuitively, the above formulation suggests that during optimization, the larger the error from one adversary, the smaller the combination weight in the ensemble. This is consistent with Proposition 2.2 where we can see that larger error essentially corresponds a better protection of the corresponding sensitive attribute, hence smaller combination weight.
To demonstrate that the effectiveness of multi-attribute protection, we also evaluate both the hard and smooth versions of our multi-attribute defense on UTKFaae dataset. We compare both versions with no defense for both attributes and defenses for single attribute, as none of the other competitors has a multi-attribute defense extension. The results shown in Figure
In Figure 4 we can see that both the hard and smooth variants help to protect two private attributes, indicated by the low private accuracies in both cases. Notably, the target accuracy does not degrade too much as compared to the one in single-attribute defense. Among the two variants, the smooth variant is slightly more effective, possibly due to the adaptive combination property.
D DETAILED EXPERIMENTS
In this section, we provide more details of the experiments. First provide the details of different existing methods we evaluate. Then we elaborate more dataset description, model architecture and training parameters in different experiments.
17

---

Under review as a conference paper at ICLR 2020
target Accuracy
Privete Accurscy (Race)
Private Accuracy (Age)
09.4
09.4
89.4
300
000
Accurscy'
60
30C
$7.
000
00
01
000
400
00c
100
00
20C
000
100
300
00C
(00
100
000
40
100
008
396
00
96
300
000
000
000
000
100
100
200
000
100
000
000
00
000
000
00
00
000
000
100
000
000
100
000
000
100
000
000
No defense
Race Race Only
Protect Age
Hard Variant
Smooth Varkent
Only
Figure 4: Classification accuracy of multi-attribut defense on the UTKFace dataset. For target accuracy, the larger accuracy the better. For private accuracy, the smaller accuracy the better.
D.1 DETAILS ON METHOSS
We provide detailed description of each method here:
1). Privacy Partial Least Squares (PPLS): It learns x Xa matrix for the feature transformation. The matrix is learned by maximizing the covariance of the learned representation and target attribute while minimizing the covariance of the learned representation and private attribute.
2). Privacy Linear Discriminant Analysis (PLDA): It learns X Xd matrix for the feature trans- formation. The matrix is learned by maximizing the Fisher's linear discriminability of the learned representation and target attribute while minimizing the Fisher's linear discriminability of the learned representation and private attribute.
3). Minimax filter with alternative update (ALT-UP): The representation learn via optimizing Equation 3 in an alternative way, first update the parameters of the feature transformation module and the target attribute classifier, and then accordingly update the private attribute classifier.
4). Maximum Entropy Adversarial Representation Learning (MAX-ENT) (Roy & Boddeti] (019): he objective equation is the slightly different from ALT-UP. The latter term contains additional entropy term to maximize unpredictability of the private attribute.
5). Gradient Reversal Layer (GRL): The objective equation is the same as ALT-UP, and we train the feature transformation module by adding gradient reversal layer between the feature transformation module and the private attribute classifier.
6). Principal Component Analysis (PCA): It generates Xa matrix for the feature transformation where the rows of the matrix are the n largest eigenvectors of the input dataset X.
7). No defense (NO-DEF): It is equivalent to normal training by setting À 0 in Equation
8). Local Differential Privacy (LDP): Standard Laplace mechanism of local differential privacy, where the noise is added to the raw representation for erasing the information of the private attribute.
9). Differentially private SGD (DPSGD) (Abadi et al. 2016): It is one of the state-of state-of-the--a the-art differential privacy methods deep learning. It adds Gaussian noise the gradients when training the model.
D.2 DETAILS ON UCI ADULT DATASET EVALUATION
UCI Adult dataset is benchmark machine learning dataset for income prediction. Each data record contains 14 categorical or numerical attributes, such as occupation, education and gender, to predict whether individual annual income exceeds $50K/year. The dataset is divided into training set (24130 examples), validation (6032 examples), and test set (15060 examples). We choose gender, age, and education as the private attributes, respectively.
18

---

Under review as a conference paper at ICLR 2020
Table 2: Data distribution of income (Y) and gender (AA) in UCI Adult dataset.
Table 3: Data distribution of income (Y)
and age (A) in UCI Adult dataset.
Y = 0
Y =
Y =
Y = 1
A = 0
20988 13026
9539
A = 0
18042 15972
2473
1669
A 1
8735
Table 4: Data distribution income (Y) and education (A) in UCI Adult dataset.
Y ==0 0
Y=
A 20447 A = 13567
4248
6960
We process each private attribute as binary label for each experiment: for age label, 0 if the person is no greater than 35 years old and I otherwise; for education label, 0 if the person has not entered college or receive higher education than college, and otherwise. the mean time, we also remove corresponding private attribute from the input, so the dimension of input data for each experiment is different. The input dimensions for income-gendel experiment, income-age experiment, and income- education experiment are 113, 104 and 99, respectively. Table 2. Table 3 and Table 4 summarize the data distribution of UCI Adult dataset for protecting different private attributes.
We use the two-layer ReLU-based neural net for f and one-layer neural net for h. The output dimensions of f are 64. We train all methods using SGD with the initial learning late 0.001 and momentum 0.9 for 40 epochs. In the DP-SGD experiment, 4.0 for small noise and large noise, respectively, and set the clipping norm as 1.0. 5) for DPSGD small noise and DPSGD large noise are (33.7, 10 and (0.572, 10 10-5), respectively. Among all methods, we report the one achieving the best performance on the target task in the validation set. We run the experiments for ten times and compute the average.
the noise multiplier as 0.45 and
D.3 DETAILS ON UTKFACE DATASET EVALUATION
UTKFace dataset is a large scale face dataset with annotations of age (range from 0 to 116 years old), gender (male and female), and ethnicity (White, Black, Asian, Indian, and Others). It contains 23,705 64 x 64 aligned and cropped RGB face images and we split the dataset into training set (15171 examples), validation set (3793 examples) and test set (4741 examples), respectively. We further process age label and ethnicity label as binary labels: 0 if the person is not greater than 35 years old for age label (is white for ethmicity label), and if the the person is greater than 35 years old for age label (is non-white for ethnicity label). 3 and Table Jummmarize the data distribution of UTTe dataset for protecting different private attributes.
Table 5: Data distribution of gender (Y) and race (A) in UTKFace dataset.
Table Data distribution of gender (Y)
and age (A) in UTKFace dataset.
Y 0
Y=1
Y= 0
Y =
A 0
5477
4601
A = 0
6889 5502
8218
A =
6914
6713
A = 1
3096
Since NO-DEF, ALT-UP, GRL and DP can directly enjoy the benefits of using the state-of-the-arr neural network architecture feature extraction module,
use the feature extraction module of Wide Residual Network (Zagoruyko & Komodakis 2016) for the (non-linear) feature transformation module, while PPLS. PLDA, and PCA learn 12288 2048 matrix filter for f. We train all methods using SGD with the initial learning late 0.01 and momentum 0.9 for 50 epochs. The learning rate is decayed by factor of 0.1 for every 20 epochs. In the DP-SGD experiment, we set the noise
19

---

Under review as a conference paper at ICLR 2020
multiplier 0.45 and 1.0 for small noise and large noise, respectively, and set the clipping norm as 1.0. (c, for DPSGD small noise and DPSGD large noise are (25.7, 10 and (2.7, 10 10-9), respectively. Among all methods, report the achieving the best performance the target task in the validation set. We run the experiments for ten times and compute the average.
For the experiment of multi-attribute defense, choose À to be 3 for both hard and smooth variants. We find that this a reasonable choice of since À cannot be too large (otherwise it will cause gradient explosion during training) and cannot be too small (otherwise the private task accuracy is still too high) in this learning task. All other parameter settings are the same as the ones described before.
20


Reviews:
[{"review_id": "rkxK9f-suB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nIn this paper, the author introduces a new privacy notation for the attribute attacks. Under the notation, the author has theoretically analyzed the trade-off between privacy preservation and model utility. Based on the theoretical finding,\nthey further propose an adversarial representation learning paradigm to achieve the best trade-off.  \n\nStrengths:\n1. This paper provides an interesting information-theoretic view to study the privacy-preserving machine learning algorithm.\nBased on this view, the author has presented a comprehensive analysis of the trade-off between privacy-preserve and model\nutility.\n2. The topic of studying the privacy guarantee against attribute attacks is important and implies a wide range of applications, such as preventing the model-inversion attack. \n3. The paper is well-written and provides an enjoyable reading experience. \n\nWeakness:\n1. In experiments, the DP based method should be an important comparison method. However, the DP method used in this paper\nseems to be a weak baseline, which injects the noise into the raw data. Many prior works have prooved that injecting noise\ninto the gradient leads to a better trade-off between the utility and the privacy budget [1]. Thus, the author should re-design their DP baseline and provides comparison results.\n2. There is a related work [2] of reducing the privacy leakage of the feature representation, which also takes a view from the information-theoretic view (i.e. a maximum entropy approach). Although this work focuses on the task of image representation, the author also conducts experiments on a non-image dataset, i.e, UCI dataset. The comparative experiments need to be conducted to show the effectiveness of the proposed method.\n3. The notation of the distribution is confusing. It seems that the author referred $\\mathcal{D}$  to both three joint distributions?\n\n[1]  Abadi et al. Deep Learning with Differential Privacy\n[2] Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyxrliSJcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this paper provides valuable insight into the trade-off between privacy preservation and utility when training a representation to fight against attribute inference attack.\n\nDetail comments:\n\nStrength:\n+ The whole paper is well organized with logic. Notations are well defined and distinguished.\n+ The final results have a good intuitive explanation.\n+ The most impressive part of this paper is the analysis of the trade-off between privacy and utility from which the upper bound is quantified.\n\nWeakness:\n+ The minimax method looks trivial. The difficulty of using such an objective should be emphasized for practical implementation.\n+ The major weakness is the experiments. The experiments only on two datasets may not be convincing for me. And the repetition times, 5 or 3, for each dataset are pretty small. Considering the experiments are conducted with random noise, e.g., DP methods, such a small repetition time is not fair since there a large chance these results could be selected.\n+ Which DP Laplacian mechanism is used is not specified. Since there are already many improvements on the DP Laplacian mechanism, e.g., [A], it is necessary to make sure the baseline should be state-of-the-art for a fair comparison.\n+ The result is not intuitively surprising that the privacy loss and utility loss will be balanced toward an upper bound (Theorem 4.2). I am not sure how the Jensen-Shannon entropy between D^Y_0 and D^Y_1 can be calculated in practice since the true conditional distribution is not observed. For example, when the data distribution is heavily biased, then the conditional distribution might show less correlation between Y and A. Then, the privacy protection will be a pretty simple task with a very high upper bound. Based on this, it is worth to ask what the upper bound looks like in the dataset used in the experiments.\n+ How efficient this algorithm could be? In comparison to other baselines, does this method provide a more efficient solution?\n\n[A] Phan, N., Wu, X., Hu, H., & Dou, D. (2017). Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning. 2017 IEEE International Conference on Data Mining (ICDM), 385–394. https://doi.org/10.1109/ICDM.2017.48", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Hyg8YbzRFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a definition of privacy against attribute inference attack and formulates it as a minimax optimization problem that balances accuracy and privacy, it then provides an information-theoretic lower bound for the inference error. It also provides an analysis of the intrinsic privacy-utility tradeoff. Finally, it shows experimental evaluation in terms of inference error and accuracy of several representation-learning based approaches for protecting against attribute inference attack and compares with the lower bounded proved in the paper. By comparing utility with a differentially private algorithm, it shows that the other algorithm achieves higher utility under similar privacy.\n\nThe formulation of the minimax problem seems pretty interesting and the information-theoretic bound seems pretty useful in measuring inference risk based on the experimental results. \nAs for the comments and experiments with differential privacy, I’m not sure if I follow the argument correctly. It seems to me DP is stronger than the privacy guarantee defined in the paper, as it prevents an adversary from knowing whether record x or x’ is in the dataset, which I think directly implies that an adversary won’t be able to infer whether a record with attribute A or A’ (with the rest of the attributes kept) is in the dataset. Can you comment more on that side? Besides, there were two epsilon values used in the experiments for DP. As your goal is not to guarantee DP anyway, I think it makes sense to try more epsilon values, especially much larger epsilon values, as the analyzed epsilon is sometimes pretty loose upper bound.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was some support for the ideas presented, the majority of reviewers felt that this submission is not ready for publication at ICLR in its present form.

The most significant concerns raised were about the strength of the experiments, and choice of appropriate baselines.
Decision: Reject