Alleviating Privacy Attacks via Causal Learning
Authors: Shruti Tople
Amit Sharma
Aditya Nori
Venue:  (2020)
Keywords: Causal learning
Membership Inference Attacks
Differential Privacy

Abstract: Machine learning models, especially deep neural networks have been shown to reveal membership information of inputs in the training data. Such membership inference attacks are a serious privacy concern, for example, patients providing medical records to build a model that detects HIV would not want their identity to be leaked.  Further, we show that the attack accuracy amplifies when the model is used to predict samples that come from a different distribution than the training set, which is often the case in real world applications.   Therefore, we propose the use of causal learning approaches where a model learns the causal relationship between the input features and the outcome. Causal models are known to be invariant to the training distribution and hence generalize well to shifts between samples from the same distribution and across different distributions. First, we prove that models learned using causal structure provide stronger differential privacy guarantees than associational models under reasonable assumptions. Next, we  show that causal models trained on sufficiently large samples are robust to membership inference attacks across different distributions of datasets and those trained on smaller sample sizes always have lower attack accuracy than corresponding associational models. Finally, we  confirm our theoretical claims with  experimental evaluation on 4 datasets with moderately complex Bayesian networks. We observe that neural network-based associational models exhibit upto 80% attack accuracy under different test distributions and sample sizes whereas causal models exhibit attack accuracy close to a random guess. Our results confirm the value of the generalizability of causal models in reducing susceptibility to privacy attacks.

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
ALLEVIATING PRIVACY LEARNING
ATTACKS
VIA
CAUSAL
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning models, especially deep neural networks have been shown to reveal membership information of inputs in the training data. Such membership inference attacks are a serious privacy concern, for example, patients providing medical records to build model that detects HIV would not want their identity be leaked. Further, we show that the attack accuracy amplifies when the model is used to predict samples that come from different distribution than the training set, which is often the case in real world applications. Therefore, we propose the use of causal learning approaches where model learns the causal relationship between the input features and the outcome. An ideal causal model is known to be invariant the training distribution and hence generalizes well to shifts between samples from the same distribution and across different distributions. First, we prove that models learned using causal structure provide stronger differential pri- vacy guarantees than associational models under reasonable assumptions. Next, we show that causal models trained on sufficiently large samples are robust to mem- bership inference attacks across different distributions of datasets and those trained on smaller sample sizes always have lower attack accuracy than corresponding associational models. Finally, confirm our theoretical claims with experimental evaluation on 4 datasets with moderately complex Bayesian networks. We observe that neural sewwork-based associational models exhibit upto 80% attack accuracy under different test distributions and sample sizes whereas causal models exhibit attack accuracy close to a random guess. Our results confirm the value of the generalizability of causal models in reducing susceptibility to privacy attacks.
INTRODUCTION
Machine learning algorithms, especially deep neural networks (DNNss have found diverse appli- cations in various fields such as healthcare (Esteva et al. 2019), gaming (Mnih et al.] 2013). and finance (Tsantekidis et al. 2017 Fischer & Krauss 2018 However, line of recent research has shown that deep learning al algorithms are susceptible to privacy attacks that leak information about the training dataset (Fredrikson et al. 2015 Rahman et al. 2018 Song & Shmatikov 2018 Hayes et al., 2017). Particularly, one such attack called membership inference reveals whether a particular data sample was present in the training dataset (Shokri et al. 2017). The privacy risks due to membership inference elevate when the DNNs are trained sensitive data such as in healthcare applications. For example, patients providing medical records to build model that detects HIV would not want to reveal their participation in the training dataset.
Membership inference attacks are shown to exploit overfitting of the model on the training dataset Yeom et al., 2018). Existing defenses propose the use of generalization techniques such as adding learning rate decay, dropout or using adversarial regularization techniques (Nasr et al. 2018b Salem al.) 2018). All these approaches assume that the test data from the same distribution as the training dataset. In practice, a model trained using data from one distribution is often used on (slightly) different distribution. For example, hospitals in one region may train a model detect HIV and share it with hospitals in different regions. However, generalizing to new context is challenge for any machine learning model. We extend the scope of membership privacy to different distributions and show that the risk from membership attack increases further DNNs the test distribution is changed. That is, the abiltity of an adversary to distinguish a member from a non-member improves with change in test distributions.
To alleviate privacy attacks, we propose using models that depend on the causal relationship between input features and the output. Causal learning has been extensively used to guarantee faimess and

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
explainability properties of the predicted output (Kusner al. 2017 Nabi & Shpitser 2018 Datta (et 20166. However, the connection of causal learning to privacy is yet unexplored. To the best of our knowledge, we provide the first analysis of privacy benefits of causal models. By definition, causal relationships are invariant across input distributions (Peters et al., 2016), and hence make the predictions of causal modely independent of the observed data distribution, let alone the observed dataset. Hence, causal models generalize better even with change in the distributions.
In this paper, we show that the generalizability property of causal models directly ensures better privacy guarantees for the input data. Concretely, we prove that with reasonable assumptions, a causal model always provides stronger (i.e., smaller e value) differential privacy guarantees than a corresponding associational model trained on the same features and the same amount of added noise to the training dataset. Consequently, we show that membership inference attacks are ineffective (equivalent to random guess) on causal models trained on infinite samples. Empirical attack accuracies on four different datasets confirm our theoretical claims. We find that 60K training samples are sufficient to reduce the attack accuracy of causal model to a random guess. In contrast, membership attack accuracy for neural network-basee associational models increase as test distributions are changed. The attack accuracy reaches nearly 80% when the target associational model is trained on 60K training samples and used to predict test data that belong to a different distribution than the training data. Our results show that causal learning approaches direction for training models on sensitive data. Section 2 describes the properties of causal models. Section proves the connections of causality to differential privacy and robustness to membership attacks. Section provides empirical results. To summarize, we make the following contributions:
a promising
For the same amount of added noise, models learned using causal structure provide stronger c-differential privacy guarantees than corresponding associational models. Causal models are provably more robust to membership inference attacks than typical associational models such as neural networks.
We simulate practical settings where the test distribution may not be the same the training distribution and find that the membership inference attack accuracy of causal models is close to a "random guess" (i.c., 50%) while associational models exhibit upto 80% attack accuracy.
2 PROPERTIES OF CAUSAL MODELS
Causal models are shown to generalize well since the output of these models depend only on the causal relationship between the input features and the outcomes instead of the associations between them. From prior work, we know that the causal relationship between the features is invariant to the their distribution (Peters et al. 2016). Using this property, we study its effects on the privacy of data.
2.1 BACKGROUND: CAUSAL MODEL
Intuitively, causal model identifies a subset of features that have a causal relationship with the outcome and learns a function from the subset to the outcome. To construct a causal model, one may use structural causal graph based on domain knowledge that defines causal features as parents of the outcome under the graph. Alternatively, one may exploit the strong relevance property from Pellet & Elisseeff (2008), use score- score-based learning algorithms (Scutari 2009) or recent methods for learning invariant relationships from training datasets from different distributions (Peters et al. 2016)] Bengio et al. 2019), or learn based on a combination of randomized experiments and observed data.
Note that this is different from training probabilistic graphical models, wherein an edge conveys an associational relationship. Further details on causal models are in Pearl (2009); Peters et al. (2017).
For ease of exposition, we assume the structural causal graph framework throughout. Consider data from a distribution (X, Y) ~ P where X is a k-dimensional vector and Y E {0, 1}. Our goal to learn a function h(X) that predicts Y. Figure [ shows causal graphs that denote the different relationships between X and Y. Nodes of the graph represent variables and a directed edge represents a direct causal relationship from a source to target node. Denote Xpa S X, the parents of Y in the causal graph. Figure D shows the scenario where X contains variables Xço that are correlated to Xpa in P. but not necessarily connected to either Xpa or Y. These correlations may change in the future, thus a generalizable model should not include these features. Similarly, Figure Ib shows parents and children of Xpa- The d-separation principle states that a node is independent of its ancestors conditioned on all its parents (Pearl 2009). Thus, Y is independent of Xşı and Xg2 conditional on Xpa- Therefore, including them in a model does not add predictive value (and further, avoids problems when the relationships between Xşı and Xg2 may also change). Finally, for
2

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
completeness, the exhaustive set of variables to include is known as the causal Markov Blanker" Xe which includes Y's parents, (Xpa), children (Yat) and parents of children. Conditioned on its Markov blanket (Figure [T]. Y is independent of all other variables in the causal graph. When Y has no descendants in the graph, then the effective Markov blanket includes only its parents, Xpa-
The key insight is that building a model for predict- ing Y using the Markov Blanket Xc ensures that the model generalizes to other distributions of X, and also to changes in other causal relationships between X, as long as the causal relationship of Xç to Y is stable. We call such a model as causal model, the features in (Xc) as the causal features, and assume that all the causal features for Y are observed. In contrast, we call a model that uses all available features an associational model.
Xst
Markoy
Blacket
Figure 1: A causal predictive model includes only the parents of Y (a) and (b). Panel (c) shows the generalization to Markov Blanket.
2.2 GENERALIZATION TO NEW DISTRIBUTIONS
We state the generalization property of causal mod- els and show how it results in stronger differential privacy guarantee. We first define In-distriben ion and Out -of widdisrribuutt refers to the loss single input and [p(,,.) .) EpI(.,-) refers to the expected value of the loss over a distribution P(X, Y). We refer f : X + Y as the ground-truth labeling function and h X - Y as the hypothesis function or simply the model. Then, L(h, h() any loss function quantifying the difference between two models h and h'.
generalization error. Throughout, L(...)
Definition 1. In-Distribution Generalization Error (IDE). Consider a dataset S ~ P(X. Y). Then for model h : X + Y trained on S, the in-distribution generalization error is given by:
IDEp (h, f) = (b, f) fb
Definition 2. Out-of- Diittiiiiiiiitiionn Generalization Error (ODE). Consider a dataset S sampled from a distribution P(X. Y). Then for a model h X - Y trained on S, the oft----distributtoo generalization error with respect another distribution P" (X. Y) is given by:
ODE,,P. p. (b, f) = Cp. (h, f) Lemp(h, f)
(2)
Definition 3. Discrepancy Distance (discr.) (Def. 4 in Mansour et al. (2009). Let H be set of hypotheses, h : + Y. Let L Y x Y - R define a loss function over Y for any such hypothesis h. Then the discrepancy distance disc, over any two distributions P(X. Y) and p* Y) is given by:
discr PP) = max |Cp(b,b') - (h, b.
(3)
Intuitively, the term d P") denotes the distance between the two distributions. Higher the distance, higher is the chance of an error when transferring h from one distribution to another. Now, we will state the theorem on the generalization property of causal models.
Theorem 1. Consider structural causal graph G that connects X to Y, and causal features Xc where Xe a Markov Blanket Yf under G. Let P(X. Y) and Pa (X. Y) be two distributions with arbitrary P(X) and Pa (X) such that the causal relationship between Xe and Y is preserved, which implies that P(Y = P* (Y)xe.. Let Xx + Y be the resuliant invariant labelling fanction such that y = f (Xe). Further, assume that He represents the set of causal models be : Xx -> Y that use all causal features and Hg represent the set of associational models bg X - Y that may use all available feaiiaalleeature such that Ho C. H.A and f E Hc.
Then, for any symmetric loss function L that obeys the triangle inequality, the upper bound of ODE from dataset S P(X, Y) to P* (called (oE-Bound]] for causal model h. E He is less than or equal to the upper bound ODE-Bound of an associational model ha E H.. with probability at least (1 - 8)2.
ODE MEE-Boundpppp p- (bc, 5) < ODE DDE-BBundp.p- (h,, f;
(4)
We call it the cousal Markov Blanket since it is based on the structural causal graph, to distinguish it from the associational Markov Blanket that based on conditional probability distribution from Bayesian network.
3

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Proof. As an example, consider colored MNIST data distribution P where the classification task is to detect whether a digit is greater than 5, and where all digits above are colored with the same color. Then, under suitably expressive class of models, the loss-minimizing associational model may use only the color feature to obtain zero error, while the loss-minimizing causal model will still use the shape (causal) features obtain zero error. On any new P that does not follow the same correlation of digits with color, we expect that the loss- monimizii associational model will have higher error than the loss-minimizing causal model.
Formally, since P(Y Xc) P" (Y|xc), the optimal causal model that minimizes loss over P is the same as the minimiiiig model over P That is, hg hapr. However, for some associational models, h.P # hgp. and thus there is an additional loss term when generalizing to data from P The rest of the proof follows from triangle inequality of the loss function and the standard bounds for IDE from past work. Detailed proof is Appendix Section A.I
Corollary 1. Consider a causal model h. : Xe + Y and an associational model h2 : ++ trained on dataset S ~ P. Let (x. E S and (x',y) y') & S be two input instances such that they share the same true labelling function y = f {xxe) and y' = f(x)). Then, the worst-case generalization error for causal model on any such x' is less than equal to that for an associational model. [Proof in Appendix Section A.2
max xES 'y' (x())
Lx (be, LLx
max
Lx (ha, f) - Lz (ba,
(5)
xE8,x'r'=((xc,
3 MAIN RESULT: PRIVACY GUARANTEES WITH CAUSALITY
We now present our main result on the privacy guarantees and attack robustness of causal models.
3.1 DIFFERENTIAL PRIVACY GUARANTEES
Differential privacy (Dwork al. 2014) provides one of the strongest notion of privacy guarantees to hide the participation of an individual sample in the dataset. To state informally, it ensures that the presence or absence of single data point in the input dataset does not change the output by much.
Definition (Differential Privacy). A mechanism with domain I and range satisfies e-differential privacy for any two datasets d, d' E I that differ only in one input for a set S C O, the following holds: Pr(M(d) E S) S e Pr(M(d') E ES)
Based on the generalization property, we show that causal models provide stronger differential privacy guarantees than corresponding associational models. The standard approach to designing a differentially private algorithm is by calculating the sensitivity of that algorithm and adding noise proportional to the sensitivity. Sensitivity captures the change in the output of an algorithm due to the change in a single data point in the input. Higher the sensitivity, larger is the amount of noise required to make an algorithm differentially private with reasonable guarantees. We first provide the formal definition of sensitivity and then show that the sensitivity of causal models is lower than or equal associational models.
Definition 5 (Sensitivity (From Def. 3.1 in Dwork et al. (2014)) Let F. be a function that maps dataset to a vector in Rd Let S, S' be two datasets such that differs from S in one data point. Then the ss-seiiiyy of a function F is defined as: AF = maxs, [[F(() F(s')ll
Lemma 1. Let S be dataset over (X, Y) values, such that (()) fx:!)) V(z(), y E where is the true labelling function over the causal features Xc. Consider neighboring dataset s' such that v'))))))")))) & xx,,')) S. and (x', x shares the same causal labelling function y' = fx²)). Let model h be specified by a set of parameters 0 E Ω C BR.." Let hain 0s) be a model learnt using as training data and bai" (x: fg:)) be the model learnt using s' as training data. Then, the sensitivity of causal learning function F. that outputs learnt empirical hypothesis h t [S) and h + =F((S') ) is lower than or equal to the sensitivity of an associational learning function Fa that outputs has + Fa(S) and hain t F.(S') using a loss function L that is strongly convex over Q, symmetric and obeys the triangle inequality,
S' = S( y) + (x', y') where (x, E S. (x',
AF. max S.3'
bein max S.5'
hags hgin 111 A.F
where the maximum is over all such datasets and s'
4

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Proof. We can write the empirical loss minimizers for the datasets and s' as:
hg = arg min Cb f) = arg min
Lx, (h,
N
hg arg min (h, arg min
Lx, (h, f) + (h, (h, f)]
From Corollary for x' & and E S, we have:
max (he, f) (he f) S max (ha, f) Lx (ha, x,x'
(8)
x,x'
Since x E S and x' and [s s the above is true for any and S',
max Ls (be: f) Ch. f) max 8,8'
Ls (ha. f) La (ha.
(9)
8,5'
Since is strongly convex function over S2 and since Hc S HA = De S ΩA. has and h that minimize La (a, f) and Ls (h, respectively should also be closer each other than h and
(Boyd& & Vandenberghe 2004 (using Eqn. 9).
max 5,9'
- S max c,5
||
0 0
max S,8'
pgs b ll < max S,8'
2,5
min
(10)
8,8'
Hence, sensitivity of causal model is lower than an associational model i.e., AF AF.
Theorem 2. Let Fe and Fa be the differentially private algorithms corresponding to causal learning and associational learning algorithms Fe and Fa respectively. Let F. and Fa provide DD and DDDP guarantees respectively. Then, for noise sampled from the same distribution, Lap(Z) for both algorithms, we have E S Ex.
Proof. According to the Def. 3.3 of Laplace mechanism from Dwork et al. 2014), we have,
AFF.
AF
F. Lap(
F. ~ Lap(
(11)
The noise added the output the learning algorithm i.e., the model parameters. Since K. is sampled from the same noise distribution,
AF Lap(
AF,
AF.
AFa
= Lap(
(12)
66
E
From Lemma 1 AF.. AF, and hence ec
While we prove the general result above, our central claim comparing differential privacy for causal and associational models also holds true for models developed using recent work (Papernot al.) 2017) that provides a tighter data-dependent differential privacy guarantee. The key idea is to produce an output label based on voting from M teacher models, each trained on a disjoint subset of the training data. We state the theorem below and provide the proof in Appendix B Given datasets from different domains, the below theorem provides constructive proof to generate a differentially private causal algorithm, following the method from Papernot et al. (2017).
Theorem 3. Let be a dataset generated from possibly a mixture of different distributions PXX, Y) such that XxY remains the same. Let n, be the votes for the jth class from M teacher models. Let M be the mechanism that produces a noisy max, arg max, (n; + Lap(2/n)} Then the privacy budget for causal model is lower than that for the associational model with the same accuracy.
3.2 ROBUSTNESS TO MEMBERSHIP ATTACKS
Deep learning models have shown to memorize or overfit on the training data during the learning process (Carlini al. 2018). Such overfitted models are susceptible to membership inference attacks that can accurately predict whether a target input belongs to the training dataset or not (Shokri et al. 2017). There multiple variants of the attack depending on the information accessible to the
5

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
adversary. An adversary with access to black-box model only sees the confidence scores for the predicted output whereas one with the white-box has access to the model parameters and observe the output at each layer in the model (Nasr et al., 2018a). In the black-box setting, membership attack is possible whenever the distribution of output scores for training data is different from the test data, and has been connected to model overfitting (Yeom et al. 2018). For the white-box setting, if an adversary knows the true label for the target input, then they may guess the input to be member of the training set whenever the loss is lower, and vice-versa. Alternatively, if the adversary knows the distribution of the training inputs, they may learn a "shadow" model based on synthetic inputs and use the shadow model's 's output build a membership classifier for any new input (Salem et al.) 2018).
Most of the existing membership inference attacks have been demonstrated for test inputs from the same data distribution as the training set. When test inputs are expected from the same distribution, methods to reduce overfitting (such as adversarial regularization) can help reduce privacy risks (Nasr al. 2018b). However, in practice, this is seldom the case. For instance, in our example of model trained to detect HIV, the test inputs may come from different hospitals. Models trained to reduce the generalization error for specific test distribution are still susceptible to membership inference when the distribution of features is changed. This is due to the problem of covariate shift that introduces a domain adaptation error term (Mansour al.) 2009). That is, the loss-minimizing model that predicts Y changes with a different distribution, and thus allows the adversary to detect differences in losses for the test versus training Aa we show below, causal models alleviate the risk of membership inference attacks. From Yeom et al. (2018), we first define a membership attack as:
Definition 6. Let model h be truined on a datasel S(X, Y) of size N. Let A be an adversary with access to h and a input X. The advantage of an adversary in membership inference is the difference between true and false positive rate in guessing whether the the input belongs to the training set.
Adv(A, h) Pr[A- = 1 1] Pr[A 1 --bb00 0 o]
(13)
where b- = 1 if the input is in the training set and 0 otherwise.
Lemma 2. [From Yeom et al. [2018)] Let M be a e-differentially private mechanism based on model h. The membership advantage for an adversary is bounded by exp(e) - 1.
Theorem 4. Given a structural causal network that connects X to Y, let S ~ P(X, Y) be a dataset P. and let p* be any distribution such that P(Y P" (Y(Xc). Then, causal model he trained on S yields lower membership advantage than an associational model ha even when the test dataset is from different distribution P*.
sampled from from
Proof. From Theorem 2 above, we can construct an DD mechanism based on a causal model, and a ca CD mechanism based on an associational model, where c. S 6a. Further, this construction works for different input distributions. From Lemma 2 the membership advantage of an adversary A is,
Adv(A, hz) S exp(<=) - 1
Adv(A, ha) S exp((a) 1
(14)
Thus, worst case advantage for causal model is always lower than that of an associational model. D
Corollary 2. Let he be causal model trained using empirical risk minimization on a dataset S ~ Y) with sample size N. As N + oc, membership advantage Adv(A, hain) O.
The proof is the based the result from Theorem that h -ner for causal model. Crucially, membership advantage does not go to zero as N- de for associational models, since hoPT 4 OPT in general. Detailed proof is in Appendix Section
Attribute Inference attacks. We prove similar results on the benefits of causal models for attribute
inference attacks in Appendix Section D
4 IMPLEMENTATION AND EVALUATION
Table 1: Details of the henchmark datasets
Benchmark Datasets. To avoid errors in learning causal structure from data, we per- form evaluation datasets for which the causal structure and the true conditional probabilities of the variables known
Dataset
Child
Alarm BP
(Sachs)
Water
Output
XnayReport
Akt
CKNL 45
No. of classes
5
3
Nodes
20
37
11
Ares
25
46
17
66
Parameters
230
509
178
10083

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Accurtics
Arbck child datarer and Krayängen oupu
Atock accunscy for cid delaeet and Jauffiezert axinst
0.8
0.0
Aateck Jocurity
0.8-
C-Atro Teal
0.7-
0.-
Cieract Test.
0.6
0.5
0.5
0.5
1.5
TI5A Dutasot asaditer Tanget Moot
Athourt of rolos added tod doorbciioon
(b)
Figure 2: Results for Child dataset with XrayReport as the output. (a) is the target model accuracy. is the attack accuracy for different dataset sizes on which target model is trained and ( is the attack accuracy for test distribution with varying amount of noise for total dataset size of 100K samples.
from prior research. We select 4 Bayesian
network datasets- Child, Sachs, Alarm and Water that range from 230-10k parameters (T (bnl). Nodes represent the number of input features and arcs denote the causal connections between these features in the network. Each causal connection is specified using a conditional probability table P(X_|Par (aaarrrtts(( x=)); we consider these probability values as the parameters in our models. To create a prediction task, we select a variable in each of these networks as the output Y. The number of classes in Table I denote all the possible values for an output variable. For example, the variable BP (blood pressure) in the alarm dataset takes 3 values i.e, LOW, NORMAL, H HIGH. The causal model uses only parents of Y whereas the associational model (DNN) uses all nodes except Y as features.
Implementation. We sample data using the causal structure and probabilities from the Bayesian network, and use 60:40% split for train-test datasets. We learn a causal model and deep neural network (DNN) on each training dataset. We implement the attacker model to perform membership inference attack using the output confidences of both these models, based on past work (Salem et al., 2018). The input features for the attacker model comprises of the output confidences from the target model, and the output is membership prediction (member / non-member) in the training dataset of the target model. In both the train and the test data for the attacker model, the number of members and non-members are equal. The creation of the attacker dataset is described in Figure in Appendix. Note that the attack accuracies reported are upper bound since we assume that the adversary has white-box access to the ML model.
To train the causal model, we use the bnlearn library in R language that supports maximum likelihood estimation of the parameters in Y's conditional probability table. For prediction, we use the parents method to predict the class of any specific variable. To train the DNN model and the attacker model, we build custom estimators in Python using Tensorflow v1 2 ten The DNN model is a multilayer perceptron (MIP) with 3 hidden layers of 1 128, 512 and 128 nodes respectively. The learning rate is set to 0.0001 and the model is trained for 10000 steps. The attacker model has 2 hidden layers with 5 nodes each, a learning rate of 0.001, and is trained for 5000 steps. Both models use Adam optimizer, ReLU for the activation function, and cross entropy as the loss function. We chose these parameters to ensure model convergence.
4.1 EXPERIMENTAL SETUP
We evaluate the DNN and the causal model sample sizes ranging from 1K to IM dataset sizes. We refer Test 1 as the test dataset which is drawn from the same distribution as the training data and Test 2 is generated from a completely different distribution except for the relationship of the output class to its parents. To generate Test 2, we alter the true probabilities Pr(X) uniformly at random (later, we consider adding noise to the original value). Our goal with generating Test is to capture the realistic behaviour of shift in distribution for input features. We refer the causal and DNN model as the target on which the attack is perpetrated.
4.2 RESULTS
We present results on the accuracy of target models (causal and DNN models) and the membership attack accuracy for different dataset sizes and test distributions.
Accuracy comparison of DNN and Causal models. Figure 2a shows the target model accuracy comparison for the DNN and the causal model trained on the Child dataset with XrayReport as the output variable. We report the accuracy of the target models only for a single run since in practice the attacker would have access to the outputs of only a single model. We observe that the DNN model
7

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Tert olall
Attack acsu way Gellnsta for sabk
froelanna Mukt Bloanee
0.75
64-
02
Chiss
Voew
Choc
Saata
Vilte
Sucho
CVM
Aler
(x)
(b)
(c)
Figure 3: a Target accuracy.
bb) Attack accuracy.
Attack accuracy for true, learned causal model and DNN.
has large difference between the train and the test accuracy (both Test 1 and Test 2) for smaller dataset sizes (IK and 2K). This indicates that the model overfits on the training data for these dataset sizes. However, after 10K samples, the model converges such that the train and Test I dataset have the same accuracy. The accuracy for the Test 2 distribution stabilizes for a total dataset size of 10K samples. In contrast, for the causal model, the train and Test accuracy are similar for the causal model even on smaller dataset sizes. However, after convergence at around 10K samples, the gap between the accuracy of train and Test 2 dataset is the same for both the DNN and the causal model. Figure Ba shows the accuracy comparison for all four datasets with similar results.
Attack Accuracy comparison of DNN and Causal models. A naive attacker classifier would predict all the samples to be members and therefore achieve 0.5 prediction accuracy. Thus, we consider 0.5 as the baseline attack accuracy which is equal to a random guess. Figure shoss the attack accuracy comparison for Test I (same distribution) and Test 2 (different distribution) datasets. Attack accuracy of the Test 1 dataset for the causal model is slightly above a random guess for smaller dataset sizes, and then converges 0.5. In comparison, attack accuracy for the DNN on Test 1 dataset is over 0.6 for smaller samples sizes and reaches 0.5 after 10K datapoints. This confirms past work that an overfitted DNN is susceptible to membership inference attacks even for test data generated from the same distribution as the training data (Yeom et al. 2018). On Test 2, the attack accuracy is always higher for the DNN than the causal model, indicating our main result that associational models "overfit" to the training distribution, in addition to the training dataset. Membership inference accuracy for DNNs is as high as 0.8 for total dataset size of 50K while that of causal models is below 0.6. Further, attack accuracy for DNN increases with sample size whereas attack accuracy for the causal model reduces to 0.5 for total dataset size over 100k even when the gap between the train and test accuracies is the same as DNNs ( as shown in Figure 2ab. These results show that causal models generalize better than DNNs across input distributions. Figure 33b shows a similar result for all four datasets. The attack accuracy for DNNs and the causal model close to 0.5 for the Test 1 dataset while for the Test dataset the attack accuracy is significantly higher for DNNs than causal model. This empirically confirms our claim that in general, causal models are robust membership inference attacks across test distributions as compared to associational models.
Attack Accuracy for Different Test Distributions. To understand the change in attack accuracy as Pr(X) changes. we also generate test data from different distributions by adding varying amount of noise to the true probabilities. We range the noise value between 0 to 2 and add the individual probabilities which are then normalized to sum up to 1. Figure 2 shows the comparison of attack accuracy for the causal model and the DNN on the child dataset for total sample size of 100K samples. We observe that the attack accuracy increases with increase in the noise values for the DNN. Even for a small amount of noise, attack accuracies increase sharply. In contrast, attack accuracies stay close to 0.5 for the causal model, demonstrating the robustness membership attacks.
Results with learnt causal model. Finally, we perform experiments to understand the effect of privacy guarantees causal structures learned from data that might be different from the true causal structure. We evaluate the attack accuracy for learned causal models on the Sachs, Chi 1 d and Al am datase 2 For these datasets, simple hill-climbing algorithm returned the true causal parents. Hence, we evaluated attack accuracy for models with hand-crafted errors in learning the structure i.c., misestimation of causal parents, see Figure [3. Specifically, we include two non-causal features as parents of the output variable along with the true causal features. The attack risk increases as learnt model deviates from the true causal structure, however it still exhibits lower attack accuracy than the corresponding associational model. Table 2 in Appendix gives fine-grained analysis.
We exclude the water r dataset bn. bit in balearn library gives error due the extreme prohabilities.
8

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
5 RELATED WORK
Privacy attacks and defenses on ML models. Shokri et al. (017 demonstrate the first membership inference attacks on black box neural network models with access only to the confidence values. Similar attacks have been shown on several other models such GANs (Hayes et al.) (017)), text prediction generative models (Carlini et al. 2018 Song & Shmatikov 2018) and federated learning models (Nasr et al. 2018b). However, prior research does not focus the severity of these attacks with change in the distribution of the test dataset. We discussed in Section 2 that existing defenses based regularization (Nasr et al.) 20186) are not practical when models are evaluated on test inputs from different distributions. Another line of defense is to add differentially private noise while training the model. However, the e values necessary mitigate membership inference attacks in deep neural networks require addition of large amount of noise that degrades the accuracy of the output model (Rahman al. 2018). Thus, there is trade-off between privacy and utility when using differential privacy for neural networks. In contrast, we show that causal models require lower amount of noise to achieve the same e differential privacy guarantees and hence retain accuracy closer the original model. Further, as training sample sizes become sufficiently large, as shown in Section 4 causal models are, by definition, robust to membership inference attacks across distributions. Causal learning and privacy.Thee There is a substantial literature on learning causal models from data; for a review see (Peters et al. 2017 Pearl 2009). Kusner et al. (2015) proposed a method to privately reveal parameters from a causal learning algorithm, using the framework of differential privacy. Instead of a specific causal algorithm, our focus is on the privacy benefits of causal models for general predictive tasks. While recent work applies causal models to study properties of machine learning models such as providing explanations (Datta et al., (016) or fairness (Kusner et al.] (017). the relation of causality privacy is yet unexplored. With this paper, we present the first result which shows the privacy benefits of causal models.
6 CONCLUSION AND FUTURE WORK
We conclude that causal learning is a promising approach to train models which are robust to privacy attacks such as membership inference and model inversion. As our future work, want to relax our assumption of known causal structure and investigate the privacy guarantees of causal models where the causal features and the relationship between them is not known apriori (Peters al. 2017).
REFERENCES
Bayesian Network Repository. www. www.bbllearn..cm/bmrrpooi.oooooe
Tensorflow.
https::/www.teet
www.:ww.we...... www.teenssrffow.org
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: 10912, 2019.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Nicholas Carlini, Chang Liu, Jernej Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Measuring unintended neural network memorization & extracting secrets. arXiv preprint arXiv:1802.002232, 2018
Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influ- ence: Theory and experiments with learning systems. In Security and Privacy (SP), 2016 IEEE Symposium on, pp. 598-617. IEEE, 2016.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-48 2014.
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature medicine, 25(1):24, 2019.
Thomas Fischer and Christopher Krauss. Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2):654-66) 2018.

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322-1333. ACM, 2015.
Jihun Hamm, Yingjun Cao, and Mikhail Belkin. Learning privately from multiparty data. In International Conference Machine Learning, pp. 555-563, 2016.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. arXiv preprint MAXiv:1705.076633 2017.
Matt J Kusner, Yu Sun, Karthik Sridharan, and Kilian Q Weinberger. Private causal inference. arXiv preprint arXiv:1512.05465 2015.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimesss In Advances in Neural Information Processing Systems, pp. 4066-4076, 2017.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902..443 2009.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5602 2013.
Razich Nabi and Ilya Shpitser. Fair inference on outcomes. In Proceedings of the... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, volume 2018, pp. NIH Public Access, 2018
NIH
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis deep learning: Stand-alone and federated learning under passive and active white-box inference attacks. arXiv preprint
2018a.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization, In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 634-646. ACM, 2018b.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In ICLR, 2017.
Judea Pearl. Causality. Cambridge university press, 2009.
Jean-Philippe Pellet and André Elisseeff. Using markov blankets for causal structure learning. Journal of Machine Learning Research, 1295::1295-1342. 2008.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant
prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-1012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
Md Atiqur Rahman, Tanzila Rahman, Robert Laganiere, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Transactions on Data Privacy, 2018.
Ahmed Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes, MI-leaks: Model and data independent membership inference attacks and defenses machine learning models. arXiv preprint arXiv: /00iv:880..00 1806.0/246. 2018.
Marco Scutari. Learning bayesian networks with the bnlearn r package. arXiv preprint arXiv:0908.3877 2009.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. doi: 00.0077/809981107298881
10

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. Security and Privacy (SP), 2017 IEEE Symposium on, pp- 3-18. IEEE, 2017.
Congzheng Song and Vitaly Shmatikov. The natural auditor: How to tell if someone used your words to train their model. arXiv preprint 1xXiv:811100011:: 2018.
Avraam Tsantekidis, Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj. and Alexandros losifidis. Using deep learning to detect price change indications in financial markets. In 2017 25th European Signal Processing Conference (EUSIPCO), pp. 2511-2515, IEEE, 2017.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection overfitting In 2018 IEEE 3 Computer Security Foundations Symposium (CSF), pp. 268-282. IEEE, 2018.
A GENERALIZATION PROPERTIES OF CAUSAL MODEL
A.1 GENERALIZATION OVER DIFFERENT DISTRIBUTIONS
Theorem 1. Consider a structural causal graph G that connects to Y, and causal features Xc where Xe Markov Blanket of under G. Let PXX. Y) and P" (X, Y) be two distributions with arbitrary P(X) and P* (X) such that the causal relationship between Xe and Y is preserved, which implies that PcY P' (Y]xc). Let f Xe + be the resultant invariant labelling function such that y = (Xc). Further, assume that He represents the set of causal models he Xe + Y that use all causal features and H4 represent the set of associational models ha X - Y that may use all available features, such that Hc S H. and f E He.
Then, for any symmetric loss function that obeys the triangle inequality, the upper bound of ODE from dataset S P(X, Y) to P" (called ODE ODE-Bound) for causal model he E He is less than or equal the upper bound -DEuddund of an associational model ha HA, with probability at least - (1-8)3.
ODE Boundp, (be: f; S ODE -EEudddd (ha, f;
(4)
Proof. Consider model h + belonging to set of models H. that was trained on S ~ P(X Y).
From Def. 2 we write,
ODEp p- (h, f) = Lp- (h, Lanp (h,
Lp. (h, Lp(h, f) + Lp f) L f)
(15)
Lp (h, L.(h, f) + (h,
where the last equation is to due to of the in-distribution generalization error.
Let us denote the optimal loss-minimizing hypotheses over 7t for P and P* hoPT and hOPT.
hp arg min Lp(h, f) hp arg min Lea (h, BEH
(16)
bEH
Using the triangle inequality of the loss function, we can write:
Lp- (b & Lp- (h, DPTT) + Lp- (hePT OP? Lx- (hOPT)
(17)
And,
(o f) 2 (h - Le (hoPT f) - (h, < Lo hPTT) ) L (bffee,, f)
(18)
Thus, combining Eq. 15
and 18 we obtain,
ODE,,p- (b, IDE, f) Lp- (h, hQTT) OPT Lp- (naeT hDPT) + Lp- (hoPT f) Cb LPT)) Lp (hopT
IDE+(h, (Le- (h, h UPT) Lo OPT Lp- (hOPT ff) Lp f) + Lp- (h OPT hPT)
(h, disci (P, P²) Lp- 55-(hhr f) Lp UPr ff) Lx- (h DPT OPT)
(19)
where the last inequality is due to the definition of discrepancy distance (Definition divides the out-of-distribution generalization error of hypothesis h in four parts:
3 Equation 19
11

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
1. IDEp (h, f) denotes the in-distribution error of h. This can be bounded by typical generalization bounds, such as the uniform error bound that depends only on the VC dimension and sample size of S (Shalev-Shwartz & BaviD 2014). Using uniform error bound based on the VC dimension, we obtain, with probability at least - 6,
(In2)S)) + +1) In(4/0) IDE- POEE-Bunnd(HH) (H, 8
IDE
(20)
Since He S HA. VC-dimension of causal models is not greater than that of associational models. Thus,
VCDim S VCDim(HAs) + IDE Boe--Bouud,,,,eee S) - IDE DEE-Bound(H,, S)
(21)
2. (P,, denotes the distance between the two distributions. Given two distributions, the discrepancy distance does not depend on h, but only on the hypothesis class H.
1 3. Lp- (h3r7, and Lr(hT', .f) measure the error due the true labeling function f being outside the hypothesis class H.
Cp- (ng*7 .ng²") denotes the loss (or difference) between the moss-mmnimizing function over P and the loss-minimizing hypothesis over P"
We next consider two classes of models, Hc that contains models that uses all causal features (Xc) from a Markov Blanket over the structural causal graph, and HA that contains associational models that may use all or a subset of all available features. Below we show that for given distribution and another distribution P" such that P(Y Xx) = P* (Y(Xe), the last term Lp (h.PT, hg.T) of Equation 19 vanishes for causal model but may remain non-zero for an associational model.
Causal Model. Given a structural causal network, let us construct a model using all variables Xe in Y's Markov Blanket. By property of the structural causal network, X5 includes all parents of Y and thus there are no backdoor paths, using Rule 2 of do-calculus from Pearl (2009):
Pr(Y)do(X_ = xe)) P(Y Xc xx) P" (Y xe)
(22)
where the last equality is since data from p also shares the same causal graph, (Y is independent of Xy given Xe in P*). Since the conditional distribution of Y given X features is the same across P and p and the true labelling function f E Hc. the loss-miiimizing models should also be the same. Defining h = arg min (b, f) and ha.-- = arg min Lp. (he, f), we obtain, E.
E
DPT he,P
OPT f h
(23)
And therefore, the term Lp- (ho.p, OPT = Also, La f) = Lpp OPT O.
Associational Model. In contrast, an associational model may use subset of X, Xs S X, that may not include all variables in the Markov Blanket, or may include the Markov Blanket but also include other extraneous variables. Let the set of such models be HAS. Note that we only consider HAs to be the set of models for which Le f) = Lp (h f) = 0 (otherwise, the loss bound for model from HAs is trivially larger). This can happen when feature x; & X does not belong to the Markov Blanket under the causal graph, but belongs to the associational Markov Blanket under the probability distribution over P or P*
Then, by definition of f. f & HAS. Thus, UPT + f OPT Further, the massmmniiizzin hypotheses OPT and hagpp- depend on the different distributions P(Y = xa) and pa (Y|Xs = xx) and thus not necessarily equal. Hence, hop * DPT
For the other subset of associational models, since X C, it possible that associational model includes only the causal features of Y. Therefore, in general, L OPT UPT
(b h 2 0.
Loss Bounds.
Hence, using Eq. 23 we write equation 19 for a causal model as:
ODEp, (he, f) = Lp- (h., f) (he, f) S disci,H, P*) + IDE, (b,, ff
(24)
Using Eqns. 20 and 21 obtain, with probability at least - 8:
ODE, p- (h., f) < disci, H. (P) IDE- Boundo (H,, S:8)
(25)
< disci.H. (P, P*) + IDE-Boundp M4, S: -DE--Bundd f:))
(26)
(27)
12

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Similarly, for an associational model, we obtain, with probability at least 8:
pDE,,p* (ha; f) Lp- (ha, f) Lgmp(ha; f)
I H. (P, P*) + IDE Boundp (:2, Bound,(HH,, Lp- (happ h
(28)
ODE BoB--Bundppp- f;
Finally, using Definition 3 for discrepancy distance, we can state He _ H4 = discL. He disci,H (P,P"). P*). Therefore, from Eq. and [28] we claim with probability (1 8)²,
(P.P")
ODE-Boundp pa (h.: f: < ODE-Boundp.> [ha,f:) f;
(29)
A.2 GENERALIZATION OVER A SINGLE DATAPOINT
Corollary 1. Consider a causal model b. Xe - Y and an associational model h, X - Y trained on a datasel S ~ Let (x,y) S and (x', ) & be two input instances such that they share the same true labelling function y = f fxxx) and y' f(x'). Then, the worst-case generalization error for a causal model on any such is less than or equal to that for an associational model. [Proof in Appendix Section A.2
max x x' (x))
Lx (he, f) Lx S
max
Lx (na: f) Lx Lba
(5)
XES,X' by (x))
Proof. Using the triangle inequality for the loss function, we obtain, Lx (b, S Lx (h, 1TT ) + Lx' LDPT)) Lx (DpeT,
(30)
L, S (I, (f + L, (horT,
(31)
Combining the two inequalities,
Lx (h, (b, f) I L (h, hp OPT) + Lx (hg DPT mpp
+ Lg (hp .f)
(h, OPT + L, (hg
CPT
(32)
< distx(x, x') + L (hg- DPT
f) + L, (hp UPT + (h OPT b.r
where dist(x,x))
(h, L, (b, h')l, analogous the disc, for distributions.
For causal model, we know that h = h.p... Further, from Theorem proof, L Lx f) (h f) = 0 and same for p and loss on x'.
=
Hence,
L (be: f) Le f) & distHs (x,x') Lg (ha f) - L, f) & distu, (x,x') + Lx (happ
(33)
OPT OPT
(34)
where (x, distH, (x, since He Ha.
Next, we show that these bounds are tight. That is, there exists an associational model h2 whose generalization error on x' is exactly the RHS Eqn. 34 and thus higher than the bound for any causal model. Below we prove by constructing one such ha-
p and (hOPT h hOPT hgp ). For simplicity in construction, let select He h Thus dista dista, (x,x)) x') whereas disty, (x, x') 2 O. We obtain the ODE-generalization bound for causal model,
Lg (be, f) L, (h., f) < L OhT f) + Lg (h P. f) ODE-Boundg.... (b., Ho)
(35)
Let us now construct an associational model, hj. such that:
Ly (bL, f) = L L 1p ) + Lx (hp 1g + L (h OPT
f)
(36)
-L, (bj, (b, 0PT + L, (hg
DPT
13

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
A simple construction for the above equalities is to select P and P", and correspondingly xf and x't such that L (ba,P f) = Lun (h f) = 0 and Ly (ha,p. OPT 0. Further, h can be selected
such that Lx/t (h h) = 0 and Lt (hī h)TT) = 0.
Then, using Eqn. 36
Len (bg, f) - 1xt (h f) = Lx ( hp Lx (h hp ) + Lyr (th ; f) + Lxt (ho.P: : f) + Lyr (hopT hg.p-)
= 0 + Lye (h DPT
a.p" f) + Lxt (ha,7
OPT , f) + Let (h ha.p
> L (happ- ; f) + Lut (ha.P: :
= MDDE-Boundy.. (he, Hc)
(37)
where the last equality comes from Eqn. 35 Combining Eqns. 37 and 33 we obtain,
max f) Ly (be: f) S max ODE -OE--Bund,xx' (he, f; He) max Ly (Dg: f) LL (h. f) (38)
x,x"
x,x'
x.x'
where the maximum is over and x' & S such that y' f(x()-
B DIFFERENTIAL PRIVACY GUARANTEES WITH TIGHTER DATA-DEPENDENT BOUNDS
In this section we provide the differential privacy guarantee of a causal model based on a recent method (Papernot et al. 2017) that provides tighter data-dependent bounds. We first present Lemma on generalization of causal models and then prove the main result.
As consequence of Theorem another generalization property of causal models is that the models trained on data from two different distributions ((X) and P* (X) are likely to output the same value for a new input.
Lemma 3. Under the conditions of Theorem 7 let he, be a causal model trained on distribution and let h.. be model trained on p Similarly, let ha.P and ha. p. be correlational models trained on P and P respectively. Assume that the correlational and causal models report the same accuracy a. Then for any new data input z,
((( = he.p- (x)) 2 p(()o,,(x) P. (x))
00, As the size of the training sample N + co, the LHS- 1.
Proof. Let OPPT arg minneh. (p f) and similarly let O.P.P. arg Min Lpr(h, (h, f) be the loss Jos-minimizing hypotheses under these two distributions, where H is the set of hypotheses. We can analogously define h.,P and h.. For a causal model, we know from Theorem I that hOpT N.PT. As N - each of models on P and pa approach their loss-minimizing functions. Then, for any input x.
a.P.
L(he (x)) = L(h (x), L(hOPT(2))
(x), hOPT (x)) =0
(39)
ha)
= I(hO,T (x), k.,p (x)) 2
(40)
= (r((h,,(x)) = h., (x)) = 1 Pr(ho,p(a)) = ha (x))
(41)
Under finite samples, let the accuracy of each model be a. Then for any new input, the causal models predict the same output if they both match ha hmmin or both do not match. The probability of this event can be captured by the accuracy of the models: a² if both models match h min and
(1 - (2- if they do not.
Pr(he.p = h-. = a² (1 - a)²
(42)
For correlational models, NOPT # NOPT for some æ. For those æ, assuming the same accuracy a,
Pr(h1 = h2) = 2a(1 - a)
(43)
Since 2a(1 - a) a² + (1-a) 2 Va E [0, 1]. we have
Pr(he,p h-p 2 Pr(ho,p ha,p-)
14

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Based on the above generalization property, we show that causal models provide stronger differential privacy guarantees than corresponding associational models. We utilize the subsample and aggregate technique (Dwork et al. 2014) that was extended for machine learning in Hamm et al.) (2016) and Papernot et al. (201 7). for constructing a differentiably private model. The framework considers M arbitrary teacher models that are trained on a separate subsample of the dataset without replacement. Then, student model is trained on some auxiliary unlabeled data with the (pseudo) labels generated from majority vote of the teachers. Differential privacy can be achieved by either perturbing the number of votes for each class (Papernot et al. 2017), or perturbing the learnt parameters of the student model (Hamm al. 2016). For any new input, the output of the model is a majority vote on the predicted labels from the M models. The privacy guarantees are better if larger number of teacher models agree on each input, since by definition the majority decision could not have been changed by modifying a single data point (or single teacher's vote). Since causal models generalize to new distributions, intuitively we expect causal models trained on separate samples to agree more. Below we show that for a fixed amount of noise, causal model is DDD compared to c-DP for a associational model, where Ce S.
Theorem 3. Let D be datasel generated from possibly a mixture of different distributions P(X, Y) such that P() remains the same. Let ny be the votes for the jth class from M teacher models. Let M be the mechanism that produces noisy max, arg max, {ny + L/p(2/7)}. Then the privacy budget e for a causal model is lower than that for the associational model with the same accuracy.
Proof. Consider a change in single input example (ar, y). leading to a new D' dataset. Since sub-datasets are sampled without replacement, only a single teacher model can change in D'. Let n' be the vote counts for each class under D', Because the change in single input can only affect one model's vote, [n. - nn11 S 1.
Let the noise added to each class be ry ~ Lap(2/7). Let the majority class (class with the highest votes) using data from 1) be and the class with the second largest votes be j. Let us consider the minimum noise r" required for class i be the majority output under M over D. Then,
74 + r" > n2 +ry
For to have the maximum votes using .M over D' too, we need,
n/ + > n'z +
In the worst case, ni = - and n² n, + for some j. Thus, we need,
n4 - 1 + > n2 + + r = n. + r; > n2 + + rr
(44)
which shows that r; > r +2. Note that r* > r3 - (n: - n,y). We have two cases:
CASE I:
The noise ry < n, - n2j, and therefore y* Writing Pr(i)D") to denote the probability that class i is chosen as the majority class under D',
1 - 0.5 0.5 exp 1 )rr"
P(ilD) P(r. 2 r" 2)
(45)
exp()) P(r; (ir;2ri r))
1 - P(()))))
(46)
where the equations on the right are due to Laplace c.d.f. Using the above equation, we can write:
P(i exp(1) + --exp(7) exp(1) + 1 exp(7)
P(r, rr) I exp(c)
(47)
P()D)
P(ID)
for some c > 0. As P)) = P(r; 2 r*) increases, the ratio decreases and thus the effective privacy budget (e) decreases. Thus, a model with a lower r* (effectively higher (r*l) will exhibit the l.
Below, we show that [r*| is higher for a causal model, and thus P(r. 2 r*) is higher. Intuitively, |r*] is higher when there is more consensus between the M teacher models since [r*| is the difference between the votes for the highest voted class with the votes for the second-higg -highst class.
Let us consider two causal teacher models ht. and h2c, and two associational teacher models, h1 and h2. From Lemma 3 for any new x, and for same accuracies of the models, there is more consensus among causal models.
(1) = h22(z)) 2 (() = h2(x))
(48)
Hence r: Sr" From Equation 47
Ec 5
15

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
CASE II: The noise rj >= R4 n.j. and therefore r 0. Following the steps above, obtain:
P(i)D') = P(r: 2 r" + 2) = exp(->)(P(r: 2 r*))
0.5exp(-15) exp( 2 (2("))
(49)
= (()p()))
(50)
Thus, the ratio does not depend on r*
P'ilD") exp(-7) D)
(51)
Under CASE II when the noise is higher to the differences in votes between the highest and second highest voted class, causal models provide the same privacy budget as associational models.
Thus, overall, 6a S
C INFINITE SAMPLE ROBUSTNESS TO MEMBERSHIP INFERENCE ATTACKS
Corollary 2. Let bes be causal model trained using empirical risk minimization on a dataset S ~ Y) with sample size N. As N + cc, membership advantage Adv(AA, hais) + 0.
Proof. hain be obtained by empirical risk minimization.
N Lxs (h, f)
min h c.5
=
arg min LS- (b, f) bE
arg min hE
(52)
N
As [S] = N + D0, 2,3 + DPT Suppose now that there exists another S' of the same size such that S' ~ p.. Then as [S']
+ D0, h 7 h.p.
From Theorem
he.F he. Thus,
lim h
lim hain K-+00
(53)
+00
Equation 53 implies that as N + oc, the learnt h does not depend on the training set, as long as the training set is sampled from any distribution P such that P(Y(Xc) = PY(()))). That is, being the global minimizer distributions, hain hg,p does not depend its training set. Therefore, ha (x) is independent of whether x is in the training set.
lim Adv(A,, ) = Pr(A = 1/b = 1) - Pr(AA- = 1|b = 0) 10000
BjAAbb 1] B|AAbb = E[A(tt))) 1] B[A(DE))) o] (54)
([((033))) -
where the second last cquality follows since any function of ha is independent of the training dataset.
D ROBUSTNESS TO ATTRIBUTE INFERENCE ATTACKS
In addition to revealing membership in the training set, model may also reveal the value of individual sensitive features of a test input, given partial knowledge of its features. For instance, given a training dataset of HIV patients, an adversary may infer other attributes of a person (e.g.. genetic information) given that they know their demographics and other public features. As another example, it can be possible to infer a person's 's face based on hill climb on the output score for a face detection model (Fredrikson et al. 2015). Model inversion is not always due to a fault in learning: a model may learn a true, generalizable relationship between features and the outcome, but still be vulnerable to a model inversion attack. This is because given (k-1) features and the true outcome label, it is possible to guess the kth feature by brute-force search on output scores generated by the model. However, inversion based on learning correlations between features, e.g., using some demographics predict disease, can be alleviated by causal models, since a feature will not be included a model unless directly affects the outcome.
16

---

Under review as a conference paper at ICLR 2020 [Please do not distribute.]
Definition 7 (From Yeom al. (2018)). Let be model trained dataset D(X,Y). Let A be adversary with access to h, and a partial test input A C z. The attribute advantage of the adversary is the difference between true and false positive rates in guessing the value of sensitive feature xa $ Xx. For a binary Xs.
Adv (A. h) = Pr(AA 1 - 1) P 1
(55)
Theorem 5. Given a dataset D) of size N and structural causal network that connects to Y, causal model he makes it impossible to infer non-causal features.
Proof. The proof follows trivially from definition causal model. he includes only causal features during training. Thus, h(x) is independent of all features not in X..
Adv(A,,) Pr(A 1 1) (r 1 = 0)
(56)
Pr(A(hh) = 1 = - (b)) = 1 = 0) Pr(A(h) = 1) - Pr(AA()) =) =0
E EXPERIMENTS
E. DATASET DISTRIBUTION
The target model is trained using the synthetic training and test data generated using the bnlearn library. We first divide the total dataset into training and test dataset in a 60:40 ratio. Further, the output of the trained model for each of the training and test dataset is again divided into 50:50 ratio. The training set for the attacker model consists of confidence values of the target model for the training well as the test dataset. The relation is explained in Figure 4 Note that the attacker model trained on the confidence output of the target models.
Total Datasel Size samples)
Model Training Set (60K)
Target Model Test Set (40K)
30K
10K
Attacker Model Test Set 10K (member)
Train Set
30K [member]]
30K (non member)
10K (non member)
Figure 4; Dataset division for training target and attacker models.
E.2 FINE-GRAINED ATTACK ANALYSIS OF LEARNED CAUSAL MODELS.
We expand the evaluation of the attack accuracy for a learnt causal model on the Sachs dataset and Akt outcome. We increment the non-causal features to the learned model in addition to the true causal features. ssoow the attack and prediction accuracy for this model when trained with the true causal, learned causal model with and non-causal features, and the results for the corresponding DNN model.
True Model
Learned Causal causal +)
DNN
Acc.
causal
I non non-causal non-cause
(%)
parents
parent
parents
Attack
50
52
61
76
Pred.
68.8
Table 2: Attack and Prediction accuracy comparison across models for Sachs dataset and Akt output variable.
17


Reviews:
[{"review_id": "r1lAKWKcKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: This paper discusses the risk of membership inference attacks that deep neural networks might face when used in a practical manner on real world datasets. Membership inference attacks can result in privacy breaches, a significant concern for many fields who might stand to benefit from using deep learning in applications. The authors demonstrate how attack accuracy goes up when one dataset is used for training while another altogether is used for testing. They propose the use of causal learning approaches in order to negate risk of membership inference attacks. Causal models can handle distribution shifts across datasets because they learn using a causal structure. \n\nContributions: In the theory part of the paper, the authors provide several proofs demonstrating that causal models have stronger differential privacy guarantees than association modes, that causal models trained on large samples are able to protect the dataset against attacks, and that causal models trained on smaller samples still have higher protection than association models trained on similarly sized samples. In addition to theoretical contributions, the authors also provide an experimental evaluation using 4 accepted experimental datasets.\n\n\nQuestions and Comments:\nPage 2: “...while association modes exhibit upto 80%...” -> “...up to...”\nMy expertise is not in causal learning or structures, so I have a few questions about using it in practice. You mentioned that the datasets used in the experimental section were used in order to avoid errors in learning causal structure. \nHow likely is it to have these errors using a different dataset? \nHow long/how much effort does it take to figure out conditional probability table? Is this a significant amount of time compared to training? Is it automatic or manually done by humans?\nIf it is done by humans, is it plausible to assume that every dataset implicitly contains a causal structure (not including random walks)?\nYour experimental results suggest that the causal model can learn on smaller amounts of data than the DNN. Does this scale for even larger input parameter datasets as well, such as water?\nYou mention this potentially being used to prevent attacks on real-world applications, such as HIV patient prediction/classification systems. Do you believe that your results prove causal models will scale to datasets that contain these kinds of complex causal structures? \nCould you provide the layer architectures of all three models used for your experiments? Are these out-of-the-box solutions from libraries, or something more custom built?\n\nHow would the causal model perform compared to state of the art techniques for these datasets, in both accuracy and attack protection? I understand that isn't the main point of this paper, this is me being curious.\n\nI give this paper a borderline acceptance, based upon the fact that the above questions need to be addressed. I'm not sure its clear to see how the experimental results demonstrate that the causal model definitely outperforms DNNs in all cases. I would like to hear the author's defense of the method when it comes to datasets with higher numbers of features, specifically the water dataset.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByxkWp-WjS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n   The authors consider a transfer learning problem where the source distribution is P(X,Y) while the target distribution is P*(X,Y) and classifier is trained on data from the source distribution. They also assume that the causal graph generating the data (X and Y) is identical while the conditional probabilities (mechanisms) could change between the source and the target. Further, they assume that if X_C is the Markov Blanket for variable Y in P and P*,  then P(Y|X_C) = P*(Y|X_C). Therefore the best predictor in terms of cross entropy loss for both distributions is identical if it focuses on the variables in the Markov Blanket. Authors define \"causal hypothesis\" as the one that uses only variables in the Markov Blanket (X_C) to predict Y.\n\n In this setting, the authors show two sets of results: a) Out of Distribution Generalization Error is less for the optimal causal predictors from any class of causal hypotheses under any loss function. b) If we tweak the definition of differential privacy where neighboring datasets are defined by replacing one of the variables in the training set by a sample from the test distribution (I have lots of questions about this definition later on), then the authors show that causal classifiers that depend only on the Markov Blanket has lower sensitivity to the change than that of associative classifiers (that use all features). This is used to prove that the causal ones have tighter differential privacy guarantees than the associative ones\n c) Using the differential privacy results, they also show that optimal causal classifiers are more resistant to membership attacks.\n\n\nThe authors demonstrate the results through membership attack accuracies on causally trained and associative models on 4 datasets where the causal graph and the parameters (conditional probabilities) of the Bayesian Network are known apriori.\n\n\nMajor Issues:\n    I have lots of issues with the theory in the paper. Thats the main reason for my recommendation.\n\n  1. Why is h_{c,P}^{OPT} = h_{c,P*}^{OPT} ?? (Equation 23 in Page 11) - Authors say that since the markov blanket is the only thing used to predict Y for causal predictors and P (Y|X_C) = P *(Y|X_C), this should be true. But I have problems with this statement/argument. First of all, this is claimed for for any loss function L (not just Cross Entropy Loss) and particularly for a generic hypothesis class H_C (that depends only on Markov Blanket).  \n\nConsider the following counter example - Suppose all the features are in the Markov Blanket of Y (even a simpler case where all features are causal parents of Y). Suppose the true P (Y|X) is a logistic model with weights w_1 on one part of the domain D_1 and with weights w_2 for another part of the domain D_2. Suppose P is a mixture distribution of D_1 and D_2. P* is another mixture distribution (mixed differently) of D_1 and D_2.\n\nSuppose I consider the class of logistic classifiers (but a single logistic model with one weight w) to be my hypothesis class and I use the standard logistic loss (logistic regression) on P, since a single logistic model with one slope cannot match the different slopes in different parts of the domain, it will result in some weight vector w^{opt}_1. Now, since the mixtures of D_1 and D_2 are different in the P* (but P (Y|X ) is identical), the optimal w^{opt}_2 for the P* will be different. \n\nSo for arbitrary hypothesis classes (that do not capture the true P(Y|X_c)) and for a non cross entropy loss - clearly this does not hold at all !! Covariate shifts amongst X_C alone with create a different classifier for an arbitrary loss (even if P (Y|X_C) is the same across both). In fact, the only way I see to salvage this is to assume Cross Entropy loss and talk about all soft classifiers (without restrictions to hypothesis class). But even if thats the case, then the best associational model will be the one that uses the Markov Blanket too ! .\n\nThis claim about h_C is crucially used in proof of theorem 1 (Page 11) and Proof of Corollary 1 (Page 13). This is fundamental to all theorems later. That brings into question the validity of many (if not all) the theoretical results in the paper. Authors must address this. \n\n2. Issues regarding definition of certain quantities.  \n\na) In equation 5 a quantity max_{x,x'} L_{x sampled from P} (h_c,f) - L{x' sampled from P*} (h_c,f) is defined - the inner quantity is a random quantity that depends on the samples x and x', Then what is the max operator over ?? - What does it mean to have worst case over samples from a distribution ??? Does it mean samples from two different domains ?? \nEven the quantity does not seem to be well defined. \n\nb) Similar issue occurs in Lemma 1 - Neighboring datasets S and S' are created by first sampling S from P and then S' is obtained by replacing an arbitrary point in S by a random point from P*. Then sensitivity is defines as a max over pair of neighboring datasets - again S and S' are random samples, so what is the max over ?? If it is the worst case - why is the sampling coming in there ? Since it uses Corollary 1 - the main result inherits the same fundamental issues the has been pointed out above.\n\nc) In theorem 2, {cal F}_a is an algorithm. What does it mean to add noise ? - Does it mean you add noise to the model parameters ?? - This is confusing at best.\n\nMinor Issues:\n1. Authors claim that connection between causality and privacy has not been explored (Page 2). Pls refer to https://arxiv.org/pdf/1710.05899.pdf where differential privacy itself is related to interventional effects in a system. This connection is very different from the scope of the current paper. However, the statement by the authors is strictly not true.\n\n2. Why is the ground truth function f:X->Y (Section 2.2.) relevant when clearly you have distribution P (X,Y) and P*(X,Y) ?? We might as well define Loss with L (h(x), y) where (x,y) is drawn according to P. What f is never defined anywhere. Authors seems to mean the suggestion I just said in the paper. Authors could clarify. This confuses stuff in the proofs too.\n\n3. Markov Blanket is not causal by any means in my opinion. It is just a minimal set of features conditioned on which Y does not depend on anything else. This only requires conditional independence tests to determine - a purely observational notion - in fact the markov blanket only depend on the moralized graph which does not change across the members of the equivalence class. So calling it causal is a bit confusing. If the features referred to least causal Parents - then still it would be consistent with the invariance in the Invariant Causal Prediction Literature (Peters et al 2016.) and would be causal.\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxFctTrcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes using causal learning models for alleviating privacy attacks, i.e. membership inference attacks. The paper proves that causal models trained on sufficiently large samples are robust to membership inference attacks; they confirm the theories with experiments on 4 synthetic data.\nThe paper is well written; theoretical proof seems correct as it combines proof of differential privacy guarantees  in Papernot et al. 2017, robustness to membership attacks in Yeom et al. 2018 with the generalization property of causal models from Pearl 2009 and Peters et al. 2017. Results are presented clearly. The paper is novel as the authors claimed they provide the first analysis of privacy benefits of causal models.\nThe main concern of this paper is the results are only confirmed on synthetic data, where all the 4 datasets are generated from known Bayesian networks (i.e., causal graphs). It doesn’t matter if these Bayesian nets are complex or not, because most of the experiments are done with the known true causal models except the last experiment in Figure 3c. Even with learnt causal models, they were learning a Bayesian net from too optimistic data that were indeed generated from Bayesian nets, but these are usually not true for real world data. So evaluations on real dataset, or other synthetic data that are not generated from Bayesian nets are necessary for validating the methods.\nAnother question is about the ‘causal models are known to be invariant to the training distribution and hence generalize well to shifts between samples from the same distribution and across different distributions.’  More explanations about ‘invariance’ is needed. For example, in Figure 2a and Figure 3a, causal models have similar performance (except Alarm data) with DNN models on test 2, where test samples are generated from different distributions than training samples. Also in Figure 3b, the attack accuracy are no different between causal models and DNN on test 1.\nThe last minor question is why only parents of Y are included in causal models in the experiments, but not the Markov blanket as stated earlier in Figure 1. \n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Maintaining the privacy of membership information contained within the data used to train machine learning models is paramount across many application domains.  Moreover, this risk can be more acute when the model is used to make predictions using out-of-sample data.  This paper applies a causal learning framework to mitigate this problem, motivated by the fact that causal models can be invariant to the training distribution and therefore potentially more resistant to certain privacy attacks.  Both theoretical and empirical results are provided in support of this application of causal modeling.

Overall, during the rebuttal period there was no strong support for this paper, and one reviewer in particular mentioned lingering unresolved yet non-trivial concerns.  For example, to avoid counter-examples raised the reviewer, a deterministic labeling function must be introduced, which trivializes the distribution p(Y|X) and leads to a problematic training and testing scenario from a practical standpoint.  Similarly the theoretical treatment involving Markov blankets was deemed confusing and/or misleading even after careful inspection of all author response details.  At the very least, this suggests that another round of review is required to clarify these issues before publication, and hence the decision to reject at this time.
Decision: Reject