Deep Learning with Data Privacy via Residual Perturbation
Authors: Wenqi Tao
Huaming Ling
Zuoqiang Shi
Bao Wang
Venue:  (2021)
Keywords: Data Privacy
Residual Perturbation
Deep Learning

Abstract: Protecting data privacy in deep learning (DL) is at its urgency. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many of the existing mechanisms achieve data privacy at the cost of significant utility degradation. In this paper, we propose a stochastic differential equation principled \emph{residual perturbation} for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap for DL. Empirically, we show that residual perturbation outperforms the state-of-the-art DP stochastic gradient descent (DPSGD) in both membership privacy protection and maintaining the DL models' utility. For instance, in the process of training ResNet8 for the IDC dataset classification, residual perturbation obtains an accuracy of 85.7\% and protects the perfect membership privacy; in contrast, DPSGD achieves an accuracy of 82.8\% and protects worse membership privacy. 

Under review as a conference paper at ICLR 2021
DEEP LEARNING WITH DATA PRIVACY VIA RESIDUAL PERTURBATION
Anonymous authors Paper under double-bling review
ABSTRACT
Protecting data privacy in deep learning (DL) is its urgency. Several celebrated privacy notions have been established and used for privacy-preserving DI., However, many of the existing mechanisms achieve data privacy at the cost of significant utility degradation. In this paper, we propose a stochastic differential equation principled residual perturbation for preerccirre DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap for DL. Empirically, we show that residual perturbation outperforms the state- of-the-art DP stochastic gradient descent (DPSGD) both membership privacy protection and maintaining the DL models' utility. For instance, in the process of training ResNet8 for the IDC dataset classification, residual perturbation obtains accuracy of 85.7% and protects the perfect membership privacy; in contrast, DPSGD achieves an accuracy of 82.8% and protects worse membership privacy.
INTRODUCTION
Many saghccap deep nets (DNs) are trained with private data, including medical images and financial transaction data (Yuen al., 2011; Feng al., 2017; Liu et al., 2017). DNs usually overfit and can memorize the private training data, which makes training DNs exposed to data privacy leakage (Fredrikson al., 2015a; Shokri et al., 2017: Salem et al., 2018; Yeom al., 2018; Sablayrolles et al., 201 8). Given a pre-trained DN, the membership inference attack can determine if an instance in the training set based on DN's response (Fredrikson et al., 2014; Shokri et al., 2017; Salem al., 2018); the model extraction attack can learn surrogate model that matches the target model, given the adversary only black-box access to the target model (Tramèr et al., 2016; Gong & Liu, 2018): the model inversion attack can infer certain features of given input from the output of target model (Fredrikson et al., 2015b; Al-Rubaie & Chang, 2016); the attribute inference attack can deanonymize the anonymized training data (Gong & Liu, 2016; Zheng al., 2018).
Machine learning (ML. with data privacy is crucial in many applications (Lindell & Pinkas, 2000; Barreno al., 2006; Hesamifard et al., 2018; Bae et al., 2019). Several algorithms have been developed to reduce privacy leakage include differential privacy (DP) (Dwork et al., 2006), federated learning (FL) (McMahan et al., 2016; Konečný et al., 2016). and k-anonymity (Sweeney, 2002; El Emam & Dankar, 2008). Objective, output, and gradient perturbations are among the most used approaches for ML with DP guarantees the cost of significant utility degradation (Chaudhuri et al., 2011: Bassily et al., 2014; Shokri & Shmatikov, 2015; Abadi et al., 2016b; Bagdasaryan et al., 2019). FL trains centralized ML models, through gradient exchange, with the training data being distributed at the edge devices. However, the gradient exchange still leak the privacy (Zhu al., 2019; Wang et al., 2019c). Most of the existing privacy is achieved at a tremendous sacrifice of utility. Moreover, training ML models using the tfte-of-thee-r art DP stochastic gradient descent (DPSGD) leads to treennoou computational cost due to the requirement of computing and clipping the per-sample privacy-preservin MLL
gradient (Abadi et al., 2016 It remains great interest to develop algorithms without excessive computational overhead or degrading the utility of the ML models.
1.1 OUR CONTRIBUTION
In this paper, we propose residual perturbation for srivacy-preserving deep learning (DL) with DP guarantees. At the core of residual perturbation is injecting Gaussian noise to each residual mapping

---

Under review as a conference paper at ICLR 2021
of ResNet (He et al., 2016), and the residual perturbation is theoretically principled by the stochastic differential equation (SDE) theory. The major advantages of residual perturbation are threefold:
It can protect the membership privacy of the training data almost perfectly and often without sacrificing ResNets' utility. Furthermore, it can even improve ResNets' classification accuracy.
It has fewer hyperparameters to tune than the benchmark DPSGD. Also, it is more computationally efficient than DPSGD, which requires to compute the per-sample gradient.
It can be easily implemented by few lines of code in modern DL libraries.
1.2 RELATED WORK
Improving the utility of ML models with DP guarantees is important task. PATE (Papernot et al., 2017: 2018) uses semi-supervised learning together with model transfer between the "student" and "teacher" models to enhance utility. Several variants of the DP notions have also been proposed to improve the privacy budget and some times can also improve the resulting model's utility at a given DP budget (Abadi et al., 2016b; Mironov, 2017; Wang et al., 2018; Dong et al., 2019). Some post-processing techniques have also been developed to improve the utility of ML models with negligible computational overhead (Wang et al., 2019a; Liang al., 2020). From the SDE viewpoint, (Li et al., 2019; Wang et al., 2015) showed that several stochastic gradient Monte Carlo samplers could reach state-of-the- performance in terms of both privacy and utility in Bayesian learning.
Gaussian noise injection in residual learning has been used to improve the robustness of ResNets (Rakin et al., 2018; Wang et al., 2019b; Liu et al., 2019). In this paper, we inject Gaussian noise to each residual mapping to achieve data privacy instead of adversarial robustness.
1.3 ORGANIZATION
We organize this paper as follows: In Section 2, we introduce the residual perturbation for privacy- preserving DL. In Section 3, we present the generalization and DP guarantees for residual perturbation. In Section 4, we numerically verify the efficiency of the residual perturbation in protecting data privacy without degrading the underlying models' utility. We end with some concluding remarks. Technical proofs and some more experimental details and results are provided in the appendix.
1.4 NOTATIONS
We denote scalars by lower or upper case letters; vectors/ matrices by lower/upper case bold face letters. For a vector æ (æ1, (..) Rd, we use ||œ||| (Σin |r; 12)1// to denote its &2 norm. For a matrix A, we use |A|| to denote its induced norm by the vector & norm. We denote the standard Gaussian in Rd as . (0, 1) with I E Rdxd being the identity matrix. The set of (positive) real numbers is denoted as (IR IR. We use (00, R) to denote the ball centered at with radius R.
2
ALGORIHTM
2.1 DEEP RESIDUAL LEARNING AND ITS CONTINUOUS ANALOGUE
Given the training set SN == (æ, 3i y. with {ai, vi) C Rd x E being data-label pair. For given w; the forward propagation of a ResNet with M residual mappings can be written as
x [+1 x' + F(x', WT), for I = 0, M -MM- with x"
fxx ),
(1)
where (., w is the nonlinear mapping of the /th residual mapping parameterized by W f is the output activation function, and ú; is the predicted label for 2; The heuristic continuum limit of (1) is
dx=()) = F (t) W()))dt, a0) á where the time variable.
The ordinary differential equation (ODE) (2) can be revertible, and thus the ResNet counterpart might be exposed data privacy leakage. For instance, we use the ICLR logo (Fig. I (a)) the initial data & in (2). Then we simulate the forward propagation of ResNet by solving (2) from t = 0 to t using the forward Euler solver with a time step size At 0.01 and a given velocity field ((x((), W(L) (see Appendix E for the details of (()()), W())), which maps the original image to its features (Fig. (b)). To recover the original image, we start from the feature and use the backward Fuler iteration, i.e., (t) a(t+ At) AtF((((t At), L++ At), to evolve #(() from / = 1 to / = 0 with ((1) = (11 being the features obtained in the forward propagation. We plot the recovered image from features in Fig. (c), and the original image can be almost perfectly recovered.
2

---

Under review as a conference paper at ICLR 2021
ICLR
CLR
ICLR
(0) (CLR logo)
(1) (ODE)
(c) (ODE)
#1 ISDE)
(()) (SDL).
Figure I: Illustrations of the forward and backward propagation of the training data using 2D ODE (2) and SDE (3) models. (a) the original image; (b) & (d) the features of the original image generated by the forward propagation using ODE and SDE, respectively; (c) & (e) the recovered images by reverse-engineeriing the features shown in (b) & (d), respectively. We see that it is easy to break the privacy of the ODE model, but harder for SDE.
2.2 RESIDUAL PERTURBATION AND ITS SDE ANALOGUE
In this part, we propose two SDE models to reduce the reversibility of (2), and the corresponding residual perturbations analogue can protect the data privacy in DL..
Strategy I. For the first strategy, we consider the following SDE model:
dx((() F(x((() W())dl + yd)((), 7 O,
where B(t) is the standard Brownian motion. We simulate the forward propagation and reverse- engineering the input from the output by solving the SDE model (3) with y using the same F(æx(L), (()) and initial data at We the following forward (4) and backward (5) Euler- Maruyama discretizations (Higham, 2001) of (3),
xtt + At) (t)) + (tF(2(t), W()) + (0 VAII,
(4)
a(t) a( At) (t At), W(( + A+) - ~WO, VATI).
(5)
for the forward and backward propagation, respectively. Figure (d) and (e) show the results of the forward and backward propagation by SDE, respectively, and these results show that it is much harder to reverse the features obtained by SDE evolution. The SDE model informs us to inject Gaussian noise, in both training and test phases, each residual mapping of ResNet to protect data privacy, which results
T ++1 = x + f('', w + yn', where n' ~ "
(6)
Strategy II. For the second strategy, we consider using the multiplicative noise instead of the additive noise used in (3) 2 and (6), and the corresponding SDE can be written as
dx(t) = F(x(t), W(t))dt + (a((t) @ dB(t), y>0, O,
(7)
where O denotes the Hadamard product. Similarly, we can use the forward and backward Euler- Maruyama discretizations of (7) propagate the image in Fig. (a), and we provide these results in Appendix D.1. The corresponding residual perturbation is
x i+1 x² F(x", W + 7w0 @ n', where n' (O,
again, the noise 7x0 n is injected to each residual mapping in both training and test phases.
We will provide theoretical guarantees for these two residual perturbation schemes, i.e., (6) and (8), in Section 3, and numerically verify their efficacy in Section 4.
2.3 UTILITY ENHANCEMENT VIA MODEL ENSEMBLE
Wang al. (2019b) showed that an ensemble of noise injected ResNets can improve models' utility. In this paper, we will also study the model ensemble for utility enhancement. We inherit notations from (Wang al., 2019b), e.g., denote an ensemble of two noise injected ResNet8 as ResNes.
3 MAIN THEORY
In this section, we will provide theoretical guarantees for the above two residual perturbations.
Liu et al. (2019); Wang et al. (2019b) used this noise injection to improve robustness of ResNets. 2 Liu al. (2019) injected multiplicative noise neural networks to improve their robustness.
3

---

Under review as a conference paper at ICLR 2021
3.1 DIFFERENTIAL PRIVACY GUARANTEE FOR STRATEGY
We consider the following function class for ResNets with residual perturbation:
Fi = [f()) = w T xM | = x + (U'x') + yn', ; ,M....MM 1, x input data + m. n2 and n² ~ (O(,,
(9)
R U Rdxd),
where x0 Fd is the noisy input U is the weight matrix in the ith residual mapping and w Rd is the weights of the last layer. 7, > O are hyperparameters. 6 = (N( with B.N being the batch normalization and du being a I.-Lipschitz and monotonically increasing activation function (e.g.. ReLU). We first recap on the definition of differential privacy below.
Definition 1 ((c, 8)-DP). (Dwork et al., 2006JA randomized mechanism M : S - R. satisfies (e, 6 for any two datasets S,S' 8N that differ by one element, and any output subset O S R. it holds that PIM(S) E O] e -P(M((S" E
E O] + 6, where ō E (0, and e > 0.
We have the following DP guarantee for Strategy I, and we provide its proof in Appendix A.
Theorem 1. Assume the input to ResNet lies in ( R) and the output of every residual mapping is normal distributed and bounded by G. in E2 norm, in expectation. Given the total number of iterations' I used for training ResNet. For any 0 and 8 À E (0, 1), the parameters U and in the ResNet with residual perturbation satisfies ((X/+ (1
A))e, D) and ((// + (1 A))c, 5)- DP, respectively, provided that π >> R (2((2bb))(NNN and > G (2Tbbba) (NN)e) where a = ( ((1 (1-AA X)e) + 1, M, and b are the number of residual mappings, training data, and batch size, respectively. In particular, when yy > G (2Tbbbx))// (N MM)) the whole model obtained by injecting noise according to strategy I satisfies (e, ()DP).
3.2 THEORETICAL GUARANTEES FOR STRATEGY II
Privacy. To analyze the residual perturbation (8), we consider the following function class:
M + Ta. M nl # = xx' + + (U'x*' + 7 @ n²), F2 = (s (x) w T x
(10)
i= 0, M 1, n (O, I), || s a)
where a is a constant; we denote the entry of x that has the largest absolute value x xmazz and ä' is defined as (sgn(z)) (z)) max( (rgl, Due to batch normalization, we assume & can be bounded by a positive constant B. The other notations are defined similar to that in (9).
Consider training F2 by using two different datasets 5 and S", and we denote the resulting models as: (x == w M + Tx" n:; x'11= = x' + Φ (Uix') + yāt @ nn, i == 0, ... .M - 1. (11) xx' / (x)5 w = M.M + raa n a =x' + tb (U2 + 7 2 n -- 0, ..... ,MM - (12)
Theorem 2. For ((](S) and f(x)S")) that are defined in (11) and (12), respectively. Let X E (0, 1),8 (0, and 0, if > (B/n) (2aM))((2) and a /2(2))) /Xe, where log(1/6)/ ((1 A)e) + 1, then P[f()[S) E O S e -P[(a]|S)] E O]+ + 5 for any input æ and any subset O in the output space.
We provide the proof of Theorem in Appendix B. Theorem guarantees the privacy of the training data given only black-box access to the model, i.e., the model will output the prediction for any input without granting adversaries access to the model itself. In particular, we cannot infer whether the model is trained S or S' no matter how query the model in black-box fashion. We leave the theoretical DP- guarantee for for Strategy II as a futuree work.
Generalization Gap. Many works have shown that overfitting in training ML models leads to privacy leakage (Salem et al., 2018), and reducing overfitting can mitigate data privacy leakage (Shokri et al., 2017; Yeom et al., 2018; Sablayrolles et al., 2018; Salem al., 2018; Wu al., 2019b). In this part, we will show that the residual perturbation (8) reduce overfitting via computing the Rademacher complexity. For simplicity, we consider binary classification problems. Suppose SN =(x,, y J.M,, is drawn from X x C +-1,+1 with X and Y being the input data and label
We add noise to the input for DP guarantee
4

---

Under review as a conference paper at ICLR 2021
spaces, respectively. Assume D is the underlying distribution of X x Y. which is unknown. Let H C V be the hypothesis class of the ML model. We first recap on the definition of Rademacher complexity.
Definition 2. (Barlett & Mendelson, 2002) Let H X + R be the space of real-valued functions on the space X. For given sample S = {I1. 22:
EN I of size N, the empirical Rademacher
complexity of H is defined
N
1 (H( = N
Ea sup hER
och(a:)),
where 01; 02," " NN are i..dd Rademacher random variables with P(o; = = P(a; -1)
Rademacher complexity is tool bound the generalization gap (Barlett & Mendelson, 2002). The smaller the generalization gap is, the less overfitting the model is. For Va, E E and constant 2 0, we consider the following two function classes:
F == {f(x, w) = wx (x)(T))da (dx)) = (2 dt, =0) = हùi w E Rxdd U Baxd]
with || | 2 S c).
9 = {ax, w) = E (wxx (T)) (dx(()) (x)(d (1) dB((), =(0) 0s E Rixd
with U E Reddd |/ |U|| Sc).
where 0 < < takes the value such that x" is well defined on the whole Rd 7 > 0 is hyperparameter and U is a eirculant matrix that corresponding to the convolution layer in DNs. B()) being the ID Brownian motion. The function class F represents the continuous analogue of ResNet without inner nonlinear activation functions, and 9 denotes F with the residual perturbation (8).
Theorem 3. Given the training set SN = {a:; yi ) We have R (9) < RSN (F)-
We provide the proof of Theorem 3 in Appendix C, where we will also provide quantitative lower and upper bounds of the above Rademacher complexities. Theorem 3 shows that residual perturbation (8) can reduce the generalization error. We will numerically verify this generalization error reduction for ResNet with residual perturbation in Section 4.
4 EXPERIMENTS
In this section, we will numerically verify that 1) can residual perturbation protect data privacy: in particular, membership privacy? ? 2) can the ensemble of ResNets with residual perturbation improve the classification accuracy? 3) are skip connections crucial in residual perturbation for DL. with data privacy? 4) what the advantage of the residual perturbation over the DPSGD? We focus on Strategy I in this section, and we provide the results of Strategy II in Appendix D.
4.1 PRELIMINARIES
Datasets. We consider both CIFAR10/CIFARIU (Krizhevsky et al., 2009) and the Invasive Ductal Carcinoma (IDC) datasets. Both CIFARTO and CIFAR 100 contain 60K 32 x 32 color images with 50K and 10K of them used for training and test, respectively. The IDC dataset is a breast cancer- related benchmark dataset, which contains 277, 277,524 patches of the size 50 x 50 with 198,738 labeled negative (0) and 78,786 labeled positive (1). Figure 2 depicts few patches from the IDC dataset. For the IDC dataset, we follow Wu et al. (2019a) and split the whole dataset into training, validation, and test set. The training set consists of 10,788 positive patches and 29,16 negative patches, and the test set contains 11 11,55 positive patches and 31,825 negative patches. The remaining patches are used as the validation set. For each dataset, we split its training into Dshadow and Dtarget with the same size. Furthermore, we split Dshadow into halves with the same size and denote them as Dirsin and Dout - and split Dearget by half into Dirain and Dont The purpose of this splitting of the training set is for the membership inference attack, which will be discussed below.
Shadow
Membership inference attack. To verify the efficiency of residual perturbation for protecting data privacy, consider the membership inference attack (Salem et al., 2018) in all the experiments below. The membership attack proceeds as follows: 1) train the shadow model by using Drrain shadow 2)
apply the trained shadow model to predict all data points in D shadow and obtain the corresponding
5

---

Under review as a conference paper at ICLR 2021
Negative Figure Visualization of few selected images from the IDC dataset.
Positive
Negative
Positive
precision
recall
train and test accuracy
AUC under different noises
threshold
0.90
0.51
54.4
64.6
0.0
48.8
8
0.50
0.51
96.6
0.3
12.9
train
100.0
58.1
88.9
AUC
0.52
64.9
0.0
0.0
0.52
91.3
0.0
0.0
test
84.7
86.7
85.7
0.53
0.53
0.0
0.0
0,2
0.3
0.3
0.0
0.2
0.3
0.0
0.2
0.3
gamma
gamma
gamma
gamma
Figure 3: Performance of residual perturbation for Eng ResNet8 with different noise coefficients (7) and membership inference attack thresholds on the IDC dataset. Residual perturbation can significantly improve membership privacy and reduce the generalization gap. ? corresponding to the baseline ResNet8. (Unit: %)
classification probabilities of belonging each class. Then we take the top three classification probabilities (or two in the case of binary classification) to form the feature vector for each data point. A feature vector is tagged as 1 if the corresponding data point is in Dirain and 0 otherwise. Then we train the attack model by leveraging all the labeled feature vectors; 3) train the target model by using Dtrain and obtain feature vector for each point in Dtarget. Finally, we leverage the attack model decide whether a data point is in train
target
Experimental settings. We consider Eng ResNet8 (ensemble ResNet8 with residual perturba- tion) and the standard ResNet8 the target and shadow models. We multilayer perceptron with hidden layer of 64 nodes, followed by a softmax output function as the attack model, which is adapted from (Salem et al., 2018). We apply the same settings as that used in (He al., 2016) to train the target and shadow models on the CIFARIO and 100AR100. For training models on the IDC dataset, we run 100 epochs of SGD with the same setting as before except that decay the learning rate by 4 at the 20th, 40th, 60th, and 80th epoch, respectively. Moreover, we run 50 epochs of Adam (Kingma & Ba, 2014) with learning rate of 0.1 to train the attack model. For both IDC and CIFAR datasets, we set π as half of y in Strategy I. which simplifies hyperparameters calibration, and based on our experiment gives good trade-off between privacy and accuracy.
Performance evaluations. We consider both classification accuracy and capability for protecting membership privacy. The attack model is binary classifier, which is to decide if data point is in the training set of the target model. For any æ E Diarget, we apply the attack model to predict its probability (p) of belonging the training set of the target model. Given any fixed threshold t if p 2 f, we classify æ as a member of the training set (positive sample), and if p < t, we conclude that obtain different attack results with different
not in the training set (negative sample); thresholds. Furthermore, we can plot the ROC curve(see details in subsection 4.5) of the attack model and use the area under the ROC curve (AUC) an evaluation of the membership inference attack. The target model protects perfect membership privacy if the AUC is 0.5 (attack model performs random guess), and the higher AUC is, the less private the target model is. Moreover, we use the precision (the fraction of records inferred as members are indeed members of the training set) and recall (the fraction of training set that correctly inferred as members of the training set by the attack model) to measure ResNets' capability for protecting membership privacy.
4.2 EXPERIMENTS ON THE IDC DATASET
In this subsection, we numerically verify that the residual perturbation in protecting data privacy while retaining the classification accuracy on the IDC dataset. We select the ResNet8 as benchmark architecture, which has ResNet8 as its baseline architecture (the details of the neural architectures are provided in Appendix F). As shown in Figure attack results with three different noise coefficients (t) when ? = 0 means the standard ResNet8 without residual perturbation. We also depict the ROC curve for this experiment in Figure 7 (c).
set four different thresholds to obtain different

---

Under review as a conference paper at ICLR 2021
Table 1: Residual perturbation vs, DPSGD in training ResNet8 and EnResNet8 for the IDC classification. Ensemble of ResNet8 with residual perturbation has higher test accuracy and protects better membership privacy (smaller AUC).
ResNet8 (DPSGD)
Eng ResNei8
Eng Nei
AUC
Training Ace
Ave
AUC
Traiiin Ace
Acc
AUC
Training Ace
Test Ave
0.503
0.831
0.828
0.496
0.864
0.852
0.497
0.869
0.857
precision
recal
train and test accurac
AUC under different noises
threshold
0.50
593
55.2
$1.0
threshold
0.50
97.7
58.0
0,58
8
8088
56.2
0.55
96.7
train
90,3
05.7
AUC
0.757
0.644
0.673
0.622
0.65
0.60
70.4
0.0
0.0
6.0
0.0
0.0
0.0
0.65
0.60
94.9
88.1
test
68.9
67.5
0.0
0.5
0.75
0.0
0.75
0.75
0.0
gamma
gamma
gamma
gamma
Figure 4: Performance of Eng with residual perturbation using different noise coefficients (7) and membership inference attack threshold on CIFARIO. Residual perturbation can not only enhance the membership privacy, but also improve the classification accuracy. 7 = 0 corresponding to the baseline ResNet8 without residual perturbation or model ensemble. (Unit: %)
threshold
0.50
0.55
812
80.5
precision
592
09:4
54.5
62.0
0.0
@heehold
0.55
0.50
91.2
E3.1
43.2
reca
7.0
0.0
train
train and test accurac
99.9
63.1
40.6
31.0
AUC
AUC under different noises
0.003
0,581
0
0.515
0.60
0.65
82.5
81.9
63.5
62:1
0.0
0.0
0.0
0.65
6.60
85.5
88.7
0.4
0.0
0.0
test
30.5
35.6
302
25.8
0.0
0.5
0.75
1.0
0.0
0.5
0.75
1.0
0.0
0.5
0.75
1.0
0.0
1.0 gamma
0.5
0.75
gamma
gamma
gamma
Figure 5: Performance of ResNet8 with residual perturbation using different noise coefficients (γ) and membership inference attack threshold on CIFAR100. Again, residual perturbation can not only enhance the membership privacy, but also improve the classification accuracy. ? 0 corresponding to the baseline ResNet8 without residual perturbation or model ensemble. (Unit: %)
ResRes is remarkably better in protecting the membership privacy, and as increases the model becomes more resistant to the membership attack. Also, as the noise coefficient increases, the gap between training and test accuracies becomes smaller, which resonates with Theorem 3. For instance, when y = 0.2 the AUC for the attack model is 0.495 and 0.563, respectively, for Eng and ResNet8; the classification accuracy of Eng ResNet8 and ResNet8 are 0.867 and 0.847, respectively.
4.2.1 RESIDUAL PERTURBATION vs. DPSGD
In this part, we compare the residual perturbation with the benchmark Tensorflow DPSGD module (McMahan et al., 2018), and we calibrate the hyperparameters, including the initial learning rate (0.1) which decays by factor of after every 20 epochs, noise multiplier (1.1), clipping threshold (1.0), micro-batchen (128), and epochs (100)* 4 such that the resulting model gives the optimal trade-off between membership privacy and classification accuracy. DPSGD is significantly more expensive due to the requirement of computing the per-sample gradient. We compare the standard ResNet8 trained by DPSGD with En, ResNet8 and Eng ResNe88 with residual perturbation (7 = 0.3). Table lists the AUC of the attack model and training and test accuracies of the target model; we see that residual perturbation can improve accuracy and protect better membership privacy.
4.3 EXPERIMENTS ON THE CIFAR IA/CIOO/IFARII 100 DATASETS
We further test the residual perturbation for ResNet8 and En; ResNet8 on the CIFARIO/CIFARION dataset. Figure 4 plots the performance of En, ResNet8 the CIFAR10 dataset under the above four different measures. Again, the ensemble of ResNets with residual perturbation is remarkably less vulnerable to the membership inference attack; for instance, the AUC of the attack model for ResNet8 and Eng ResNe88 (7 0.75) is 0.757 and 0.573, respectively. Also, the classification accuracy of Net (67.5%) is higher than that of ResNet8 (66.9%) for CIFARIO classification. Figure depicts the results of ResRes for CIFAR 100 classification. These results confirm again that residual perturbation can protect membership privacy and improve classification accuracy.
7

---

Under review as a conference paper at ICLR 2021
ensemble num 5
ensemble num
ensemble num
AUC
train accuracy
test accuracy
1
0. 757
0.671
0.634
100.0
99.1
96.1
66.9
70.2
67.5
0.676
0.692
0.610
100.0
99.1
94.2 2
3
70.8 8
70.3
68.7
90906
0.644
0.673
100.0
96.5
86.7
S
67.9
69.3
67.5
0.0
0.5
0.75
0.0
0.5
0.75
0.0
0.5
0.75
gamma
gamma
gamma
Figure 6: The performance of residual perturbation with different noise coefficients (7) and different number of models in the ensemble. The optimal privacy-utility tradeoff lies in the choice of these two options. (Unit: %)
true postiive rate
true positive rate
true positve 4
1.0
1.0
1.0
0.8
0.8
0.8
gan0.0
gan0.0 0.0
00:0.0
0.2
ge-0.5 5
gam0.75
0.2
ge-0.5
ga^0.00
0.2
ga
0a:0.3
0.0
0.0
0.2 0.4 0.6 0.8
1.0
0.0
0.2
044 0.6 0.8 1,0
0.2
0.4 0.8 0.8 1.0
CIFARIO Reshett))
CIFARIOO (Eng ResNetS)
ResNet8)
Figure 7: ROC curves for different datasets. (ga: noise coefficient ?)
4.3.1 EFFECTS OF THE NUMBER OF MODELS IN THE ENSEMBLE
In this part, we consider the effects of the number of residual perturbed ResNets in the ensemble. Figure illustrates the performance of EnResNet8 for CIFARIO classification measured in the AUC, and training and test accuracy. These results show that tuning the noise coefficient and the number of models in the ensemble is crucial to optimize the trade-off between accuracy and privacy.
4.4 ON THE IMPORTANCE OF SKIP CONNECTIONS
Residual perturbations theoretically relies the irreversibility of the SDEs (3) and (7), and this ansatz. lies in the skip connections the ResNet. We test both standard ResNet and the modified ResNet without skip connections. For CIFAR1O classification, under the same noise coefficient (7 = 0.75), the test accuracy is 0.675 for the EngResNet8 (with skip connection); while the test accuracy 0.653 for the Eng ResNet8 (without skip connection). Skip connections makes EnResNet more resistant to noise injection which is indeed crucial for the success of residual perturbation for protecting data privacy.
4.5 ROC CURVES FOR EXPERIMENTS ON DIFFERENT DATASETS
The receiver operating characteristic (ROC) curve can be used to illustrate the classification ability of a binary classifier. ROC curve is obtained by plotting the true positive rate against the false positive rate at different thresholds. The true positive rate, also known as recall, is the fraction of the positive set (all the positive samples) that is correctly inferred as a positive sample by the binary classifier. The false positive rate can be calculated by I-specificity, where specificity is the fraction of the negative set (all the negative samples) that is correctly inferred as a negative sample by the binary classifier. In case, the attack model is a binary classifier. Data points in the training set of the target model tagged as positive samples, and data points out of the training set of the target model are tagged as negative samples. Then we plot ROC curves for different datasets (as shown in Figure 7). These ROC curves show that if y is sufficiently large, the attack model's prediction will be nearly random guess.
4.6 REMARK ON THE PRIVACY BUDGET
In the experiments above, we set the constants G and R to 30 for Strategy For classifying IDC with ResNet8, the DP budget for Strategy I is (e 1.1e5, 8 = 1e - 5) and the budget for DPSGD - 5). For classifying CIFAR10 with ResNet8, the DP budget for Strategy is (c 3e5, 5 = Ie 5) and the DP-budget for DPSGD is (c 22.33, 8 = le 5). Note that theorem offers a quite loose DP budget compared to DPSGD. There are several difficulties we need to overcome to get tight DP bounds for Strategy I. Compared to DPSGD, it is significantly harder. In particular, 1) the loss function of the nose injected ResNets is highly nonlinear and very complex with respect to the weights, also the noise term appears in the loss function due to the noise injected in each residual mapping. These together make the tight estimate very difficult. 2) In our proof, we leveraged the framework of subsampled Rényi-DP (Wang et al., 2018) to find feasible range of
(c 15. 15.79,66
8

---

Under review as a conference paper at ICLR 2021
noise variance parameter, and then convert to DP get the value of y for given DP budget. This procedure will significantly reduce the accuracy of the estimated y. We leave the tight DP guarantee future work. In particular, how to reduce the accuracy of estimating due to the conversion between Rényi-DP and DP.
5 CONCLUDING REMARKS
In this paper, we proposed residual perturbations, whose theoretical foundation lies in the theory of stochastic differential equations, to protect data privacy for deep learning. Theoretically, prove that the residual perturbation can reduce the generalization gap with differential privacy guarantees. Numerically, have shown that residual perturbations are effective for protecting membership privacy on some benchmark datasets. In particular, on the IDC benchmark, residual perturbations protect better membership privacy than sfate-of-the-an ahr- differentially private stochastic gradient descent and achieve remarkably better classification accuracy.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Invasive ductal carcinoma (ide) histology image dataset. URL. http:////wwwaaddeeeyjjooee/ooooo use-case-
luse-cuee--invvaive--uue wwwasive-ductal-caciooooo wee--iddddd-eee idd-segmeettttinn..
Martin Abadi, Andy Chu, Ian Goodfellow, H. McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In 23rd ACM Conference on Computer and Communica- tions Security (CCS 2016), 2016a.
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308 -188-18,777 2016b.
Mohammad Al-Rubaie and Morris Chang. Reconstruction attacks against mobile-based continuous authentication systems in the cloud. IEEE Transactions on Information Forensics and Security, 11 (12):2648 22648 2016.
Ho Bae, Dahuin Jung, and Sungroh Yoon. Anomigan: Generative adversarial networks for anonymiz- ing private medical data. arXiv preprint arXiv:/901. 2019.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. In Advances in Neural Information Processing Systems. pp. 15453- 15462, 2019.
Peter L Barlett and Sahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Jounal of Machine Leurning Research, 2002.
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and Doug Tygar. Can machine learning be secure? In Proceedings of the 2006 ACM Symposium on Information, computer and communications security, pp. 16-25, 2006.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 20/4 IEEE 55th Annual Symposium on Foundations of Computer Science, pp. 464-473. IEEE, 2014.
Kamalika Chaudhuri, Claire Monteleoni, and Anand Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(Mar):1069-1109, 2011.
Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. arXiv preprint arXiv:I905.n 023383 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Khaled El Emam and Fida Kamal Dankar. Protecting privacy using k-anonymity. Journal of the American Medical Informatics Association, (5(5):627-637) 2008.
Wei Feng, Zheng Yan, Hengrun Zhang, Kai Zeng, Xiao, and Y. Hou. Survey on Security, Privacy and Trust in Mobile Crowdsourcing. IEEE Internet of Things Journal, 2017.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model Inversion Attacks that Exploit Confi- dence Information and Basic Countermeasures. 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS 2015), 2015a.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322-1333. 2015b.
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study personalized warfarin dosing. In 23rd (USENIX) Security Symposium (( (USEEIIX) Security 14), pp. 17-32, 2014.
Neil Zhenqiang Gong and Bin Liu. You are who you know and how you behave: Attribute inference attacks via users' social friends and behaviors. In 25th {USENIX] Security Symposium (!USENIX) Security 16), pp. 979-995, 2016.
10

---

Under review as a conference paper at ICLR 2021
Neil Zhenqiang Gong and Bin Liu. Attribute inference attacks in online social networks. ACM Transactions on Privacy and Security (TOPS), 21(1):1330, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Ehsan Hesamifard, Hassan Takabi, Mehdi Ghasemi, and Rebecca N Wright. Privacy-preserving machine learning as service. Proceedings on Privacy Enhancing Technologies, 2018(3): 123--333:111333 2018.
Desmond Higham. An algorithmic introduction numerical simulation of stochastic differential equations. SIAM review, +3(3):525-544 2001.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Fima C Klebaner. Introduction to stochastic calculus with applications. Springer, 2005.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Bai Li, Changyou Chen, Hao Liu, and Lawrence Carin. On connecting stochastic gradient meme and differential privacy. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 557-566. PMLR, 2019.
Zhicong Liang, Bao Wang, Quanquan Gu, Stanely Osher, and Yuan Yao. Exploring private federated learning with laplacian smoothing. arXiv preprint 201::20055002188 2020.
Yehuda Lindell and Benny Pinkas. Privacy preserving data mining. In Annual International Cryptology Conference, pp. 36-54. Springer, 2000.
Xuanqing Liu, Tesi Xiao, SiSi, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint arXiv:/906.0222 /906.0235, 2019.
Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E. Dahl, and Martin C. Stumpe. Detecting Cancer Metastases on Gigapixel Pathology Images. 1rXiv::703.002444 2017.
Ledoux M and Talagrand. Probability in Banach spaces: Isoperimetry and processes. Springer, 2002.
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-effiiieenn learning deep networks from decentralized data. arXiv preprint arXiv: /62..066 2016.
Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, and Peter Kairouz. A general approach to adding differential privacy to iterative training procedures. arXiv preprint 18/ 2018
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Nicolas Paperot, Martín Abadi, Lfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semisupervised Knowledge Transfer for Deep Learning from Private Training Data. In 5th International Conference on Learning Representation (ICLR 2017), 2017.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, and Lfar Erlingsson. Scalable Private Learning with PATE. In International Conference on Learning Representations (ICLR 2018), 2018
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Parametric noise injection: Trainable random- ness to improve deep neural network robustness against adversarial attack. arXiv preprint arXiv: 181 1.093 10, 2018.
11

---

Under review as a conference paper at ICLR 2021
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. DV 'ejeaa an empirical evaluation of the memorization properties of convnets. arXiv preprint arXiv: 1809006399 2018.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint wrXiv:/806.0126. 2018.
Reza Shokri and Vitaly Shmatikov. Privacy-Preserving Deep Learning. 22nd ACM SIGSAC Conference Computer and Communications Security (CCS 2015), 2015.
Reza Shokri, Marco Stronati, Congzheng Song. and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp- 3-18. IEEE, 2017.
Latanya Sweeney. k-anonymity: A model for protecting privacy, International Journal of Uncertainty, Fuzziness and Knowledge-Beck Based Systems, 10(05):557-570, 2002.
Florian Tramèr, Fan Zhang. Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th {USENIX) Security Symposium ((USENIX) Security 16), pp. 601-618, 2016.
Bao Wang, Quanquan Gu, March Boedihardjo, Farzin Barekat, and Stanley J Osher. Dp-lssgd: A stochastic optimization method to lift the utility in privacy-preserving erm. arXiv preprint arXiv:/906. 20056, 2019a.
Bao Wang. Zuoqiang Shi, and Stanley Osher. Resnets ensemble via the feynman formalism to improve natural and robust accuracies. In Advances in Neural Information Processing Systems, pp. 1655-1665, 2019b.
Yu-Xiang Wang, Stephen Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In International Conference on Machine Learning, pp. 2502-2502, 2015.
Yu Xiang Wang, Borja Balle, and Shiva Kasiviswanathan. Subsampled R\ 'enyi Differential Privacy and Analytical Moments Accountant. arXiv preprint arXiv:18083 00087, 2018.
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Beyond inferring class representatives: User-level privacy leakage from federated learning. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 2512-2520. IEEE, 2019c.
Bingzhe Wu, Chaochao Chen, Shiwan Zhao, Cen Chen, Yuan Yao, Guangyu Sun, Li Wang, Xiaolu Zhang, and Jun Zhou. Characterizing membership privacy in stochastic gradient langevin dynamics. 2019a.
arXiv preprint ar Ar:iv
Bingzhe Wu, Shiwan Zhao, Guangyu Sun, Xiaolu Zhang, Zhong Su, Caihong Zeng, and Zhihong Liu. P3sgd: Patient privacy preserving sgd for regularizing deep cnns in pathological image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
pp.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268-282. IEEE, 2018
Yuen, Man-Ching, King, Irwin, Leung, and Kwong-Sak. A Survey of Crowdsourcing Systems. In Proceedings of the IEEE international conference on social computing (Socialcom 2011), 2011.
Xu Zheng, Zhipeng Cai, and Yingshu Li. Data linkage in smart internet of things systems: consideration from privacy perspective. IEEE Communicutions Magazine, 56(9):55-61, 2018.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, pp. 14747-14756, 2019.
12

---

Under review as a conference paper at ICLR 2021
The appendiees structured follows. In Sections A, B, and C, prove Theorems 1, 2, and 3, respectively. In Section D, we provide numerical results for the Strategy II. More experimental details provided in Section E. Finally, in Section F, summarize the architecture of deep neural networks used in our experiments.
Part
Appendices
Table of Contents
Proof Theorem
A.1 Rényi Differential Privacy
13
A.2 Proof of Theorem
14
B Proof of Theorem 2
15
Proof Theorem 3
Some Lemmas
................................
C.2 The proof of Theorem
................................
17
D Experiments on Strategy II
18
D.1 Forward and Backward Propagation Using (7)
................................
18
Experiments on the IDC Dataset..
................................
18
D.3 Experiments on the CIFARIOICIFARI00 Datasets
................................
D.4 ROC Curves for the Experiments Different Datasets
19
E More Experimental Details
19
Architectures of the Used DNs
20
A PROOF OF THEOREM
A.1 RÉNYI DIFFERENTIAL PRIVACY
We will use the notion of Rényi differential privacy (RDP) to prove the differential privacy (DP) guarantees for the proposed residual perturbations. First, let's review the definition and several results of the Rényi differential privacy (Mironov, 2017).
Definition 3. (Mironov, 2017) (Rényi divergence) For any two probability distributions P and Q defined over the distribution D, the Rényi divergence of order 1
1
Da(PIIQ)
1 log logge-o(P/QQ-
Definition 4. (Mironov, 2017) ((a, RDDPPP A randomized mechanism D - R is said have e-Rényi differential privacy order or (a,e) -RDP for short, if for any adjacent S,S' E D that differ by only one entry, it holds that
D. (M(())))M((') Se.
Lemma 1. (Mironov, 2017) Let J D - Ri be (or, 61 RDDP and Ri x D - - be (a, c)DD) then the mechanism defined as (X,Y) where X S(D) and Y (X, D), satisfies (a, + 62)- RDP.
Lemma 2. (Mironov, 2017) (From RDP to (c. 6D- If f is an (a, RDD mechanism, then also satisfies (e (logg)/(cc 1), d'dffferretii privacy for any 0 < 8 1.
13

---

Ep- For the remaining residual mappings, denote the response of Under review as a conference paper at ICLR 2021
Lemma 3. (Mironov, 2017) (Post- PPost-processing lemma) Let M : that is (a,e) c)DDP, and let R + R. be arbitrary randomized mapping. Then f(M()))
- R be randomized algorithm
R is (DD
A.2 PROOF OF THEOREM 1
In this subsection, we will give a proof of Theorem I, ie.. DP-guarantee for the Strategy 1.
Proof. We will prove Theorem by mathematical induction. Consider two adjacent datasets S = (. 11: NN J, { N Rx) that differ by one entry. For the first residual mapping, it is easy to check that when 7 Z Rv/2a/Ep we have (a ( a|løı
the ith residual mapping, for any two input data NN and di æjy. .'yy respectively. Based on our assumpution, we have
x'x + (U'x'))) N(
V + ((U')))) ~ V
where μu., and AN. are both bounded by the constant G. If Da(xw/llk I cp/i, according the post-procsssing lemma (Lemma 3), we have
alllN,, u',,ll?
Da'a'' + + ('''DN))l ll ((U'))))
cp/i,
20
so when 7 > 22GG2/ we further have
alluN, MN,
Da + <(U'
m + ((U4())) + +mm)
S cp
2(0 + 72)
i
which implies that D. (ar i+1 | [ < - On the other hand, note that
V 2 =1' (x**) + (U xx o' (U xN 12)
V = (# + ( o (UI+Ia (T
Leveraging the post- post-processing lemma (Lemma 3) again, we get
Da V Vu+ 0 - | U 4 (lay2) < Cp
i+1
Let B, be the index set with |B|| = b, and we update U as following:
U U a
Σ
V
( U
jeB.
U = U a Σ
Vo U
jEB.
where U is the weights updated after the fth training iterations. When N & Br it's obviously
that Da (u ++1 (| = 0; when N E B, the equations which we use to update U can be
rewritten as
Uttlls a
Σ
Vu 4t1 .F+1) a(1bb)V o; ((x))) U
JEB.-NN3
U
Σ
V Vitti .u+l) - m (axt) D
(EB:-(N)
According to the post-processing lemma (Lemma 3), we have Da (U | (S') S <p 1). Because there are only (Tb)/N steps where we use the information of æx and æN. Replace Ep by (Tbe))/N and composition theorem can get after T steps the output U satisfies (a, (p + RDD and w satisfies (a, e.),M--DDDP By Lemma 2, we can easily establish the guarrataeeee for Strategy I, as stated in Theorem 1.
14

---

Under review as a conference paper at ICLR 2021
B PROOF OF THEOREM 2
In this section, we will provide a proof for Theorem 2.
Proof. Let Φ- = (e)) where BN is batch normalization operation and ab an activation function. Because of the property of batch normalization, we assume that can bounded by positive constant B. To show that the model (9) guarantees training data privacy given only black-box access the model. Consider training the model (9) with two different datasets S and S' and we denote the resulting model as (-) and f-11S',, respectively. In the following, we prove that with appropriate choices of and (f [ (x](s)) Ey for any input
First, we consider the convolution layers, and let Conv(a), be the ith entry of the vectorized Conv( Then have Conv(x:); z; < (Uz)); +ā, For any two different training datasets, denote
Conv(x)) x; + ó x + yē, (zx + ((U1)); 2z)))
and
Conv(x);|5' = z; + @(U2z); + ~ N ( 2z7)
Therefore, if > (B/n) (2a (Ep). we have
Da (N (a. + Φ (UIz), 22) IW (a. + a (U2));,,
a(o(U1z); (U22))))
1
272212
27
2uB²
S C. 7T
Furthermore, 7 > (b/n) (2aM)/(Ep) guarantees (a, cp/M)-DDD for every convolution layer. For the last fully connected layer, if π > a (2aM)) /cp, we have
Da (N (wi x. n ,²x,) I
(w" a n x²)))
a (wn x: wh x))
4 ||a11
2T 12|2
2n 1212
2aa²
n² S E
ie.. т > (2 /ep guarantees that the fully connected layer to be (a, e//M)- According to Lemma 1, have (a, RDDP guarantee for the ResNet of M residual mappings if y > (Lb/n) (2adM))(())) and π > a (2aM) /cp. Let X (0, for any given (ep, 6) pair, if Ep I de and -(log (a 1), 8) (1 - A)e, then we have get the (e, 8)-P guarantee for the ResNet with M residual mapping using the residual perturbation Strategy II.
C PROOF OF THEOREM 3
this section, we will proof that the residual perturbation (8) can reduce the generalization error computing the Rademacher complexity. Let us first recap some related lemmas the stochastic differential equation and the Rademacher complexity.
C.I SOME LEMMAS
Let V Y + [0, B] be the loss function. Here we assume & is bounded and B positive constant. In addition, we denote the function class Ex = {(x, y) - {(h(=), v) h E H, (æ. y) E X Y). The goal of the learning problem is to find E H such that the population risk R(h) (( [((h(z), y)] is minimized. The gap between population risk and empirical risk Rsy (h)
(1/N) Σ ((((x,). 9:) is known as the generalization error. We have the following lemma and theorem connect the population and empirical risks via Rademacher complexity.
15

---

Under review as a conference paper at ICLR 2021
Lemma 4. (Ledoux- Talagrand inequality) (M & Talagrand, 2002)Let H. be bounded real valued function space and let Φ - R be a Lipschitz with constant and ((0) = 0. Then we have
ES
sup WEN
0. (h: _
Ea
sup hEH t=1
oah (mre)
Lemma 5. (Barlett Mendelson, 2002) SN {(x11 y1)..... .... (xN, yN }} be samples chosen i.d. according to the distribution D. If the loss function is bounded by B >> O. Then for any 8 e (0, 1), with probability at least 1 5, the following holds for all h E H.
R (h) S Rsx (h) 2BRsN (4x) (BBBBBOog(2/0)/222
In addition, according to the Ledoux-Talagren inequality and assume loss function is I-lippscittz we have
Rsv (Cx) S LRss (H).
So the population risk be bounded by the empirical risk and Rademacher complexity of the function class H. Because we can't minimize the population risk directly, we can minimize indirectly by minimizing the empirical risk and Rademacher complexity of the function class H. Next, we will further discuss Rademacher complexity of the function class H. We first introduce several lemmas below.
Lemma 6. (Klebaner, 2005) For any given matrix U, the solution to the equation
dmx(t) Un(t)dt xx0)) +
has the following expression
a11) = exp(UI)á,
Also, we can write the solution to the following equation
(dy(t) (y(t) +rv(t)dB(t) (y(0)= = à
1 v() exp(UI 7 -//I++B((0)) Obviously, we have (() = (().
Lemma 7. A matrix C E Rdxdd is circulant, if there exists real number a1. 04 such that
01
02
ad
a
a1
C
02
a1
For any circulant matrix, we have the following eigen- digee-decomppsitioon
q 11 CV = A
where
Vdv
mi
ma-1
m
m
and m& the roots of unity and X; 01 a2m4
nam d-1
16

---

Under review as a conference paper at ICLR 2021
C.2
THE PROOF OF THEOREM 3
Proof. By the definition of Rademacher complexity (Def. 2), we have
Rss (F) (1/N) Eg
sup
(T)
E.
sup
o
(T) 12
JEF
|
IaSe
Let u, = Var. and denote the jth element of u, as Wz,-- Then by lemma 7, we have
1/2
Rsy (F) = [c/N)), E. sup [
Σ
o,
N
= (c/N)) E. sup PA.
(1
Σ oi (u.,1 exp(A,Tp). ... Uy exp(AxTT))) 12)
N
(c/N) E. sup sai
Σ
Juu.y exp (A/Tp) 11/2= (c/N) exp(cTp)E,
Σ oi or
g2/2
jml
(c/N) exp (cTp) E,
Σ o,u, 2 (c/N) exp (cTp) E. V Σ 0
cx/N) E.
Σ o,
Note that E (wx((T)) wE (x. (T)) and according to Lemma 6, similar to proof for the function
class F we have
1/2
RSN (9) = (c/NN
sup
Σ (Ee
1UlbaSe
N (/ Σ
= (c/N)Et sup Dr.Se
oz (11, exp(A1 r(1 p)/2122),
U exp(Axtt p(1 - 2)/2T/2))
d
= [c/NN) sup SA.Se
Ui exp (A₁Tp - p(1 - p)/ p1/2
N
p(1 -2 p((
11/2
N = (cTTppxpcTTp-pp1 (1 E.A
a,u;ll 12
1=1
p1 77
E, 1
,ap! 1|2
i=1
(c/N) (cTp- p(1 p)/2T/2) E.
o ll2
Rsw (F)
Therefore, we have completed the proof for the fact that the ensemble of Gaussian noise injected ResNets can reduce generalization error compared the standard ResNet.
17

---

Under review as a conference paper at ICLR 2021
D EXPERIMENTS ON STRATEGY II
D.1 FORWARD AND BACKWARD PROPAGATION USING (7)
Figure plots the forward and backward propagation of the ICLR logo using the SDE model (7). Again, we cannot simply use the backward Euler-Maruyane discretization to reverse the features generated by the propagating through the forward Mareer-aauuyaaa discretization.
(a) (1) (SDE)
(b) ((0) (SDE)
Figure 8: Illustrations of the forward and backward propagation of the training data using the SDE model (7). (a) is the features of the original image generated by the forward propagation using SDE; (b) is the recovered images by reverse-engineering the features shown in (a).
D.2 EXPERIMENTS THE IDC DATASET
In this subsection, we consider the performance of the second residual perturbation in protecting membership privacy while retaining the classification accuracy on the IDC dataset. We use the same ResNet models as that used for the first residual perturbation. We list the results in Fig. 9, these results confirm that the residual perturbation (8) can effectively protect data privacy and maintain or improve the classification accuracy. In addition, depict the ROC curve for this experiment in Figure 12 (c). We note that, as the noise coefficient increases, the gap between training and testing accuracies narrows, which is consistent with Theorem 3.
precision
threshold
recall
train and test accurac
AUC under different noises
threshold
0.50
0.50
0.51
54.6
45.4
0.0
0.51
95.4
244
0.0
train
AUG
0.563
0.493
0.503
0.52
0.52
test
84.7
57.4
87.2
0.53
55.0
0.0
0.0
0.53
87.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
coefficient
coefficient
coefficient
noise coefficient
Figure 9: Performance of residual perturbation (8) for Eng with different noise coefficients (7) and membership inference attack thresholds on the IDC dataset. Residual perturbation can significantly improve membership privacy and reduce the generalization gap. ? corresponding the baseline ResNet8. (Unit: %)
D.2.1 RESIDUAL PERTURBATION VS. DIFFERENTIALLY PRIVATE STOCHASTIC GRADIENT DESCENT
We have shown that the first residual perturbation (6) outperforms the DPSGD in protecting mem- bership privacy and improving classification accuracy. In this part, we further show that the second residual perturbation (8) also outperforms the benchmark DPSGD with the above settings. Table lists the AUC of the attack model and training & test accuracy of the target model; the second residual perturbation can also improve the classification accuracy and protecting better membership privacy.
that
D.3 EXPERIMENTS THE
DATASETS
In this subsection, will test the second residual perturbation (8) the CIFARIO/CIFARION datasets with the same model using the same settings as before. Figure 10 plots the performance of Eng ResNet8 the CIFAR 10 dataset. These results show that the ensemble of ResNets with residual
18

---

Under review as a conference paper at ICLR 2021
Table 2: Residual perturbation (8) vs. DPSGD in training ResNet8 for the IDC dataset classification. Ensemble of ResNet8 with residual perturbation is more accurate for classification (higher test acc) and protects better membership privacy (smaller AUC).
ResNet8 (DPSGD)
Enj ResNet8
ResNet8
AUC
Training Ace
Test Acc
AUC
Training Acc
Test Ace
AUC
Training Acc
Test Acc
0.503
0.831
0.828
0.509
0,880
0.868
0.500
0.895
0.872
perturbation (8) is significantly more robust to the membership inference attack. For instance, the AUC of the attack model for ResNet8 and Ens.Res (7 2.0) is 0.757 and 0.526, respectively. Also, the classification accuracy of EngResNet8 (γ = 2.0) is higher than that of ResNet8, and their accuracy is 71.2% and 66.9% for CIFARIO classification. Figure 11 shows the results of EngResNet8 for CIFAR 100 classification. These results confirm that residual perturbation (8) can protect membership privacy and improve classification accuracy once more.
precision
threshold
recal
train and test accurac
AUC under different noises
threshold
0.55
0.50
88.6
67.3
54.3
5277
0.
0.0
0.50
0.85
95.7
59.5
91.3
66.5
25.6
90.0
0.0
634
0.0
train
100000000
00.
AUG
7.7 A 0.095 0.536 526
0.80
0.80
962
0.
test
727
72.8 71.2
0.65
204
68.5
00
08
0.
0.85
891
27.
00
0.0
0.0
0.0
0.5
1.0
1.5 2.0
0.0
0.5
1.0 1.5
2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
noise coefficient
noise coefficient
noise coefficient
noise coefficient
Figure 10: Performance of EngResNet8 with residual perturbation (8) using different noise co-
efficients (7) and membership inference attack threshold on CIFARI 10. Residual perturbation (8) can not only enhance the membership privacy, but also improve the classification accuracy. corresponding to the baseline ResNet8 without residual perturbation or model ensemble. (Unit: %)
precision
recall
train and test accurac
AUC under different noises
threshold
0.50
0.60
0.55
0.65
512
50.5
61.
52.5
755 53.7
24
73a
728
1
0.0
524
00
51.5
0.0
Em
0.50
0.60
0.55
98.1
88.7
75.
67.5
543
0.
4.0
0.0
43
0.0
0.0
0.0
train
test
41.7
59.5
AUG
0 TEA 010.855 52528 6.121
2.0
1.0
0.0 0.5 1.0 1.5 2.0
noise coefficient
noise coefficient
noise coefficient
noise coefficient
Figure 11: Performance of En; ResNet8 with residual perturbation (8) using different noise coeffi- cients (7) and membership inference attack threshold CIFARI00. Again, residual perturbation (8) can not only enhance the membership privacy, but also improve the classification accuracy. y = corresponding to the baseline ResNet8 without residual perturbation model ensemble. (Unit: %)
D.4 ROC CURVES FOR THE EXPERIMENTS ON DIFFERENT DATASETS
Figure 12 plots the ROC curves for the experiments on the different datasets with different models using the second residual perturbation strategy. These ROC curves again show that if is sufficiently large, the attack model's prediction will be nearly random guess.
true positive rate
1.0
@ 1.0
0.8
true positvve
0.8
0.8
0.2
0.4
1,0100
0,2
0.4
non0.6
n271.0
0
0.2
0.4
np-1 1.0
5
0.0
0.0
0.6
CIFARIO (esNee))))
CIFARIO0 (Eng ResNetS)
(Bng ReaNer8)
Figure 12: ROC curves for different datasets. (nc: noise coefficient)
E MORE EXPERIMENTAL DETAILS
We give the detailed construction of the velocity field F(æ(t),, (t)) in (3) and (7) that used to generate Figs. I and 8 in Algorithm 1.
19

---

Under review as a conference paper at ICLR 2021
Algorithm 1 The expression of ((x(t), W(t))
Input: image=æ (); rows, cols, channels = image.shape. Output: F(x(t), (iittiiiee
for k in range(channels) do
for i range(rows):
for j in range(cols): do offset(j) =int( + (ww.0coss2/i1/1000 (0ommm///))))
offset(() =int(i 50.0sin ( 18]] (roww;;
dirtyimageli.33
image](: (i + (ilset())))rows.. (j + oflsst()))%cols, k]:
return dirtyimage
F ARCHITECTURES OF THE USED DNs
Figure 13 shows the architectures of ResNets used in this paper, and we plot basic blocks in Fig. 14.
PreactResNet8
input
3+3 16
Busic Block1
Dasic Block2
Basic Block3
BN-ReLU
avgpool
Figure 13: Architectures of the ResNet8 used in our experiments.
20

---

Under review as a conference paper at ICLR 2021
Basic Block1
input
Basic Block2
Basic Block3
input
input
BN+ReLU
BN-ReLU
BN+ReLU
3+3 Conv, 16
3"3 Conv,32, /2
3*3 Conv,64, /2
BN+ReLU
BN+ReLU
BN+ReLU
3*3 16nv,16
3 Conv,32
3*3 Conv;64
(a) Basic Block1
(b) Basic Block2
(c) Basic Block3
Figure 14: Architectures of the basic building block of ResNets studied in this paper.
21


Reviews:
[{"review_id": "waKnmaPJa6I", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper presents a method for training ResNets with differential privacy. Rather than the usual methods based on noisy gradient descent, the authors propose adding noise at each layer of the network during both training and testing. The authors prove differential privacy guarantees for two strategies of this type (one with additive and one with multiplicative noise). They also show some evidence that the noise can help generalization, by showing that the Rademacher complexity of a continuous linearized version of the model is lower when noise is added.\n\n### Evaluation: Theory\n\nThe theoretical results are a start, but have some limitations which seem significant to me:\n\n* Theorem 1 needs the output of any residual mapping to be bounded in expectation. It is not clear whether this actually holds.\n\n* Theorem 2 only gives privacy guarantees for a single prediction. This is interesting but not as a strong as outputting the model.\n\n* Theorem 3 is about a continuous linearized analogue of the models. It also does not have quantitative bounds on how much the Rademacher complexity can improve.\n\n### Evaluation: Experiments\n\nThe experiments are promising: the authors run a membership inference attack against their models and against a model trained with DPSGD, to test privacy. They also test accuracy. I am, however, bothered that there is no attempt to give tight differential privacy bounds and to compare the two algorithms with choices of parameters that give the same provable differential privacy bounds. The membership inference attack is supposed to be a proxy for that but I am not convinced that just running one attack and getting a slightly lower AUC is good evidence that the proposed model preserves privacy as strongly as DPSGD. In general, I do not want to see work on privacy in ML adopt the strategy of running a single membership inference attack to verify privacy. This is not convincing: what if a slightly different attack does a lot better? The point of rigorous privacy guarantees is that they hold against all attacks. \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "tmVzgjbFJhL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper at question tackles the well-known problem of differentially private (DP) deep learning:  already for moderate privacy guarantees, the model performance suffers greatly.\n\nThe paper proposes a particular SDE based  method for obtaining privacy for ResNets. DP and stochastic differential equations have been considered in conjunction before e.g. in\n\nWang, Y.X., Fienberg, S. and Smola, A., 2015, June. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In International Conference on Machine Learning (pp. 2493-2502).\n\nLi, B., Chen, C., Liu, H. and Carin, L., 2019, April. On connecting stochastic gradient MCMC and differential privacy. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 557-566). PMLR.\n\n(you might consider adding these references).\n\nAlthough providing an interesting approach by combining DP deep learning and SDEs, I think the paper has some major deficits.\nOne of them is certain sloppiness with the presentation. I do not understand how the forward and backward Euler discretisations correspond to backward and forward propagation of ResNet layers. The main results are regarding the DP-privacy guarantees of the method (Thm. 1 and Thm. 2).\nHowever these DP-guarantees are not used anywhere in the experiments, and it remains a mystery to me what is actually DP protected.\nAnother criticism is regarding the experiments: when comparing to DP-SGD, there are no eps,delta-values given, it remains mystery how private the method is and how private is DP-SGD for the choice of hyperparameters listed.\n\nThe paper would require a major revision and therefore I cannot recommend it for publication in ICLR.\n\nEDIT: The authors have answered my questions and it clarified a lot. Thus I raise my score by one. However, I am still suspicious about the value of the DP result: for example, it is not discussed why would the residual mapping have an L2-sensitivity G (as stated in the assumptions of the theorem). Also, the reported privacy values in the end of the revised version of the paper (epsilon > 1000) are not meaningful. As far as I see, the experiments give an example where this given membership inference attack works better for DP-SGD protected model than for this residual perturbed version. However the paper does not give any privacy guarantees for the residual perturbation method. Whether the L2-sensitivity can be obtained with e.g. batch normalisation remains unclear to me. I think that the paper would require a careful rewriting.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mSkoeA811Y2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on the topic of differentially private deep learning. Specifically, based on the deep residual learning, they first see it as an ODE. Then, to reduce the reversibility of the ODE, they modify the model as an SDE. By discretizing the SDE, they get a perturbed version of residual learning and use this to design DP-algorithms. The first strategy is directly followed the SDE while the second strategy is with an addition multiplicative noise of the additive noise. Finally, they show that their methods to defend membership inference attack both theoretically and practically. \nI tend to accept the paper since I think the paper is well-motivated, also there are privacy guarantees and some theoretical results on defending privacy attack (Theorem 3). However, I still have the following concerns: \n\n1) It is notable that compared with Strategy 1(S1), S2 can only preserve the prediction privacy (since there is and addition multiplicative noise of the additive noise). So in my opinion, comparing the Rademacher complexity between S1 and S2 is unnecessary. So I want to see more comments about this. \n2) Moreover, the motivation of S1 is clear which is just followed by the SDE. However, the motivation of S1 is unclear, why the authors add an addition multiplicative noise to the additive noise? Is there any other previous work on it? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2U27eLD-Onv", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper studies an important problem and proposes the novel residual perturbation to protect privacy while maintaining the ResNet models’ utility.  Two SDE models are provided to inject noises with abundant theoretical proof are provided. Experimental results demonstrate the performance of privacy protection and classification accuracy on benchmark datasets. My major concern is about the utility enhancement and the DP guarantee (see cons below). Hope the authors can address my concern in the rebuttal period.\n\nPros: \n1. The paper studies a fundamental problem on how to protect privacy while retaining model utility. The problem itself will have great impacts on real-world scenarios. \n2. The proposed two strategies are novel for injecting noise to each residual mapping of ResNet theoretically principled by the stochastic differential equation theory. Great amounts of theoretical proof are provided and seem technically sound.\n3. Experiments on the real-world dataset provide some interesting insights about the advantages of residual perturbation.\n\nCons: \n1. Experiments demonstrate that utility is enhanced after noise injection. In general, there is a tradeoff between privacy and utility. This paper can increase both, and the authors owe this enhancement to the ensemble of noise injected ResNets. It seems not very clear to me that how to conduct the ensemble. What’s more, if it is just because of the ensemble, what is the contribution of this paper?\n2. There should be proof that each iteration of the whole model by strategy I and II can satisfy DP, while authors only prove the parameters can satisfy DP.\n\nQuestions during the rebuttal period: Please address and clarify the cons above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes techniques for differentially private training of ResNets inspired by SDEs. The idea has some promise but the paper does not give convincing evidence, either theoretical or empirical that it outperforms existing tehniques. Unfortunately, comparisons with existing techniques are presented in a misleading way that does not clearly provide the privacy parameters. 
Another issue with the work is that the authors appear to be unaware of (and hence do not compare with) existing work on privacy of a single prediction (referred to as strategy II in this work). 
Approaches for this problem are described in these theoretical works (and several follow ups)

* Dwork, Feldman. Privacy-preserving Prediction. COLT 2018
* Bassily,Thakkar,Tahkurta. Model-Agnostic Private Learning via Stability. NIPS 2018

Practical results are mentioned in PATE papers of Papernot et al. and more recent work 
* van der Maaten,  Hannun. The Trade-Offs of Private Prediction https://arxiv.org/abs/2007.05089
Decision: Reject