Year: 2020

Abstract:
Protecting data privacy in deep learning (DL) is at its urgency. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many of the existing mechanisms achieve data privacy at the cost of significant utility degradation. In this paper, we propose a stochastic differential equation principled residual perturbation for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap for DL. Empirically, we show that residual perturbation outperforms the stateof-the-art DP stochastic gradient descent (DPSGD) in both membership privacy protection and maintaining the DL models’ utility. For instance, in the process of training ResNet8 for the IDC dataset classification, residual perturbation obtains an accuracy of 85.7% and protects the perfect membership privacy; in contrast, DPSGD achieves an accuracy of 82.8% and protects worse membership privacy.

heading: 1 INTRODUCTION
text: Many high-capacity deep nets (DNs) are trained with private data, including medical images and financial transaction data (Yuen et al., 2011; Feng et al., 2017; Liu et al., 2017). DNs usually overfit and can memorize the private training data, which makes training DNs exposed to data privacy leakage (Fredrikson et al., 2015a; Shokri et al., 2017; Salem et al., 2018; Yeom et al., 2018; Sablayrolles et al., 2018). Given a pre-trained DN, the membership inference attack can determine if an instance is in the training set based on DN’s response (Fredrikson et al., 2014; Shokri et al., 2017; Salem et al., 2018); the model extraction attack can learn a surrogate model that matches the target model, given the adversary only black-box access to the target model (Tramèr et al., 2016; Gong & Liu, 2018); the model inversion attack can infer certain features of a given input from the output of a target model (Fredrikson et al., 2015b; Al-Rubaie & Chang, 2016); the attribute inference attack can deanonymize the anonymized training data (Gong & Liu, 2016; Zheng et al., 2018).
Machine learning (ML) with data privacy is crucial in many applications (Lindell & Pinkas, 2000; Barreno et al., 2006; Hesamifard et al., 2018; Bae et al., 2019). Several algorithms have been developed to reduce privacy leakage include differential privacy (DP) (Dwork et al., 2006), federated learning (FL) (McMahan et al., 2016; Konečnỳ et al., 2016), and k-anonymity (Sweeney, 2002; El Emam & Dankar, 2008). Objective, output, and gradient perturbations are among the most used approaches for ML with DP guarantees at the cost of significant utility degradation (Chaudhuri et al., 2011; Bassily et al., 2014; Shokri & Shmatikov, 2015; Abadi et al., 2016b; Bagdasaryan et al., 2019). FL trains centralized ML models, through gradient exchange, with the training data being distributed at the edge devices. However, the gradient exchange can still leak the privacy (Zhu et al., 2019; Wang et al., 2019c). Most of the existing privacy is achieved at a tremendous sacrifice of utility. Moreover, training ML models using the state-of-the-art DP stochastic gradient descent (DPSGD) leads to tremendous computational cost due to the requirement of computing and clipping the per-sample gradient (Abadi et al., 2016a). It remains a great interest to develop new privacy-preserving ML algorithms without excessive computational overhead or degrading the utility of the ML models.

heading: 1.1 OUR CONTRIBUTION
text: In this paper, we propose residual perturbation for privacy-preserving deep learning (DL) with DP guarantees. At the core of residual perturbation is injecting Gaussian noise to each residual mapping
of ResNet (He et al., 2016), and the residual perturbation is theoretically principled by the stochastic differential equation (SDE) theory. The major advantages of residual perturbation are threefold: • It can protect the membership privacy of the training data almost perfectly and often without
sacrificing ResNets’ utility. Furthermore, it can even improve ResNets’ classification accuracy. • It has fewer hyperparameters to tune than the benchmark DPSGD. Also, it is more computationally
efficient than DPSGD, which requires to compute the per-sample gradient. • It can be easily implemented by a few lines of code in modern DL libraries.

heading: 1.2 RELATED WORK
text: Improving the utility of ML models with DP guarantees is an important task. PATE (Papernot et al., 2017; 2018) uses semi-supervised learning together with model transfer between the “student” and “teacher” models to enhance utility. Several variants of the DP notions have also been proposed to improve the privacy budget and some times can also improve the resulting model’s utility at a given DP budget (Abadi et al., 2016b; Mironov, 2017; Wang et al., 2018; Dong et al., 2019). Some post-processing techniques have also been developed to improve the utility of ML models with negligible computational overhead (Wang et al., 2019a; Liang et al., 2020). From the SDE viewpoint, (Li et al., 2019; Wang et al., 2015) showed that several stochastic gradient Monte Carlo samplers could reach state-of-the-art performance in terms of both privacy and utility in Bayesian learning.
Gaussian noise injection in residual learning has been used to improve the robustness of ResNets (Rakin et al., 2018; Wang et al., 2019b; Liu et al., 2019). In this paper, we inject Gaussian noise to each residual mapping to achieve data privacy instead of adversarial robustness.

heading: 1.3 ORGANIZATION
text: We organize this paper as follows: In Section 2, we introduce the residual perturbation for privacypreserving DL. In Section 3, we present the generalization and DP guarantees for residual perturbation. In Section 4, we numerically verify the efficiency of the residual perturbation in protecting data privacy without degrading the underlying models’ utility. We end with some concluding remarks. Technical proofs and some more experimental details and results are provided in the appendix.

heading: 1.4 NOTATIONS
text: We denote scalars by lower or upper case letters; vectors/ matrices by lower/upper case bold face letters. For a vector x = (x1, · · · , xd) ∈ Rd, we use ‖x‖2 = ( ∑d i=1 |xi|2)1/2 to denote its `2 norm. For a matrix A, we use ‖A‖2 to denote its induced norm by the vector `2 norm. We denote the standard Gaussian in Rd as N (0, I) with I ∈ Rd×d being the identity matrix. The set of (positive) real numbers is denoted as (R+) R. We use B(0, R) to denote the ball centered at 0 with radius R.

heading: 2 ALGORITHMS

heading: 2.1 DEEP RESIDUAL LEARNING AND ITS CONTINUOUS ANALOGUE
text: Given the training set SN := {xi, yi}Ni=1, with {xi, yi} ⊂ Rd × R being a data-label pair. For a given xi the forward propagation of a ResNet with M residual mappings can be written as
xl+1 = xl + F̂ (xl,Wl), for l = 0, 1, · · · ,M − 1, with x0 = xi; ŷi = f(xM ), (1)
where F̂ (·,Wl) is the nonlinear mapping of the lth residual mapping parameterized by Wl; f is the output activation function, and ŷi is the predicted label for xi. The heuristic continuum limit of (1) is
dx(t) = F (x(t),W(t))dt, x(0) = x̂, where t is the time variable. (2)
The ordinary differential equation (ODE) (2) can be revertible, and thus the ResNet counterpart might be exposed to data privacy leakage. For instance, we use the ICLR logo (Fig. 1 (a)) as the initial data x̂ in (2). Then we simulate the forward propagation of ResNet by solving (2) from t = 0 to t = 1 using the forward Euler solver with a time step size ∆t = 0.01 and a given velocity field F (x(t),W(t)) (see Appendix E for the details of F (x(t),W(t))), which maps the original image to its features (Fig. 1 (b)). To recover the original image, we start from the feature and use the backward Euler iteration, i.e., x̃(t) = x̃(t+∆t)−∆tF (x̃(t+∆t), t+∆t), to evolve x̃(t) from t = 1 to t = 0 with x̃(1) = x(1) being the features obtained in the forward propagation. We plot the recovered image from features in Fig. 1 (c), and the original image can be almost perfectly recovered.

heading: 2.2 RESIDUAL PERTURBATION AND ITS SDE ANALOGUE
text: In this part, we propose two SDE models to reduce the reversibility of (2), and the corresponding residual perturbations analogue can protect the data privacy in DL.
Strategy I. For the first strategy, we consider the following SDE model:
dx(t) = F (x(t),W(t))dt+ γdB(t), γ > 0, (3)
where B(t) is the standard Brownian motion. We simulate the forward propagation and reverseengineering the input from the output by solving the SDE model (3) with γ = 1 using the same F (x(t),W(t)) and initial data x̂. We use the following forward (4) and backward (5) EulerMaruyama discretizations (Higham, 2001) of (3),
x(t+ ∆t) = x(t) + ∆tF (x(t),W(t)) + γN (0, √ ∆t I), (4)
x̃(t) = x̃(t+ ∆t)−∆tF (x̃(t+ ∆t),W(t+ ∆t)) + γN (0, √ ∆t I), (5)
for the forward and backward propagation, respectively. Figure 1 (d) and (e) show the results of the forward and backward propagation by SDE, respectively, and these results show that it is much harder to reverse the features obtained by SDE evolution. The SDE model informs us to inject Gaussian noise, in both training and test phases, to each residual mapping of ResNet to protect data privacy, which results in
xi+1 = xi + F̂ (xi,Wi) + γni, where ni ∼ N (0, I)1. (6)
Strategy II. For the second strategy, we consider using the multiplicative noise instead of the additive noise used in (3) 2 and (6), and the corresponding SDE can be written as
dx(t) = F (x(t),W(t))dt+ γx(t) dB(t), γ > 0, (7)
where denotes the Hadamard product. Similarly, we can use the forward and backward EulerMaruyama discretizations of (7) to propagate the image in Fig. 1 (a), and we provide these results in Appendix D.1. The corresponding residual perturbation is
xi+1 = xi + F̂ (xi,Wi) + γxi ni, where ni ∼ N (0, I), (8)
again, the noise γxi ni is injected to each residual mapping in both training and test phases. We will provide theoretical guarantees for these two residual perturbation schemes, i.e., (6) and (8), in Section 3, and numerically verify their efficacy in Section 4.

heading: 2.3 UTILITY ENHANCEMENT VIA MODEL ENSEMBLE
text: Wang et al. (2019b) showed that an ensemble of noise injected ResNets can improve models’ utility. In this paper, we will also study the model ensemble for utility enhancement. We inherit notations from (Wang et al., 2019b), e.g., we denote an ensemble of two noise injected ResNet8 as En2ResNet8.

heading: 3 MAIN THEORY
text: In this section, we will provide theoretical guarantees for the above two residual perturbations. 1Liu et al. (2019); Wang et al. (2019b) used this noise injection to improve robustness of ResNets. 2Liu et al. (2019) injected multiplicative noise to neural networks to improve their robustness.

heading: 3.1 DIFFERENTIAL PRIVACY GUARANTEE FOR STRATEGY I
text: We consider the following function class for ResNets with residual perturbation:
F1 := {f(x) = wTxM |xi+1 = xi + φ ( Uixi ) + γni, i = 0, · · · ,M − 1, (9)
x0 = input data + πn,n and ni ∼ N (0, I),w ∈ Rd,Ui ∈ Rd×d},
where x0 ∈ Rd is the noisy input 3, Ui is the weight matrix in the ith residual mapping and w ∈ Rd is the weights of the last layer. γ, π > 0 are hyperparameters. φ = BN(ψ) with BN being the batch normalization and ψ being a L-Lipschitz and monotonically increasing activation function (e.g., ReLU). We first recap on the definition of differential privacy below. Definition 1 (( , δ)-DP). (Dwork et al., 2006)A randomized mechanism M : SN → R satisfies ( , δ)-DP if for any two datasets S, S′ ∈ SN that differ by one element, and any output subset O ⊆ R, it holds that P[M(S) ∈ O] ≤ e · P[M(S′) ∈ O] + δ,where δ ∈ (0, 1) and > 0.
We have the following DP guarantee for Strategy I, and we provide its proof in Appendix A. Theorem 1. Assume the input to ResNet lies in B(0, R) and the output of every residual mapping is normal distributed and bounded by G, in `2 norm, in expectation. Given the total number of iterations T used for training ResNet. For any > 0 and δ, λ ∈ (0, 1), the parameters Ui and w in the ResNet with residual perturbation satisfies ((λ/i+ (1− λ)) , δ)-DP and ((λ/M + (1− λ)) , δ)DP, respectively, provided that π > R √ (2Tbα)/(Nλ ) and γ > G √ (2Tbα)/(Nλ ), where α = log(1/δ)/ ((1− λ) ) + 1, M,N and b are the number of residual mappings, training data, and batch size, respectively. In particular, when γ > G √ (2Tbα)/(NMλ ) the whole model obtained by injecting noise according to strategy I satisfies ( , δ)-DP.

heading: 3.2 THEORETICAL GUARANTEES FOR STRATEGY II
text: Privacy. To analyze the residual perturbation (8), we consider the following function class:
F2 := {f (x) = wTxM + πxMn| xi+1 = xi + φ ( Uixi ) + γx̃i ni), (10)
i = 0, · · · ,M − 1,ni ∼ N (0, I), ‖w‖2 ≤ a}
where a > 0 is a constant; we denote the entry of xi that has the largest absolute value as ximax, and x̃i is defined as (sgn(xij) max(|xij |, η))dj=1. Due to batch normalization, we assume φ can be bounded by a positive constant B. The other notations are defined similar to that in (9).
Consider training F2 by using two different datasets S and S′, and we denote the resulting models as: f (x|S) := wT1 xM + πxMnM ; xi+1 = xi + φ ( Ui1x i )
+ γx̃i ni, i = 0, · · · ,M − 1. (11) f ( x|S ′ )
:= wT2 x M + πxMnM ; xi+1 = xi + φ ( Ui2x i ) + γx̃i ni, i = 0, · · · ,M − 1. (12)
Theorem 2. For f(x|S) and f(x|S′) that are defined in (11) and (12), respectively. Let λ ∈ (0, 1), δ ∈ (0, 1), and > 0, if γ > (B/η) √ (2αM)/(λ ) and π > a √ (2αM) /λ , where α = log(1/δ)/ ((1− λ) ) + 1, then P[f(x|S) ∈ O] ≤ e · P[f(x|S′ ] ∈ O] + δ for any input x and any subset O in the output space.
We provide the proof of Theorem 2 in Appendix B. Theorem 2 guarantees the privacy of the training data given only black-box access to the model, i.e., the model will output the prediction for any input without granting adversaries access to the model itself. In particular, we cannot infer whether the model is trained on S or S′ no matter how we query the model in a black-box fashion. We leave the theoretical DP-guarantee for for Strategy II as a future work.
Generalization Gap. Many works have shown that overfitting in training ML models leads to privacy leakage (Salem et al., 2018), and reducing overfitting can mitigate data privacy leakage (Shokri et al., 2017; Yeom et al., 2018; Sablayrolles et al., 2018; Salem et al., 2018; Wu et al., 2019b). In this part, we will show that the residual perturbation (8) can reduce overfitting via computing the Rademacher complexity. For simplicity, we consider binary classification problems. Suppose SN ={xi, yi}Ni=1 is drawn from X × Y ⊂ Rd×{−1,+1} with X and Y being the input data and label
3We add noise to the input for DP guarantee
spaces, respectively. Assume D is the underlying distribution of X × Y , which is unknown. Let H ⊂ V be the hypothesis class of the ML model. We first recap on the definition of Rademacher complexity. Definition 2. (Barlett & Mendelson, 2002) Let H : X → R be the space of real-valued functions on the space X . For a given sample S = {x1,x2, · · · ,xN} of size N , the empirical Rademacher complexity ofH is defined as
RS(H) := 1
N Eσ[sup
h∈H N∑ i=1 σih(xi)],
where σ1, σ2, · · · , σN are i.i.d. Rademacher random variables with P(σi = 1) = P(σi = −1) = 12 .
Rademacher complexity is a tool to bound the generalization gap (Barlett & Mendelson, 2002). The smaller the generalization gap is, the less overfitting the model is. For ∀xi ∈ Rd and constant c ≥ 0, we consider the following two function classes:
F := {f(x,w) = wxp(T )|dx(t) = Ux(t)dt,x(0) = xi; w ∈ R1×d,U ∈ Rd×d} with ‖w‖2, ‖U‖2 ≤ c},
G := {f(x,w) = E (wxp(T )) |dx(t) = Ux(t)dt+ γx(t) dB(t),x(0) = xi; w ∈ R1×d, with U ∈ Rd×d, ‖w‖2, ‖U‖2 ≤ c},
where 0 < p < 1 takes the value such that xp is well defined on the whole Rd. γ > 0 is a hyperparameter and U is a circulant matrix that corresponding to the convolution layer in DNs. B(t) being the 1D Brownian motion. The function class F represents the continuous analogue of ResNet without inner nonlinear activation functions, and G denotes F with the residual perturbation (8). Theorem 3. Given the training set SN = {xi, yi}Ni=1. We have RSN (G) < RSN (F).
We provide the proof of Theorem 3 in Appendix C, where we will also provide quantitative lower and upper bounds of the above Rademacher complexities. Theorem 3 shows that residual perturbation (8) can reduce the generalization error. We will numerically verify this generalization error reduction for ResNet with residual perturbation in Section 4.

heading: 4 EXPERIMENTS
text: In this section, we will numerically verify that 1) can residual perturbation protect data privacy; in particular, membership privacy? 2) can the ensemble of ResNets with residual perturbation improve the classification accuracy? 3) are skip connections crucial in residual perturbation for DL with data privacy? 4) what is the advantage of the residual perturbation over the DPSGD? We focus on Strategy I in this section, and we provide the results of Strategy II in Appendix D.

heading: 4.1 PRELIMINARIES
text: Datasets. We consider both CIFAR10/CIFAR100 (Krizhevsky et al., 2009) and the Invasive Ductal Carcinoma (IDC) datasets. Both CIFAR10 and CIFAR100 contain 60K 32× 32 color images with 50K and 10K of them used for training and test, respectively. The IDC dataset is a breast cancerrelated benchmark dataset, which contains 277,524 patches of the size 50× 50 with 198,738 labeled negative (0) and 78,786 labeled positive (1). Figure 2 depicts a few patches from the IDC dataset. For the IDC dataset, we follow Wu et al. (2019a) and split the whole dataset into training, validation, and test set. The training set consists of 10,788 positive patches and 29,164 negative patches, and the test set contains 11,595 positive patches and 31,825 negative patches. The remaining patches are used as the validation set. For each dataset, we split its training set into Dshadow and Dtarget with the same size. Furthermore, we split Dshadow into two halves with the same size and denote them as Dtrainshadow and Doutshadow, and split Dtarget by half into D train target and D out target. The purpose of this splitting of the training set is for the membership inference attack, which will be discussed below.
Membership inference attack. To verify the efficiency of residual perturbation for protecting data privacy, we consider the membership inference attack (Salem et al., 2018) in all the experiments below. The membership attack proceeds as follows: 1) train the shadow model by using Dtrainshadow; 2) apply the trained shadow model to predict all data points in Dshadow and obtain the corresponding
classification probabilities of belonging to each class. Then we take the top three classification probabilities (or two in the case of binary classification) to form the feature vector for each data point. A feature vector is tagged as 1 if the corresponding data point is in Dtrainshadow, and 0 otherwise. Then we train the attack model by leveraging all the labeled feature vectors; 3) train the target model by using Dtraintarget and obtain feature vector for each point in Dtarget. Finally, we leverage the attack model to decide whether a data point is in Dtraintarget.
Experimental settings. We consider En5ResNet8 (ensemble of 5 ResNet8 with residual perturbation) and the standard ResNet8 as the target and shadow models. We use a multilayer perceptron with a hidden layer of 64 nodes, followed by a softmax output function as the attack model, which is adapted from (Salem et al., 2018). We apply the same settings as that used in (He et al., 2016) to train the target and shadow models on the CIFAR10 and CIFAR100. For training models on the IDC dataset, we run 100 epochs of SGD with the same setting as before except that we decay the learning rate by 4 at the 20th, 40th, 60th, and 80th epoch, respectively. Moreover, we run 50 epochs of Adam (Kingma & Ba, 2014) with a learning rate of 0.1 to train the attack model. For both IDC and CIFAR datasets, we set π as half of γ in Strategy I, which simplifies hyperparameters calibration, and based on our experiment it gives a good trade-off between privacy and accuracy.
Performance evaluations. We consider both classification accuracy and capability for protecting membership privacy. The attack model is a binary classifier, which is to decide if a data point is in the training set of the target model. For any x ∈ Dtarget, we apply the attack model to predict its probability (p) of belonging to the training set of the target model. Given any fixed threshold t if p ≥ t, we classify x as a member of the training set (positive sample), and if p < t, we conclude that x is not in the training set (negative sample); so we can obtain different attack results with different thresholds. Furthermore, we can plot the ROC curve(see details in subsection 4.5) of the attack model and use the area under the ROC curve (AUC) as an evaluation of the membership inference attack. The target model protects perfect membership privacy if the AUC is 0.5 (attack model performs random guess), and the higher AUC is, the less private the target model is. Moreover, we use the precision (the fraction of records inferred as members are indeed members of the training set) and recall (the fraction of training set that is correctly inferred as members of the training set by the attack model) to measure ResNets’ capability for protecting membership privacy.

heading: 4.2 EXPERIMENTS ON THE IDC DATASET
text: In this subsection, we numerically verify that the residual perturbation in protecting data privacy while retaining the classification accuracy on the IDC dataset. We select the En5ResNet8 as a benchmark architecture, which has ResNet8 as its baseline architecture (the details of the neural architectures are provided in Appendix F). As shown in Figure 3, we set four different thresholds to obtain different attack results with three different noise coefficients (γ) when γ = 0 means the standard ResNet8 without residual perturbation. We also depict the ROC curve for this experiment in Figure 7 (c).
En5ResNet is remarkably better in protecting the membership privacy, and as γ increases the model becomes more resistant to the membership attack. Also, as the noise coefficient increases, the gap between training and test accuracies becomes smaller, which resonates with Theorem 3. For instance, when γ = 0.2 the AUC for the attack model is 0.495 and 0.563, respectively, for En5ResNet8 and ResNet8; the classification accuracy of En5ResNet8 and ResNet8 are 0.867 and 0.847, respectively.

heading: 4.2.1 RESIDUAL PERTURBATION VS. DPSGD
text: In this part, we compare the residual perturbation with the benchmark Tensorflow DPSGD module (McMahan et al., 2018), and we calibrate the hyperparameters, including the initial learning rate (0.1) which decays by a factor of 4 after every 20 epochs, noise multiplier (1.1), clipping threshold (1.0), micro-batches (128), and epochs (100) 4 such that the resulting model gives the optimal trade-off between membership privacy and classification accuracy. DPSGD is significantly more expensive due to the requirement of computing the per-sample gradient. We compare the standard ResNet8 trained by DPSGD with En1ResNet8 and En5ResNet8 with residual perturbation (γ = 0.3). Table 1 lists the AUC of the attack model and training and test accuracies of the target model; we see that residual perturbation can improve accuracy and protect better membership privacy.

heading: 4.3 EXPERIMENTS ON THE CIFAR10/CIFAR100 DATASETS
text: We further test the residual perturbation for ResNet8 and En5ResNet8 on the CIFAR10/CIFAR100 dataset. Figure 4 plots the performance of En5ResNet8 on the CIFAR10 dataset under the above four different measures. Again, the ensemble of ResNets with residual perturbation is remarkably less vulnerable to the membership inference attack; for instance, the AUC of the attack model for ResNet8 and En5ResNet8 (γ = 0.75) is 0.757 and 0.573, respectively. Also, the classification accuracy of En5ResNet8 (67.5%) is higher than that of ResNet8 (66.9%) for CIFAR10 classification. Figure 5 depicts the results of En5ResNet8 for CIFAR100 classification. These results confirm again that residual perturbation can protect membership privacy and improve classification accuracy.
4https://github.com/tensorflow/privacy/tree/master/tutorials

heading: 4.3.1 EFFECTS OF THE NUMBER OF MODELS IN THE ENSEMBLE
text: In this part, we consider the effects of the number of residual perturbed ResNets in the ensemble. Figure 6 illustrates the performance of EnResNet8 for CIFAR10 classification measured in the AUC, and training and test accuracy. These results show that tuning the noise coefficient and the number of models in the ensemble is crucial to optimize the trade-off between accuracy and privacy.

heading: 4.4 ON THE IMPORTANCE OF SKIP CONNECTIONS
text: Residual perturbations theoretically relies on the irreversibility of the SDEs (3) and (7), and this ansatz lies in the skip connections in the ResNet. We test both standard ResNet and the modified ResNet without skip connections. For CIFAR10 classification, under the same noise coefficient (γ = 0.75), the test accuracy is 0.675 for the En5ResNet8 (with skip connection); while the test accuracy is 0.653 for the En5ResNet8 (without skip connection). Skip connections makes EnResNet more resistant to noise injection which is indeed crucial for the success of residual perturbation for protecting data privacy.

heading: 4.5 ROC CURVES FOR EXPERIMENTS ON DIFFERENT DATASETS
text: The receiver operating characteristic (ROC) curve can be used to illustrate the classification ability of a binary classifier. ROC curve is obtained by plotting the true positive rate against the false positive rate at different thresholds. The true positive rate, also known as recall, is the fraction of the positive set (all the positive samples) that is correctly inferred as a positive sample by the binary classifier. The false positive rate can be calculated by 1-specificity, where specificity is the fraction of the negative set (all the negative samples) that is correctly inferred as a negative sample by the binary classifier. In our case, the attack model is a binary classifier. Data points in the training set of the target model are tagged as positive samples, and data points out of the training set of the target model are tagged as negative samples. Then we plot ROC curves for different datasets (as shown in Figure 7). These ROC curves show that if γ is sufficiently large, the attack model’s prediction will be nearly a random guess.

heading: 4.6 REMARK ON THE PRIVACY BUDGET
text: In the experiments above, we set the constantsG andR to 30 for Strategy I. For classifying IDC with ResNet8, the DP budget for Strategy I is ( = 1.1e5, δ = 1e− 5) and the DP-budget for DPSGD is ( = 15.79, δ = 1e − 5). For classifying CIFAR10 with ResNet8, the DP budget for Strategy I is ( = 3e5, δ = 1e − 5) and the DP-budget for DPSGD is ( = 22.33, δ = 1e − 5). Note that theorem 1 offers a quite loose DP budget compared to DPSGD. There are several difficulties we need to overcome to get tight DP bounds for Strategy I. Compared to DPSGD, it is significantly harder. In particular, 1) the loss function of the nose injected ResNets is highly nonlinear and very complex with respect to the weights, also the noise term appears in the loss function due to the noise injected in each residual mapping. These together make the tight estimate very difficult. 2) In our proof, we leveraged the framework of subsampled Rényi-DP (Wang et al., 2018) to find a feasible range of
noise variance parameter, and then convert to DP to get the value of γ for a given DP budget. This procedure will significantly reduce the accuracy of the estimated γ. We leave the tight DP guarantee as future work. In particular, how to reduce the accuracy of estimating due to the conversion between Rényi-DP and DP.

heading: 5 CONCLUDING REMARKS
text: In this paper, we proposed residual perturbations, whose theoretical foundation lies in the theory of stochastic differential equations, to protect data privacy for deep learning. Theoretically, we prove that the residual perturbation can reduce the generalization gap with differential privacy guarantees. Numerically, we have shown that residual perturbations are effective for protecting membership privacy on some benchmark datasets. In particular, on the IDC benchmark, residual perturbations protect better membership privacy than state-of-the-art differentially private stochastic gradient descent and achieve remarkably better classification accuracy.

heading: A Proof of Theorem 1 13
text: A.1 Rényi Differential Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

heading: B Proof of Theorem 2 15

heading: C Proof of Theorem 3 15
text: C.1 Some Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 C.2 The proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

heading: D Experiments on Strategy II 18
text: D.1 Forward and Backward Propagation Using (7) . . . . . . . . . . . . . . . . . . 18 D.2 Experiments on the IDC Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D.3 Experiments on the CIFAR10/CIFAR100 Datasets . . . . . . . . . . . . . . . . 18 D.4 ROC Curves for the Experiments on Different Datasets . . . . . . . . . . . . . . 19

heading: E More Experimental Details 19

heading: F Architectures of the Used DNs 20

heading: A PROOF OF THEOREM 1
text: A.1 RÉNYI DIFFERENTIAL PRIVACY
We will use the notion of Rényi differential privacy (RDP) to prove the differential privacy (DP) guarantees for the proposed residual perturbations. First, let’s review the definition and several results of the Rényi differential privacy (Mironov, 2017). Definition 3. (Mironov, 2017) (Rényi divergence) For any two probability distributions P and Q defined over the distribution D, the Rényi divergence of order α > 1 is
Dα(P ||Q) = 1
α− 1 logEx∼Q(P/Q)α.
Definition 4. (Mironov, 2017) ((α, )-RDP) A randomized mechanismM : D → R is said to have -Rényi differential privacy of order α or (α, )-RDP for short, if for any adjacent S,S ′ ∈ D that differ by only one entry, it holds that
Dα(M(S)||M(S ′ ) ≤ ε.
Lemma 1. (Mironov, 2017) Let f : D → R1 be (α, 1)-RDP and g : R1 ×D → R be (α, )-RDP, then the mechanism defined as (X,Y ), where X ∼ f(D) and Y ∼ g(X,D), satisfies (α, 1 + 2)- RDP. Lemma 2. (Mironov, 2017) (From RDP to ( , δ)-DP) If f is an (α, )-RDP mechanism, then it also satisfies ( − (log δ)/(α− 1), δ)-differential privacy for any 0 < δ < 1.
Lemma 3. (Mironov, 2017) (Post-processing lemma) LetM : D → R be a randomized algorithm that is (α, )-RDP, and let f : R → R′ be an arbitrary randomized mapping. Then f(M(·)) : D → R′ is (α, )-RDP
A.2 PROOF OF THEOREM 1
In this subsection, we will give a proof of Theorem 1, i.e., DP-guarantee for the Strategy I.
Proof. We will prove Theorem 1 by mathematical induction. Consider two adjacent datasets S = {x1, · · · ,xN−1,xN},S ′ = {x1, · · · ,xN−1, x̂N} that differ by one entry. For the first residual mapping, it is easy to check that when γ ≥ R √ 2α/ p we have Dα(x1N ||x̂1N ) = α‖xN − x̂N‖2/(2γ2) < εp. For the remaining residual mappings, we denote the response of the ith residual mapping, for any two input data xN and x̂N , as xiN , x̂ i N , respectively. Based on our assumpution, we have
xiN + φ(U ixiN ) ∼ N (µN,i, σ2N,i)
x̂iN + φ(U ix̂iN ) ∼ N (µ̂N,i, σ2N,i),
where µN,i and µ̂N,i are both bounded by the constant G. If Dα(xiN ||x̂iN ) ≤ p/i, according the post-processing lemma (Lemma 3), we have
Dα(x i N + φ(U ixiN )||x̂iN + φ(Uix̂iN )) = α‖µN,i − µ
′ N,i‖2
2σ2N,i ≤ p/i,
so when γ > √ 2αG2/ p, we further have
Dα(x i N + φ(U ixiN ) + γn||x̂iN + φ(Uix̂iN ) + γn) = α‖µN,i − µ
′ N,i‖2
2(σ2N,i + γ 2)
≤ p i+ 1 ,
which implies that Dα(xi+1N ||x̂ i+1 N ) < p i+1 . On the other hand, note that ∇Ui+1`|xi+1N = l ′ ( xi+1N + φ ( Ui+1xi+1N )) φ ′ (Ui+1xi+1N )(x i+1 N ) T
∇Ui+1`|x̂i+1N = l ′ ( x̂i+1N + φ ( Ui+1x̂i+1N )) φ ′ (Ui+1x̂i+1N )(x̂ i+1 N ) T .
Leveraging the post-processing lemma (Lemma 3) again, we get
Dα(∇Ui+1`|xi+1N ||∇Ui+1`|x̂i+1N ) < p i+ 1 .
Let Bt be the index set with |Bt| = b, and we update Ui as following:
Ui+1t+1|S = Uit − α 1
b ∑ j∈Bt ∇Ui+1t `(x i+1 j ,U i+1 t )
Ui+1t+1|S ′ = Uit − α 1
b ∑ j∈Bt ∇Ui+1t `(x i+1 j ,U i+1 t ),
where Ui+1t is the weights updated after the tth training iterations. When N /∈ Bt, it’s obviously that Dα(Ui+1t+1|S||U i+1 t+1|S ′) = 0; when N ∈ Bt, the equations which we use to update Ui can be rewritten as
Ui+1t+1|S = Uit − α(1/b) ∑
j∈Bt−{N}
∇Ui+1t `(x i+1 j ,U i+1 t )− α(1/b)∇Ui+1t `(x i+1 N ,U i+1 t )
Ui+1t+1|S ′ = Uit − α(1/b) ∑
j∈Bt−{N}
∇Ui+1t `(x i+1 j ,U i+1 t )− α(1/b)∇Ui+1t `(x̂ i+1 N ,U i+1 t ).
According to the post-processing lemma (Lemma 3), we have Dα(Ui+1t+1|S||U i+1 t+1|S ′) ≤ p/(i+ 1). Because there are only (Tb)/N steps where we use the information of xN and x̂N . Replace p by (Tb p)/N and use composition theorem we can get after T steps the output Ui+1T satisfies (α, p/(i + 1))-RDP and w satisfies (α, p/M)-RDP. By Lemma 2, we can easily establish the DP-guarantee for Strategy I, as stated in Theorem 1.

heading: B PROOF OF THEOREM 2
text: In this section, we will provide a proof for Theorem 2.
Proof. Let φ = BN(ψ), where BN is batch normalization operation and ψ is an activation function. Because of the property of batch normalization, we assume that φ can be bounded by a positive constant B. To show that the model (9) guarantees training data privacy given only black-box access to the model. Consider training the model (9) with two different datasets S and S′, and we denote the resulting model as f(·|S) and f(·|S′), respectively. In the following, we prove that with appropriate choices of γ and π, Dα(f(x|S)||f(x|S ′ )) < p for any input x.
First, we consider the convolution layers, and let Conv(x)i be the ith entry of the vectorized Conv(x). Then we have Conv(x)i = xi+φ(Ux)i+γx̃ini. For any two different training datasets, we denote
Conv(x)i|S = xi + φ(U1x)i + γx̃ini ∼ N (xi + φ(U1x)i, γ2x̃2i ),
and Conv(x)i|S ′ = xi + φ(U2x)i + γx̃ini ∼ N (xi + φ(U2x)i, γ2x̃2i ).
Therefore, if γ > (B/η) √ (2αM)/( p), we have
Dα ( N ( xi + φ (U1x)i , γ 2x̃2i ) ||N ( xi + φ (U2x)i , γ 2x̃2i ))
≤ α(φ(U1x)i − φ(U2x)i) 2 2γ2η2 ≤ 4αL 2B2‖x‖22 2γ2x̃2i ≤ 2αB 2
γ2η2 ≤ p/M.
Furthermore, γ > (b/η) √
(2αM)/( p) guarantees (α, p/M)-RDP for every convolution layer. For the last fully connected layer, if π > a √ (2αM) / p, we have
Dα ( N ( wT1 x, π 2x̃2i ) ||N ( wT2 x, π 2x̃2i ))
= α ( wT1 x−wT2 x )2 2π2‖x‖2 ≤ 4αa 2‖x‖22 2π2‖x‖2
≤ 2αa 2
π2 ≤ p/M,
i.e., π > a √
(2αM) / p guarantees that the fully connected layer to be (α, p/M)-RDP. According to Lemma 1, we have (α, p)-RDP guarantee for the ResNet of M residual mappings if γ > (Lb/η) √ (2αdM)/( p) and π > a √ (2αM) / p. Let λ ∈ (0, 1), for any given ( p, δ) pair, if
p ≤ λ and −(log δ)/(α − 1), δ) ≤ (1 − λ) , then we have get the ( , δ)-DP guarantee for the ResNet with M residual mapping using the residual perturbation Strategy II.

heading: C PROOF OF THEOREM 3
text: In this section, we will proof that the residual perturbation (8) can reduce the generalization error via computing the Rademacher complexity. Let us first recap on some related lemmas on the stochastic differential equation and the Rademacher complexity.
C.1 SOME LEMMAS
Let ` : V × Y → [0, B] be the loss function. Here we assume ` is bounded and B is a positive constant. In addition, we denote the function class `H = {(x, y) → `(h(x), y) : h ∈ H, (x, y) ∈ X × Y }. The goal of the learning problem is to find h ∈ H such that the population risk R(h) = E(x,y)∈D[`(h(x), y)] is minimized. The gap between population risk and empirical risk RSN (h) = (1/N) ∑N i=1 `(h(xi), yi) is known as the generalization error. We have the following lemma and theorem to connect the population and empirical risks via Rademacher complexity.
Lemma 4. (Ledoux-Talagrand inequality) (M & Talagrand, 2002)LetH be a bounded real valued function space and let φ : R→ R be a Lipschitz with constant L and φ(0) = 0. Then we have
1 n Eσ [ sup h∈H n∑ i=1 σiφ (h (xi)) ] ≤ L n Eσ [ sup h∈H n∑ i=1 σih (xi) ] .
Lemma 5. (Barlett & Mendelson, 2002)Let SN = {(x1, y1) , · · · , (xN , yN )} be samples chosen i.i.d. according to the distribution D. If the loss function ` is bounded by B > 0. Then for any δ ∈ (0, 1), with probability at least 1− δ, the following holds for all h ∈ H,
R (h) ≤ RSN (h) + 2BRSN (`H) + 3B √ (log(2/δ)/(2N).
In addition, according to the Ledoux-Talagrand inequality and assume loss function is L-lipschitz, we have
RSN (`H) ≤ LRSN (H) .
So the population risk can be bounded by the empirical risk and Rademacher complexity of the function class H. Because we can’t minimize the population risk directly, we can minimize it indirectly by minimizing the empirical risk and Rademacher complexity of the function class H. Next, we will further discuss Rademacher complexity of the function class H. We first introduce several lemmas below.
Lemma 6. (Klebaner, 2005) For any given matrix U, the solution to the equation{ dx(t) = Ux(t)dt
x(0) = x̂
has the following expression x(t) = exp(Ut)x̂,
Also, we can write the solution to the following equation{ dy(t) = Uy(t)dt+ γy(t)dB(t)
y(0) = x̂
as
y(t) = exp(Ut− 1 2 γ2tI + γB(t)I)x̂.
Obviously, we have E[y(t)] = x(t). Lemma 7. A matrix C ∈ Rd×d is circulant, if there exists real number a1, · · · , ad such that
C =  a1 a2 . . . ad ad a1 . . . . . . . . . . . . . . . a2
a2 . . . ad a1
 .
For any circulant matrix, we have the following eigen-decomposition
ΨHCΨ = diag(λ1, . . . , λd),
where
√ dΨ =  1 1 . . . 1 1 m1 . . . md−1 . . . . . . . . . . . .
1 md−11 . . . m d−1 d−1  , and mis are the roots of unity and λi = a1 + a2mi + · · ·+ admd−1i .
C.2 THE PROOF OF THEOREM 3
Proof. By the definition of Rademacher complexity (Def. 2), we have
RSN (F) = (1/N)Eσ [ sup f∈F N∑ i=1 σiwx p i (T ) ] = (c/N)Eσ [ sup ‖U‖2≤c ‖ N∑ i=1 σix p i (T ) ‖2 ] .
Let ui = Ψx p i and denote the jth element of ui as ui,j . Then by lemma 7, we have
RSN (F) = (c/N)Eσ sup ‖U‖2≤c ∑ i,j σiσj〈xpi (T ),x p j (T )〉 1/2
= (c/N)Eσ sup |λi|≤c (‖ΨH N∑ i=1 σi ( ui,1 exp(λ1Tp), · · · , ui,d exp(λdTp))T ‖2 ) = (c/N)Eσ sup
|λi|≤c { d∑ j=1 [ N∑ i=1 σiui,j exp (λjTp) ]2 }1/2 = (c/N) exp(cTp)Eσ{ d∑ j=1 [ N∑ i=1 σiui,j ]2 }1/2
= (c/N) exp (cTp)Eσ‖ N∑ i=1 σiui‖2 = (c/N) exp (cTp)Eσ‖Ψ N∑ i=1 σix p i ‖2
= (c/N) exp (cTp)Eσ‖ N∑ i=1 σix p i ‖2
Note that E (wxpi (T )) = wE (x p i (T )) and according to Lemma 6, similar to proof for the function class F we have
RSN (G) = (c/N)Eσ sup ‖U‖2≤c ∑ i,j σiσj〈Expi (T ),Ex p j (T )〉 1/2
= (c/N)Eσ sup |λi|≤c (‖ΨH N∑ i=1 σi(ui,1 exp(λ1Tp− p(1− p)γ2T/2),
· · · , ui,d exp(λdTp− p(1− p)γ2T/2))T ‖2)
= (c/N)Eσ sup |λi|≤c { d∑ j=1 [ N∑ i=1 σiui,j exp ( λjTp− p(1− p)γ2T/2 )]2 }1/2 = (c/N) exp(cTp− p(1− p)γ2T/2)Eσ{ d∑ j=1 [ N∑ i=1 σiui,j ]2 }1/2 = (c/N) exp ( cTp− p(1− p)γ2T/2 ) Eσ‖
N∑ i=1 σiui‖2
= (c/N) exp ( cTp− p(1− p)γ2T/2 ) Eσ‖Ψ N∑ i=1 σix p i ‖2 = (c/N) exp ( cTp− p(1− p)γ2T/2 ) Eσ‖
N∑ i=1 σix p i ‖2 < RSN (F)
Therefore, we have completed the proof for the fact that the ensemble of Gaussian noise injected ResNets can reduce generalization error compared to the standard ResNet.

heading: D EXPERIMENTS ON STRATEGY II
text: D.1 FORWARD AND BACKWARD PROPAGATION USING (7)
Figure 8 plots the forward and backward propagation of the ICLR logo using the SDE model (7). Again, we cannot simply use the backward Euler-Maruyama discretization to reverse the features generated by the propagating through the forward Euler-Maruyama discretization.
D.2 EXPERIMENTS ON THE IDC DATASET
In this subsection, we consider the performance of the second residual perturbation in protecting membership privacy while retaining the classification accuracy on the IDC dataset. We use the same ResNet models as that used for the first residual perturbation. We list the results in Fig. 9, these results confirm that the residual perturbation (8) can effectively protect data privacy and maintain or even improve the classification accuracy. In addition, we depict the ROC curve for this experiment in Figure 12 (c). We note that, as the noise coefficient increases, the gap between training and testing accuracies narrows, which is consistent with Theorem 3.
D.2.1 RESIDUAL PERTURBATION VS. DIFFERENTIALLY PRIVATE STOCHASTIC GRADIENT DESCENT
We have shown that the first residual perturbation (6) outperforms the DPSGD in protecting membership privacy and improving classification accuracy. In this part, we further show that the second residual perturbation (8) also outperforms the benchmark DPSGD with the above settings. Table 2 lists the AUC of the attack model and training & test accuracy of the target model; we see that the second residual perturbation can also improve the classification accuracy and protecting better membership privacy.
D.3 EXPERIMENTS ON THE CIFAR10/CIFAR100 DATASETS
In this subsection, we will test the second residual perturbation (8) on the CIFAR10/CIFAR100 datasets with the same model using the same settings as before. Figure 10 plots the performance of En5ResNet8 on the CIFAR10 dataset. These results show that the ensemble of ResNets with residual
perturbation (8) is significantly more robust to the membership inference attack. For instance, the AUC of the attack model for ResNet8 and En5ResNet8 (γ = 2.0) is 0.757 and 0.526, respectively. Also, the classification accuracy of En5ResNet8 (γ = 2.0) is higher than that of ResNet8, and their accuracy is 71.2% and 66.9% for CIFAR10 classification. Figure 11 shows the results of En5ResNet8 for CIFAR100 classification. These results confirm that residual perturbation (8) can protect membership privacy and improve classification accuracy once more.
D.4 ROC CURVES FOR THE EXPERIMENTS ON DIFFERENT DATASETS
Figure 12 plots the ROC curves for the experiments on the different datasets with different models using the second residual perturbation strategy. These ROC curves again show that if γ is sufficiently large, the attack model’s prediction will be nearly a random guess.

heading: E MORE EXPERIMENTAL DETAILS
text: We give the detailed construction of the velocity field F (x(t),W(t)) in (3) and (7) that used to generate Figs. 1 and 8 in Algorithm 1.
Algorithm 1 The expression of F (x(t),W(t)) Input: image=x(t); rows, cols, channels = image.shape. Output: F (x(t),W(t))=dirtyimage. for k in range(channels) do
for i in range(rows): do for j in range(cols): do
offset(j) =int([j + 50.0cos(2πi/180)]%cols); offset(i) =int([i+ 50.0sin(2πi/180)]%row); dirtyimage[i, j, k] = image[(i+ offset(j))%rows, (j + offset(i))%cols, k];
return dirtyimage

heading: F ARCHITECTURES OF THE USED DNS
text: Figure 13 shows the architectures of ResNets used in this paper, and we plot basic blocks in Fig. 14.

Review summaries:
The paper focuses on the topic of differentially private deep learning. Specifically, based on the deep residual learning, they first see it as an ODE. Then, to reduce the reversibility of the ODE, they modify the model as an SDE. By discretizing the SDE, they get a perturbed version of residual learning and use this to design DP-algorithms. The first strategy is directly followed the SDE while the second strategy is with an addition multiplicative noise of the additive noise. Finally, they show that their methods to defend membership inference attack both theoretically and practically. 

The paper presents a method for training ResNets with differential privacy. Rather than the usual methods based on noisy gradient descent, the authors propose adding noise at each layer of the network during both training and testing. The authors prove differential privacy guarantees for two strategies of this type (one with additive and one with multiplicative noise). They also show some evidence that the noise can help generalization, by showing that the Rademacher complexity of a continuous linearized version of the model is lower when noise is added.

This paper studies an important problem and proposes the novel residual perturbation to protect privacy while maintaining the ResNet models’ utility.  Two SDE models are provided to inject noises with abundant theoretical proof are provided. Experimental results demonstrate the performance of privacy protection and classification accuracy on benchmark datasets. My major concern is about the utility enhancement and the DP guarantee (see cons below). Hope the authors can address my concern in the rebuttal period.