RelaxLoss: Defending Membership Inference Attacks without Losing Utility
Authors: Dingfan Chen
Ning Yu
Mario Fritz
Venue: ICLR 2022 Spotlight (2022)
Keywords: membership inference attack
defense

Abstract: As a long-term threat to the privacy of training data, membership inference attacks (MIAs) emerge ubiquitously in machine learning models.
Existing works evidence strong connection between the distinguishability of the training and testing loss distributions and the model's vulnerability to MIAs. Motivated by existing results, we propose a novel training framework based on a relaxed loss ($\textbf{RelaxLoss}$) with a more achievable learning target, which leads to narrowed generalization gap and reduced privacy leakage. RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead. Through extensive evaluations on five datasets with diverse modalities (images, medical data, transaction records), our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs as well as model utility. Our defense is the first that can withstand a wide range of attacks while preserving (or even improving) the target model's utility.

Published as a conference paper at ICLR 2022
RELAXLOSS: DEFENDING MEMBERSHIP INFERENCE ATTACKS WITHOUT LOSING UTILITY
Dingfan Chen' Ning Yu Mario Fritz'
'CISPA Helmholtz Center for Information Security SSlesforce Research
University of Maryland 4Max Planck Institute for Informatics (dingfan. chen, fritz) @cispa.depppp..
ning. yussallesoorreccomm
ABSTRACT
As a long-term threat to the privacy of training data, membership inference attacks (MIAs) emerge ubiquitously in machine learning models. Existing works evidence strong connection between the distinguishabilityy of the training and testing loss distributions and the model's vulnerability to MIAs. Motivated by existing results, we propose novel training framework based on relaxed loss (RelaxLoss) with a more achievable learning target, which leads to narrowed generalization gap and reduced privacy leakage. RelaxLoss is applicable any classification model with added benefits of easy implementation and negligible overhead. Through extensive evaluations on five datasets with diverse modalities (images, medical data, transaction records), our approach consistently outperforms state-of-the- art defense mechanisms in terms of resilience against MIAs as well as model utility. Our defense is the first that can withstand wide range of attacks while at preserving (or even improving) the target model's 's utility. Source code available
INTRODUCTION
While deep learning (DL) models have achieved tremendous success in the past few years, their deployments in many sensitive domains (e.g., medical, financial) bring privacy concerns since data misuse in these domains induces severe privacy risks to individuals. In particular, modern deep neural networks (NN) are prone memorize training data due their high capacity, making them vulnerable to privacy attacks that extract detailed information about the individuals from models (Shokri et al., 2017; Song et al., 2017; Yeom al., 2018)
In membership inference attack (MIA), an adversary attempts to identify whether specific data sample was used to train a target victim model. This threat is pervasive in various data domains (e.g., images, medical data, transaction records) and inevitably poses serious privacy threats to individuals (Shokri et al., 2017; Nasr al., 2018; Salem et al., 2019), even given only black-box access (query inputs in. posterior predictions out) (Shokri et al., 2017; Salem et al., 2019; Song & Mittal, 2020) or partially observed output predictions (e.g., top-k predicted labels) (Choo et al., 2020).
Significant advances have been achieved to defend against MIAs. Conventionally, regularization methods designed for mitigating overfitting such as dropout (Srivastava et al., 2014) and weight- decay (Geman et al., 1992) regarded defense mechanisms (Salem al., 2019; Jia et al., 2019; Shokri et al., 2017). However, as conveyed by Kaya et al. (2020); Kaya & Dumitras (2021), vanilla regularization techniques (which are not designed for MTA), despite slight improvement towards reducing the generalization gap, are generally unable to eliminate MIA. In contrast, recent works design defenses tailored MIA. common strategy among such defenses is adversarial training (Goodfellow et al., 2014b;a), where a surrogate attack model (represented as NN) is used to approximate the real attack and subsequently the target model is modified to maximize prediction errors of the surrogate attacker via adversarial training. This strategy contributes to remarkable success defending NN-based attacks (Nast et al., 2018; Jia al., 2019). However, these methods are greatly
"This work was done when Ning Yu was in joint Ph.D. program with the University of Maryland and Max Planck Institute for Informaties.

---

Published as a conference paper at ICLR 2022
restricted by strong assumptions on attack models, thereby failing to generalize to novel attacks unanticipated by the defender (e.g., a simple metric-based attack) (Song & Mittal, 2020). In order to defend attacks beyond the surrogate one, differentially private (DP) training techniques (Abadi et al., 2016; Papernot et al., 2016; 66 2018) 8) that provide strict guarantees against MIA are exploited. Nevertheless, evidenced by Rahman et al. (2018); Jia et al. (2019); Hayes et al. (2019); Jayaraman & Evans (2019); Chen et al. (2020); Kaya & Dumitras (2021), incorporating DP constraints inevitably compromises model utility and increases computation cost.
In this paper, we present an effective defense against MIAs while avoiding negative impacts on the defender's model utility. Our approach is built on two main insights: (i) the optimal attack only depends on the sample loss under mild assumptions of the model parameters (Sablayrolles et al., 2019); (ii) a large difference between the training loss and the testing loss provably causes high membership privacy risks (Yeom et al., 201 8). By intentionally 'relaxing' the target training loss to a level which is more achievable for the test loss, our approach narrows the loss gap and reduces the distinguishabiliyy between the training and testing loss distributions, effectively preventing various types of attacks in practice. Moreover, our approach allows for a tulesty-preseriing (or even improving) defense, greatly improving upon previous results. As a practical benefit, our approach is easy to implement and can be integrated into any classification models with minimal overhead.
Contributions. (i) We propose RelaxLoss, simple yet effective defense mechanism to strengthen target model's resilience against MIAs without degrading its utility. To the best of our knowledge, our approach for the first time addresses wide range of attacks while preserving (or even improving) the model utilliy. (ii) We derive our method from a Bayesian optimal attacker and provide both empirical and analytical evidence supporting the main principles of our approach. (iii) Extensive evaluations on five datasets with diverse modalities demonstrate that our method outperforms state-of-the-an approaches by large margin in membership inference protection and privacy-utility trade-off.
RELATED WORK
Membership Inference Attack. Inferring membership information from deep NNs has been investigated in various application scenarios, ranging from the white-box setting where the whole target model is released (Nasr et al., 2019; Rezaei & Liu, 2020) to the black-box setting where the complete/partial output predictions are accessible to the adversary (Shokri et al., 2017: Salem et al., 2019; Yeom et al., 2018; Sablayrolles et al., 2019; Song & Mittal, 2020; Choo et al., 2020; Hui et al., 2021; Truex et al., 2019). An adversary first determines the most informative features (depending on the application scenarios) that faithfully reflect the sample membership (e.g., logits/posterior predictions (Shokri et al., 2017; Salem et al., 2019; Jia et al., 2019), loss values (Yeom et al., 2018; Sablayrolles et al., 2019), and gradient norms (Nast et al., 2019; Rezaci & Liu, 2020)), and subsequently extracts common patterns in these features among the training samples for identifying membership. In this work, work towards an effective defense by suppressing the common patterns that an optimal attack relies on.
Defense. Existing defense mechanisms against MIA are mainly divided into three main categories: (i) regularization techniques to alleviate model overfitting, (ii) adversarial training to confuse surro- gate attackers, and (iii) differentially private mechanism offering rigorous privacy guarantees. Our proposed approach can be regarded as a regularization technique owing to its effect in reducing gen- eralization gap. Unlike previous regularization techniques, our method is explicitly tailored towards defending MIAs by reducing the information that an attacker can exploit, leading to significantly better defense effectiveness. Algorithmically, our approach shares similarity with techniques that suppress the target model's confidence score predictions (e.g., label-smoothing (Guo et al., 2017; Müller et al., 2019) and confidence-penall (Pereyra et al., 2017)), but ours is fundamentally different in the sense that we modulate the loss distribution with gradient ascent.
Previous state-of-the-an defense mechanisms against MIA, such as Memguard (Jia et al., 2019) and Adversarial Regularization (Nasr et al., 2018), are built on top of the idea of adversarial train- ing (Goodfellow et al., 201 2014b;a). Such approaches usually rely on strong assumptions about attack models, making their effectiveness highly dependent on the similarity between the surrogate and the real attacker (Song& & Mittal, 2020). In contrast, our method does not rely on any assumptions about the attack model, and has shown consistent effectiveness across different attacker types.
2

---

Published as a conference paper at ICLR 2022
Differential privacy (Dwork, 2008; Dwork et al., 2014; Abadi al., 2016; Paperot et al., 2016) provides strict worst-case guarantees against arbitrarily powerful attackers that exceed practical limits, but inevitably sacrifices model utility (Rahman et al., 2018; Jia et al., 2019; Hayes et al., 2019; Chen et al., 2020; Kaya & Dumitras, 2021; Jayaraman & Evans, 2019) and meanwhile increases computation burden (Goodfellow, 2015; Dangel et al., 2019). In contrast, we focus on practically realizable attacks for utility-preserving and computationally efficient defense.
3 PRELIMINARIES
Notations. We denote by z; = (ax, 3/) one data sample, where x; and y; are the feature and the one-hot label vector, respectively. f(:; O) represents a classification model parametrized by 0, and p = fmx 0) E [0, 1 denotes the predicted posterior scores (after the final softmax layer) where C denotes the number of classes. denotes the indicator function, i.e., I [p] equals 1 if the predicate is true, else 0. We use subscripts for sample index and superscripts for class index.
Attacker's Assumptions, We consider the standard setting of MIA: the attacker has access to query set S = {(z,, m, containing both member (training) and non- member (testing) samples drawn from the same data distribution Pdata. where m, is the membership attribute (m; = 1 if z; is member). The task is to infer the value of the membership attribute m, associated with each query sample 2,. We design defense for a general attack with full access to the target model. The attack A(z., f(; 0)) is a binary classifier which predicts m, for a given query sample x; and a target model parametrized by 0. The Bayes optimal attack Aopa (Zz, [(; 0)) will output 1 if the query sample is more likely to be contained in the training set, based on the real underlying membership probability P(m; 2; f), which is usually formulated as non-negative log ratio:
P(m: 1| 0)
Aopt (z,, f(: 11
log P(m: = 0 z., 0)
(1)
Defender's Assumptions. We closely mimic an assumption-frn scenario in designing our defense method. In particular, we consider a knowledge- limite---iiidddddee defender which: (i) does not have access to additional public (unlabelled) training data (in contrast to Papernot et al. (2016; 2018)); and (ii) lacks prior knowledge of the attack strategy (in contrast to Jia et al. (2019); Nasr et al. (2018)), For added rigor, we also study attacker's countermeasures our defense in Section 6.4.
4 RELAXLOSS
The ultimate goal of the defender is two-fold: (i) privacy: reducing distinguishability of member and non-member samples; (ii) utility: avoiding the sacrifice of the target model's 's performance. We hereby introduce each component of our method targeting at privacy (Section 4.1) and utility (Section 4.2).
4.1 PRIVACY: REDUCE DISTINGUISHABILITT VIA RELAXED TARGET LOss
We begin by exploiting the dependence of attack success rate on the sample loss (Yeom et al., 2018; Sablayrolles et al., 2019). In particular, a large gap in the expected loss values the member and non-mene membeeeer data, i.e., E|E]uon E[fmmem is proved to be sufficient for conducting attacks (Yeom et al., 2018). Along this line, Sablayrolles et al. (2019) further show that the Bayes optimal attack only depends on the sample loss under a mild posterior assumption of the model parameter: P(0)zz,... 2n) x e + MN =.)) Formally,
(2) where 7 denotes a threshold function 2 Intuitively, Equation 2 shows that is likely to be used for training if the target model exhibits small loss value on it. These results motivate our approach to mitigate MIAs by reducing distinguishablityy between the member and non-member loss distributions.
Aope f(: 0)) 1[-(((,, [(2))]
1 This corresponds to Bayesian perspective, i.e., 0 is regarded as a random variable that minimizes the empirical risk Σ m, - {(0, 21). T the temperature that captures the stochasticity.
2 We summarize both the strategy MALT and MAST from Sablayrolles al. (2019) in Equation 2, where is constant function for MALTT
3

---

Published as a conference paper at ICLR 2022
Relaxing Loss Target with Gradient Ascent.
Algorithm 1: RelaxLoss
Directly operating on member and non-member loss distributions, however, is impractical, since the exact distributions are intractable and a large amount of additional hold-out samples are re- quired for estimating the distribution of non- member data. In order to bypass these issues and reduce the distinguishability between the member and non- member loss distributions, we propose to simplify the problem by considering the mean of the loss distributions, and subse- quently set a more achievable mean value for the target loss, where the loss target is relaxed to a level that is casier to be achieved for the non-member data.
Input: Dataset {(x; )= training epochs
E, learning rates т, batch size B, number of output classes C, target loss
value a
Output: Model f-; 0) with parameters 0 Initialize model parameter 0: for epoch in {1...., }} do
for batch_index in (1.... K} do
Get sample batch { 3/: )
Perform forward pass: p; = f( 0)
Compute cross entropy loss (()) on the batch
if (() 2 a then
// gradient descent
Algorithmically, instead of pursuing zero train- ing loss of the target model, we relax the target mean loss value to be larger than zero and ap- ply a gradient ascent step as long as the average loss of the current batch is smaller than a.
0 0- 0 T VL(
else
if epoch %: = 0 then
// gradient ascent
0 0 100++VVVC VC
else
// posterior
4.2 UTILITY:
APPLY POSTERIOR FLATTENING
flattening
Construct softlabel t; with
AND NORMAL GRADIENT OPERATIONS
p 4 PC)
of =1
With the relaxed target loss, the predicted pos- terior score of the ground-truth class pat is no longer maximized towards 1. If the probabil- ity mass of all non- con-gruund-trthh classes - pa concentrates on only few of them (e.g.. hard samples that are close to the decision boundary between two classes), it is very likely that one non-ground- -ruth class has score larger than 2o² (i.e., maXc,e p > pot), thus leading to incorrect predictions. To address this issue, we propose to encourage a large margin between the prediction score of the ground-truth-clan and the others by flattening the target posterior scores for con-ground-trute classes. Specifically, we dynamically construct softlabels during each epoch by: (i) retaining the score of the ground- truth class, i.e., the current predicted value pot, and (ii) re-allocating the remaining probability mass evenly to all non-ground-trutt classes.
otherwise
Compute cross entropy loss with the softlabel: ")
{(0, 2,) =
- (()) = せ D e(0 2.)
c=1 sg [ log
Update model parameters: 0 < 0 100--VV(( (V)
end end end return model 
end
sg stands for the stopgradient operator that is de- fined as identity at forward pass and has zero partial derivatives, i.e., 4; is non-updated constant.
In summary, we run a repetitive training strategy to balance privacy and utility, which consists of two steps: (i) if the model is not well- vell-trained, i.e., the current loss is larger than the target mean value a, we run a normal gradient descent step; (ii) otherwise, we apply gradient ascent or the posterior flattening step (See Algorithm 1).
5 ANALYTICAL INSIGHTS
In this section, we analyze the key properties that explain the effectiveness of RelaxLoss. We provide both analytical and empirical evidence showing that RelaxLoss can (i) reduce the generalization gap, and (ii) increase the variance of the training loss distribution, both contributing to mitigating MIAs.
RelaxLoss reduce the generalization gap. We apply RelaxLoss to CIFAR-10 dataset and plot the resulting loss histograms in Figure 1. With a more achievable learning target, RelaxLoss blurs the
4

---

Published as a conference paper at ICLR 2022
1.0
non-member
ron-menter ricmbrr
membb
nomelaee frepuercy
no fropprny
0.5
00
02
rone Red
0.4
Via ue
7. 0g
[03
0.15
Via
0.34
2
01
0.
100
135
211
25
3.1
355
40
0M
00.
3.0
2.0
23
355
(a) Vanilla
(b) Ours
0.5)
Ours (a 1.0)
Figure 1: Loss histograms on CIFAR-10 10 with ResNet20 architecture when applying (a) vanilla training, (b) our method with 0.5, and (c) our method with a 1.0. The empirical mean and variance of the loss distributions are shown in the figure. The AUC of loss thresholding attack equals to 0.84 in (a), 0.67 in (b), and 0.57 (c). We observe that our method fits the target mean (Section 4), increases the variance of the training loss distribution and reduces the distinguishability between member and non-member distributions (Section 5).
B(f) &
member
non member
large AT
(a) Vanilla gradient descent
member non member
large-IAr) |A/
A+
Gradient ascent
Figure 2: Comparison between vanilla gradient descent and the gradient ascent step in RelaxLoss (demonstrated in 2D). The loss contour lines plotted in the figure, the bottom part of which corresponds to a low loss region. The target loss level a is visualized in the figure. Training with vanilla gradient descent step results in near zero loss for member samples, and large loss values for non-member samples. In contrast, large loss value tends to trigger large update | during the gradient ascent step. As result, RelaxLoss spreads out the training loss distribution and blurs the gap between the distributions (Section 5).
gap between the member and non- nom-member loss distributions (Figure 1), which naturally leads to narrowed generalization gap (Appendix Figure 7) and reduced privacy leakage (Yeom et al., 2018).
RelaxLoss increases the variance of the training loss distribution. We observe that RelaxLoss spreads out the training loss distribution (i.e., increase the variance) due to its gradient ascent step (Appendix I) large loss samples tend to have a more significant increase in its loss value during the gradient ascent step (See Figure 2 for demonstration). In contrast, except DP-SGD, existing defense methods do not have this property (Appendix C.10). Intuition suggests that the increase of training loss variance suppresses the common pattern among training losses and reduces the information that can be exploited by an attacker, thus contributing the protection against attacks. To verify the association between the variance increasing effect and the defense effectivvness, we conduct experiments on CIFAR-10 10 dataset and measure the Pearson's 's correlation coefficients between the training loss variance and the attack AUC (Appendix Figure 6). With overall score of -0.85 (-0.77 and -0.94 for black-bol and white-box attacks, respectively), we conclude a fairly strong negative relationship. When considering typical Gaussian assumption of the loss distributions (Yeom et al., 2018; Li et al., 2020). we further show this variance increasing property helps to lower an upper bound of the attack AUC, and provide formal analysis in Appendix A.2.
5

---

Published as a conference paper at ICLR 2022
6 EXPERIMENTS
In this section, we rigorously evaluate the effectiveness of our defense across a wide range of datasets with diverse modalities, various strong attack models, and eight defense baselines representing the previous state of the art.
6.1 EXPERIMENTAL SETUP
Settings. We set up seven target models, trained on five datasets (CIFAR-IO, CIFAR-100, CH- MNIST, Texas Purchase 100) with diverse modalities (natural and medical images, medical records, and shopping histories). For image datasets, we adopt 20-layer ResNet (He et al., 2016) and 1 ll-layer VGG (Simonyan & Zisserman, 2015); and for non-image datasets, we adopt MLPs with the same architecture and same training protocol as in previous works (Nasr et al., 2018; Jia et al., 2019). We evenly split each dataset into five folds and use each fold as the training/testing set for the target/shadow model", and use the last fold for training the surrogate attack model (for Jia al. (2019); Shokri et al. (2017)). We fix the random seed and training setting for fair comparison among different defense methods. See Appendix B for implementation details.
Attack methods. To broadly handle attack methods in our defense evaluation, consider attacks in a variety of application scenarios (black-boo and white-box) and strategies. We consider the following state-of-the-arr attacks from two categories:(i) White-box attacks: Both Nasr al. (2019); Rezaei & Liu (2020) are based on gradient norm thresholding. We denote these attacks by the type of gradient followed by its norm. Grad-x and Grad-w stand for the gradient w.r.t. the input and the model parameters, respectively; (ii) Black-box attacks: Salem et al. (2019) (denoted as NN, standing for the proposed neural network attack model). We adopt the implementation provided by Jia et al. (2019) and use the complete logits prediction as input to the attacker. Sablayrolles et al. (2019) (denoted as Loss for their loss thresholding method). We use a general threshold independent the query sample, as the adaptive thresholding version is more expensive computational with no only marginal improvements. Song & Mittal (2020) (denoted as Entropy and M-Entropy for their proposed attack by thresholding the prediction entropy and modified version, respectively.). We exclude attacks that only use partial output predictions (c.g., top-1 predicted label) from our evaluation as they are strictly weaker than the attacks we include above (Choo al., 2020).
Evaluation metrics. We evaluate along two fronts: utility (measured by test accuracy of the victim model) and privacy. For privacy, in line with previous works (Song & Mittal, 2020; Jia al., 2019; Sablayrolles et al., 2019; Shokri et al., 2017; Nasr et al., 2018; Salem et al., 2019), we consider the following two metrics: (i) attack accuracy: We evaluate the attack accuracy
balanced query set, where a random guessing baseline corresponds to 50% accuracy. For threshold based attack methods, following Song & Mittal (2020), we select the threshold value be the one with the best attack accuracy on the shadow model and shadow dataset; (ii) attack AUC: The area under the receiver operating characteristic curve (AUC), corresponding to an integral over all possible threshold values, represents the degree of separability. A perfect defense mechanism corresponds to AUC=0.5.
6.2 COMPARISON TO BASELINES
Defense Baselines. We consider two state-of-the-ard defense methods: Memguard (Jia et al., 2019) and Adversarial regularization (Adv-Reg) (Nasr et al., 2018). Additionally, we compare to five regu- larization methods: Early-stopping, Dropout (Srivastava et al., 2014), Label-smoothing (Guo et al., 2017; Müller et al., 2019). Confidence-penalty (Pereyra et al., 2017) and Desti-)Dittllltiinn (Hinton et al., 2015; Zhang et al., 2019). Moreover, we compare to differential private mechanism, i.e., Differ- entially private stochastic gradient descent (DP-SGD) (Abadi al., 2016)². We exclude defenses that additionally require public (unlabelled) data (Papernot et al., 2016; 2018) for training the target model from our evaluation.
3 Shadow models used for training the attack models in the NN-hased attack and selecting the optimal threshold for all metric-based attacks.
4 In line with previous work (Choo al., 2020), we adopt small noise scale (<0.5) for maintaining target model's utility at decent level, which leads meaninglessly large e values.

---

Published as a conference paper at ICLR 2022
13
03
03
03
43
43
13
87
35
E4
AUC
83
07
1.0
05
AZaaLAUC
08
09
06 07
ABC
87 02
A- Car
Confersoperaa dy
29-500
Libiinochingg
Dopad
Earp-tspaang
Desiltiinn
Ad-Pa
Mergued
50 5000
(a) CIFAR-10
43
Crachen
26
16
15
1.0
Astick
Prace AIX
Anach
Artack
Arso
Gurfermeree dy
3P-5GD
Libhowochig
Orgas
- Eape-
Stallation
Morguad
(b) CIFAR-100 100
Figure 3: Comparisons of all defense mechanisms on CIFAR-10 and CIFAT-100 dataset with ResNet20 architecture. Each subplot corresponds one attack method. The x-axis corresponds to the attack AUC (the lower the better) while the y-axis is the target model's 's test accuracy (the higher the better). For visualization purposes, we plot the ideal defense on the top-left corner, whose X-coordinate equals 0.5 and y-coordinate is set to be the highest test accuracy among all models.
Privacy-utility trade-off. We vary the hyperparameters that best describe the privacy-utiiity trade-off of each method across their effective ranges (See Appendix B for details) and plot the corresponding privacy-utility curves. We set the attack AUC value (privacy) as x-axis and the target model's s performance (utility) as the y-axis. A better defense exhibits privacy-utility curve approaching the top-left corner, i.e., high utility and low privacy risk. As shown in Figure 3(a) and 3(b) (and Appendix Figure 9-13), we observe that our method improves the privacy- -utiliyy trade-off over baselines for almost all cases: (i) Previous state-of-the
defenses (Memguard and Adv-Reg) are effective for the NN attack, but can hardly generalize to the other types of attack, which is also verified in Song & Mittal (2020). Moreover,
test-time defense, Memguard is not applicable to white-box attacks. In contrast, our method is consistently effective irrespective of the attack model and applicable to all types of attacks. (ii) In comparison with other regularization methods, our method showcases significantly better defense effectiveness. Specifically, when compared with Farly-stopping, the generally most effective regularization-based defense baseline, our approach decreases the attack AUC (i.c., relative percentage change) by up to 26% on CIFAR-10 10 and 46% on CIFAR- 100 for same level of utility. (iii) DP-SGD is generally the most effective defense baseline, despite the meaningless large values, which is consistent with Choo et al. (2020). In comparison with DP-SGD, our method improves the target model's test accuracy by around 16% on CIFAR-10 and 12% on CIFAR- 100 (relative percentage increase) across different privacy levels. (iv) (See detailed results in Appendix C.9) Our approach the only one that exhibit consistent defense effectiveness across various data modalities and model architectures, while the best baseline methods can only show effectiveness for at most one data modality (c.g., DP-SGD for images, and aabol-smoothing for transaction records).
6.3 RELAXLOSS Vs. ATTACKS
As can be seen from the privacy-utility curves in Figure and Appendix C.9, our approach is the only one that can consistently defend various MIAS without sacrificing the model utility. We then evaluate to what extend the attacks can be defended without loss in the model utility. To this end, we select corresponds to the model with the lowest privacy risk (lowest attack AUC averaged over all attacks), under the constraint that the defended model achieve a top-1 test accuracy not worse than the undefended model.
Utility. Table summarizes the test accuracy of target models defended with our method. Compared with vanilla training, our method achieves consistent improvement (up to 7.46%) in terms of utility
7

---

Published as a conference paper at ICLR 2022
(a)
(b)
Dutaset
CIFAR-IO
Nacis
12000
(ResNec20)
CIFARIO
(VGGL))
CIEARIO
(ResNer20)
CIFAR1O0
CIPARIOO (VGGLL)
CH-MNNST
Texas100
Purchase 100
CIFAR-100
12000
Top-1
top-3
tog-1
Top-5
top-1
tog-3
top-1
top-3
1og-1
Top-3
top-1
1og-3
Top-1
top-5
CH-MNIST
1900
wa defense
70.5
96.6
73.8
97.0
33.2
63.0
414
67.5
77.1
99.6
52.3
82.6
89.1
99.8
Texas100
with defenise
73.8
98.2
74.4
998
35.1
67.7
41.4
69.9
18.4
1927
85.3
86.8
89.1
19.6
Purchase100 100
39465
4.68
1.00
0.81
0.82
5.72
7.46
0.00
3.56
1.59
0.10
5.74
5.08
0.00
-0.20
Table 1: (a) Size of the target model's training set. (b) Target model's test accuracy (in %) with and without (wo) applying our defense. The relative difference (A) is in % and the increase is highlighted
in green and decrease in red See Appendix C.3 for more details.
Eneopy
W-Earngg
deforse
207
Grte 99
case yor
Teesbo
craau Rg
cosar-ee aot (
Prokknsto
OPi ng
Or Orumery
Tosasbo
Dast son
cose pet por
Ca cr-seerr
Towsho
com
co
(a) Black-box attacks
Grzbic
Geadie
defere
2.02
2,7
45
1G
45
GPM 1000
CONA sae ()
or ar-eear
cone aat my
Foedome tte
cosst zer on)
cost ses on
The Teaatio
Foctbow op
cown use
cree un
cool uee
(b) White-box attacks
Figure 4: Attack AUC on target models trained with and without (wo) applying our defense method. Each subplot is titled with the corresponding attack method's name. (R) and (V) denotes ResNet and VGG network, respectively. The corresponding target model's 's utility is shown in Table 1.
across different datasets and model architectures, albeit a 0.2% accuracy drop for saturated top-5 accuracy (99.6% compared to 99.8%).
Privacy. Figure 4 shows the membership privacy risk (AUC) of the target models in Table We observe that our method is consistently effective for all types of attacks, datasets, and model architectures. In particular, our method consistently reduces the attack AUC: (i) to <0.6 for all non-image datasets; (fi) from 0.7 to 0.55 for CH-MNIST; (iii) from >0.8 to 0.55 for CIFAR-10 (R) and from >0.9 to <0.6 for CIFAR- (R). We also include the attack accuracy values in Appendix Table 5, which shows method reduces most attacks to random- guuesiingg level. We thus conclude that our method improves both target models' utility as well as their resilience against MIAs.
6.4 ADAPTIVE ATTACK
We further analyze the robustness of our method against attack's 's countermeasures. Namely, we consider the situations where attackers have full knowledge about our defense mechanism and the selected hyperparameters, and have tailored the attacks to our defense method. We simulate the adaptive attacks by: (i) training shadow models with the same configuration used for training our defended target models in Table 1, (ii) simulating the adaptive attacks using the calibrated shadow models. We report the highest attack accuracy (i.e., worst-case privacy risk) among different adaptive attacks in Table 2 (See Appendix C.4 for details). We observe that despite being less effective in defending against adaptive attacks than non -da-aive attacks, our method still greatly decreases the highest adaptive attacker's accuracy by 13.6%.37.cm 7.6% compared to vanilla training.
8

---

Published as a conference paper at ICLR 2022
CIFARIO (ResNer20)
CIFARIO (VGGIT)
CI
CIFARI00 (VGGIT)
(ResNet20)
CH-MNIST
Texas100
Purchuse
w/o defense
87.3
80.7
92.6
97.5
67.1
79.0
65.7
w/ defense (non-adaptive)
50.0
50.0
50.0
50.0
50.7
50.0
50.1
2 (nonn-ddiptiv iidiiiiei
-42.7
-38.0
-46.0
-48.7
-243
36.7
223,9
w/ defense (adaptive)
56.0
68.2
37.8
84.2
56.6
53.8
56.0
(adaptive)
-35.9
-15.5
-37.6
-13.6
-15.6
-31.9
-14.8
Table 2: The highest attack accuracy (in %) among different adaptive attacks (and the corresponding non-adaptive attack accuracy is shown for reference) evaluated on the target model with (w/) or without (wo) defense. A corresponds to the relative difference (in %) in attack accuracy when applying our defense compared to vanilla training. The used target models are the same as in Table
6.5 ABLATION STUDY
We study the impact each component our approach and plot the results in Figure 5. We observe that while applying posterior flattening alone (without gradient ascent) has limited effects, using together with gradient ascent indeed improves the model's 's test accuracy across a wide range of attack AUC, which validates the necessity of all components of our method.
M
X
10
Oc.
36
Figure Ablation study on CIFAR-100 with ResNet architecture. We validate the necessity of our gradient ascent (Section 4.1) and posterior flattening step (Section 4.2)
DISCUSSION
Properties of RelaxLoss. RelaxLoss enjoys several properties which explain its superiority over existing defense methods. In particular, we provide empirical and analytical evidence showing that in contrast to most existing methods (Appendix C.10), RelaxLoss reduces the generalization gap and spreads out the training loss distributions (Section 5), thereby effectively defeating MIAs. Moreover, we observe that RelaxLoss soften the decision boundaries (Appendix C.11), which contributes to improving model generalization (Zhang et al., 2017: Pereyra al., 2017).
Practicality. We consider the practicality of our method from the following aspects: (i) Hyperpa- rameter tuning: Our method involves a single hyperparameter a that controls the trade-off between privacy and utility. A fine-grained grid search utility trade-off with varying value of a, and subsequently selecting the a corresponding to the desired privacy/utility level) allows precise control (ii) Computation cost: Our method incurs negligible additional computation cost when compared with backpropagation in vanilla training (Appendix C.7). In contrast, baseline methods generally suffer from larger computation burden. For instance, Memguard slows down the inference due to its test-time optimization step, while the training speed DP-SGD and Adv-Reg is greatly hindered by per-sample gradient computation (Goodfellow, 2015; Dangel et al., 2019) and adversarial update step, respectively.
validation (i.e., first estimating the privacy-
the expected privacy/utility level of the target model.
8 CONCLUSION
In this paper, we present RelaxLoss, novel training scheme that is highly effective in protecting against privacy attacks while improving the utility of target models. Our primary insight is that membership privacy risks can be reduced by narrowing the gap between the loss distributions. We validate the effectiveness of our method on a wide range of datasets and models, and evidence its superiority when compared with eight defense baselines which represent previous state of the art. As RelaxLoss exhibits superior protection and performance and easy to be implemented in various machine learning models, expect it to be highly practical and widely used.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work is partially funded by the Helmholtz Association within the projects "Trustworthy Federated Data Analytics (TFDA)" (ZT-I-001 4) and "Protecting Genetic Data with Synthetic Cohorts from Deep Generative Models (PRO-GENE-GGEN" (ZT-I-PF-5-23). Ning Yu was partially supported by Twitch Research Fellowship. We acknowledge Max Planck Institute for Informatics for providing computing resources.
REPRODUCIBILITY STATEMENT
The authors strive to make this work reproducible. The appendix contains plenty of implementation details. The source code and models are available GitHub.
ETHICS STATEMENT
This work does not involve any human subject, nor is our dataset related to any privacy concerns. The authors strictly comply with the ICLR Code of Ethics".
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security (CCS). 2016.
Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 343-362, 2020.
Christopher Choquette Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. arXiv preprint arXiv:2007.14321, 2020.
Felix Dangel, Frederik Kunstner, and Philipp Hennig. Backpack: Packing more into backprop. arXiv preprint arXiv:1912.10990 2019.
Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation. Springer, 2008
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4), 2014.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4(1), 1992.
Ian Goodfellow. Efficient per-example gradient computations. arXiv preprint 2015.
Ian Goodfellow, Jean Pouget-Abadics Mchdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 1406.266,, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 2014b.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference Machine Learning (ICML). PMLR, 2017.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019.
https://ietr.cc/pibllccc
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531 2015.
Bo Hui, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. Practical blind membership inference attack via differential comparisons. arXiv preprint arXiv:2101.01341, 2021.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th {USENIX] Security Symposium ({ (USSENIX) Security 19), 2019.
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong, Memguard: De- fending against black-box membership inference attacks via adversarial examples. Proceedings of the 20/9 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2019.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning (ICML.). PMLR, 2015.
Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific reports, 6(1), 2016.
Yigitcan Kaya and Tudor Dumitras. When does data augmentation help with membership inference attacks? In International Conference on Machine Learning. PMLR, 2021.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. On the effectiveness of regularization against membership inference attacks. arXiv preprint arXiv:2006. 05336, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning, arXiv preprint arXiv:2002.07394, 2020.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. Advances in neural information processing systems (NeurIPS), 2018
Zinan Lin, Vyas Sekar, and Giulia Fanti. On the privacy properties of gan- gennggeneeted samples. In International Conference Artificial Intelligence and Statistics. PMLR, 2021.
Rafael Müller, Simon Korbblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv:1906.02629, 2019.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2018
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In IEEE symposium security and privacy (SP). IEEE, 2019,
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755. 2016.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with pate. arXiv preprint /80i 08908, 2018.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.000 065448 2017.
11

---

Published as a conference paper at ICLR 2022
Md Atiqur Rahman, Tanzila Rahman, Robert Laganière, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Trans. Data Priv., 11(1), 2018.
Shahbaz Rezaci and Xin Liu. Towards the infeasibility of membership inference on deep models. arXiv preprini arXiv:2005. 1000005.37700 13702, 2020.
Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier, Cordelia Schmid, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. International Conference on Machine Learning (ICML). PMLR, 2019.
Ahmed Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In Annual Network and Distributed System Security Symposium (NDSS), 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In JEEE Symposium on Security and Privacy (SP). IEEE, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations (ICLR), 2015.
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security (CCS), 2017.
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. arXiv preprint arXiv:2003. 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research (JMLR), 2014.
Ton Steerneman. On the total variation and hellinger distance between signed measures; an application to product measures. Proceedings of the American Mathematical Society, 88(4))6844688, 1983.
Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wengi Wei. Demystifying membership inference attacks in machine learning as service. IEEE Transactions on Services Computing. 2019.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE 31st Computer Security Foundations Symposium (CSF). IEEE, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2019.
12

---

Published as a conference paper at ICLR 2022
Appendix
Table of Contents
Theoretical Analysis
A.1 How Does Gradient Ascent Step Increase Loss Variance?
13
A.2 How Does RelaxLosss Affect MIA?
15
B Experiment Setup
16
B.1 Datasets
................................
16
B.2 Model Architectures
................................
17
B.3 Implementation details
................................
B.4 Required Resources
B.5 Defense Methods
17
C Additional Results and Discussion
19
C.I Limitations and Future Works
19
C.2 Correlation Between Loss Variance and Attack AUC
19
C,3 RelaxLoss Vs. Attacks
19
C.4 Adaptive Attack
20
C.5 Generalization Gap
20
Compatibility with Data Augmentation
21
C.7 Computational Complexity
21
C.8 Effect on Different Classes of Individuals
21
C.9 Privacy-utility Curves
22
C.10 Loss Histograms
26
C.11 Analysis of Model Generalization
28
This appendix provides additional support to the main ideas presented in the submission: $A provides additional theoretical analysis giving rise insights on the foundations of method. Moreover, as we have conducted a rigorous and broad experimental analysis that goes beyond the key insights presented in the main paper, we provide additional details on the experimental setup in $B and range of additional evaluation results and discussion in SC.
A THEORETICAL ANALYSIS
A.1 HOW DOES GRADIENT ASCENT STEP INCREASE LOSS VARIANCE?
In this section, we show how the gradient ascent step in RelaxLoss increase the loss variance. We write l for the loss instead of E(O, z²) for brevity if the dependence is not relevant for our argumentation.
Theorem A.1. If Cov (e, A > 0, then the variance of loss distribution Var(e) is increased after gradient ascent step.
Proof. The variance of loss distribution before and after applying gradient ascent step amounts to Var(E) and Var(f + Al), respectively. Following from the fact that
Var(f + Al) Var(!) Var(Ae) + 2Cov AE)
and the non noganigayyvy of Var(Af) as well as Cov(E, Al). we conclude Var(é + AC) > Var(f), ie.. the loss variance will increase.
13

---

Published as a conference paper at ICLR 2022
We focus the loss increase after the gradient ascent step (i.e., assuming that A O, which holds for most cases, despite the stochasticity) and interpret A as the rate of loss change. The condition Cov(e, Ae) 0 can be understood versa. This is a reasonable assumption for most training algorithms for achieving convergence. We reason the exact condition and the related assumptions below.
the larger the loss value is, the faster it changes, and vice
Condition A.1. The gradient magnitude (squared &g norm) is positively correlated to the loss value, Le., [[]] > 0. Intuitively, it means the gradient norm tends to decrease the loss decreases.
We use the cross-entropy loss an example:
lcE(O, z.) Σ
2/ log P
(3)
The gradient given by:
v(c=(0, (p: y1)
(4)
where Je; represents the jacobian of the logits (before the final softmax layer) w.r.t. the model parameter 0. Once the loss on sample 2; becomes smaller, we have ||p: y: yilla - O, i.e. the prediction get closer to the ground-truth label. By the submultiplicatiiity of matrix norm and the continuity of the squared function, we then have VVCE =:)1 - 0, i.c. the gradient norm decreases the loss value gets smaller. Hence, (C z.) has the desired property required by Condition A.
Condition A.2. The change in loss after the gradient ascent step A is linear in the squared gradient norm / i.e., Al = e1 / 02 with C1, O2 the constants quantifying the linear relationship. Corollary A.1. Given the assumption that the gradient of each sample within batch (/) has the same norm and (2) has non-negative inner product (i.e., well-aligned) with each other and the gradient alignments remain the same over different batches, the gradient ascent step: ((++1) (()) (()( satisfy the linearity (in Condition A.2) with c > O, where the superscript t corresponds to the (-th iteration, and V L denotes the batch gradient with batchsize = B.
Proof. This follows from the nature the first-order gradient-based optimization method. Applying first-order Taylor-expansion of the sample loss at ((t), we obtain:
((((++))) 2;) ((e()) + ((VE( ), VL(O()) + O(T
Af= = ((0(++1), 22) - (() 2,)
(5)
= T (V(( ,0,11 + T Σ (V(( 2;), VE(O()) ,2,)) + +o(
B
B iti
VV(O 211111 B
V 21)22 (V 22)1l2 cos(au)) + O(T)
B
T V 22 211 +
Σ cos(a:;)) + 0
= B
(6)
where cos(a;)) is the cosine of the angle between gradients of sample i and j. and O(T) summarizes the higher-order terms and regarded as a constant (i.c., c2). Given the assumption that each sample within batch exhibits well-aligned gradients with the same norm, i.e., ||Ve( 2, 2,)112 = |[V)((()) z,)ll2 and cos(a.)) 2 for all j + i, we have Equation and _ {11 Ej cos(a.;)) > Additionally, given that the gradient alignments remain the same over different batches, i.c.. (1 Ejtt cos(a;))) is constant for all j, we have = 3
(1 Σnjt cos(or,)) > 0.
Lemma A.I. Given the Condition A. and A.2, have the desired property Cov(f, Al) by linearity.
Proof.
Cov AC) Cov(e, V 02) = Cov(E, [(( >
(8)
where Equation 7 and 8 are yielded by using Condition A.2 and A.I, respectively.
14

---

Published as a conference paper at ICLR 2022
A.2 How DOES RELAXLOSS AFFECT MIA?
In this section, we show how RelaxLoss affects the optimal MIA Aopt (measured by its AUC value). We exploit the following results for relating the attack AUC to statistical distance between the loss distributions.
We first regard the MIA a binary hypothesis testing problem with the null Ho and alternate hypothesis H1 defined as follows:
Ho 2i a member sample, i.e., 2; E Dirain
H1 2; is a non-member sample, i.c., 2; & Drrain
The attacker need to make a decision on whether the query sample came from Drrain based on rejection region Sreject- As discussed in Section 4.1, under posterior assumption on the model parameter, Sreject for the optimal attack Appt (Sablayrolles et al., 2019) fully depends on the sample loss, i.e. Aopt rejects the null hypothesis if E(O, z,) E Sreject The type I error (i.e., the Ho is true but rejected) is defined as P(E(O, Derain) E Sreject), and the type II error (i.e., the Ho is false but retained) is defined as P(E(O, Ptrain) Sreject).
Theorem A.2. (Kairouz et al., 2015; Lin et al., 2018; 2021) Let TP and FP denote the true positive rate (1- type II error) and false positive rate (type error) of Aopt respectively, their relation to the total variation distance between the loss distributions Drv (P, Q) quantified follows (See Kairouz et al. (2015) Appendix A for the derivation):
TP < FP min(Drv(PPQ),1-FF))
where P and Q denote the distribution of the training loss ((O, Dirain) and the testing loss E(O, Derain). respectively.
The ROC curve obtained by plotting the largest achievable true positive (TP) rate on the vertical axis against the false positive (FP) rate the horizontal axis, while the AUC value corresponds to summation over pairs of TP and FP.
Corollary A.2. The AUC value can be upper bounded as follows (Lin al. (2021) Corollary 1): (P.
AUC - DTv() (rv 112
(10)
For case of analysis, then upper bound the total variation distance via the Hellinger distance, which is symmetric and has a closed-fone foom expression for common distributions such as Gaussian. Theorem A.3. (Steerneman, 1983)The total variance distance can be upper bounded by the Hellinger distance:
Drv (P. Q) S (2Dn(P,Q)
(11)
where the Hellinger distance satisfies 0 S QH(P,Q) <
For Gaussian distributions, the Hellinger distance has a closed form:
20 exp (-1 (M) H22
(P.
V o +
(12)
4 o. + oz
where μl, #2 denote the mean and d2 denote the variance of P and Q.
Let 02/01 denote the ratio of the training and testing loss variance, we see that D (P, Q) is fully characterized by: (i) the value of the training loss variance of, (ii) the squared distance between the mean (f1 )2 and (iii) the variance ratio
2c 1
1 (11-122) 22)
(P.()) =
exp
+1
(13)
(*+)
Our approach decreases the (ny ? and increases of (Section 5) well. both of which lead to a decrease of the Hellinger distance, and thus decreases the upper bound of the attacker AUC as desired.
15

---

Published as a conference paper at ICLR 2022
It remains consider how our approach will change and how the change in will affect the Hellinger distance. First, we observe that с 2 1, i.c., the testing distribution has larger variance than the training distribution (See Appendix C.10). Moreover, gets closer to when applying approach (See Figure| in the main paper). As a result, (+) will increase (Corollary A.3) and (**) will decrease (Corollary A.4).
Corollary A.3. If 2 1, then V Itc² 2
2c
Itez
Proof. Let f(c) = <= c= i..., (1(1) =0 and (c) 0.
2c
We have f'(c)
It obvious that f has critical point at V2c(c2+1)072
l-ca
Ito22
Corollary A.4. For fixed value of μl μ2 and 01, if 2 cz 1 1, then
- 22)²
1 (11 - H22
exp
4 (1+
S exp
( (1 c2
Proof. It is obvious as e occurs in the denominator inside the exponential term.
Additionally, we notice that the change in the (x) dominates in most cases: the (**) term commonly has value within [0.9. 1.0]. while the (+) term changes from 10-3 to I when our approach is applied. Therefore, under Gaussian assumption of the loss distributions, our method can decrease the Hellinger distance between the distributions, thereby reducing an upper bound the attack AUC.
B EXPERIMENT SETUP
B. DATASETS
CIFAR-10 (Krizhevsky al., 2009) is dataset of 60k color images with shape 32 32 3. Each image corresponds to label of 10 classes which categorizes the object inside the image. Following normalize the image pixel value have zero mean and
the standard preprocessing procedure unit standard deviation.
CIFAR-100 (Krizhevsky et al., 2009) consists of 60k color images of size 32 32 3 in 100 classes. Same as for the CIFAR- 10 dataset, we perform mean mean-subtraction and standardization.
CH-MNIST (Kather al., 2016) contains 5000 greyscale images of different types of tissues from patients with colorectal cancer. We obtain the preprocessed dataset from Kaggle and use images of size 28x28 for our experiments. All images are normalized to [-,,
Texas100 contains medical records of 67,330 patients published by the Texas Department of State Health Services 8, Each patient's record contains 6,169 binary features (such diagnosis, generic information, and procedures the patient underwent) and is labeled by its most suitable procedure (among the 100 most frequent ones). We use the preprocessed data provided by Shokri al. (2017); Song & Mittal (2020)).
Purchase 100 is dataset customers' shopping records released by the Kaggle Acquire Valued Shoppers Challenge 10 We use the preprocessed version provided by Shokri et al. (2017); Song & Mittal (2020).. which contains 197,324 data samples. Each sample, representing one user's purchase history, consists of 600 binary features. Each feature denotes the presence of one product in the corresponding user's purchase history. The data is clustered into 100 classes different purchase styles. The classification task is to predict the purchase style given the 600 binary features.
We summarize all datasets details in Table 3.
7 http://wwwggaooooooooo/oooooooaa 9 https:/wwwwccaaa...rrrrr/rr/r//r
https://www.dss.. wtwp:/ssss...tssss...sso
http:://g.hh.....irrrrrrmmmooooe uutiion https://www,Kar
16

---

Published as a conference paper at ICLR 2022
Feature
Neraia/ (target model)
Narain/
Dataset
Data type
Feature type
Miotal
dimension
(shadow model)
CIFAR-II
color image
numerical
3072
60000
12000
12000
CIFAR- 100
color image
numerical
3072
60000
12000
12000
CH-MNIST
grayscale image
numerical
784
5000
1000
1000
Texas100
medical record
categorical
6169
67330
13466
13466
Purchase
purchase record
categorical
197324
39465
39464
Table 3: Summary of datasets, Nosta denotes total dataset size. Ntrain and Nust are the size of the training and testing set, respectively.
B.2 MODEL ARCHITECTURES
For CIFAR-I0 and CIFAR- 100 datasets, we use a 20-layer ResNet and an 11-layer VGG architec- adopt 20-layer ResNet. And for the non-image datasets, adopt the
ture ". For CH-MNIST, same architecture as used in Nasr et al. (2018) ? a 4-layer fully-comnected neural network with layer size [1024, 512, 256, 100] for Purchase 100, and 5-layer fully-connected neural network with layer size [2048, 1024, 512, 256, 100] for Texas100.
B.3 IMPLEMENTATION DETAILS
We apply SGD optimizer with momentum=0.9 and weight-decay=le-4 by default. We set the initial learning rate т 0.1 and drop the learning rate by factor of 10 at each decay epoch 11 We list below the decay epochs square brackets and the total number of training epochs are marked in parentheses: CIFAR-10 10 and CIFAR- 100 [150,225] (300); CH-MNIST [40,60] (80); Texas 100 and Purchase100 [50,100] (120). Additionally, we adopt the following techniques for improved performance across heterogeneous data modalities: restrict the scope of posterior flattening to incorrect predictions for natural image datasets (CIFAR-10 and CIFAR-100); and we further suppress the target posterior scores of ground-truth class pat to small values (50.3) during the posterior flattening step on non-image data (Texas 100 and Purchase 100). By default, no data augmentation is applied.
B.4 REQUIRED RESOURCES
All our models and methods implemented in PyTorch. Our experiments are conducted with Nvidia Tesla V100 and Quadro RTX8000 GPUs. Our method introduces minimal changes and negligible additional cost compared with vanilla training and thus can be flexibly integrated into any deep learning framework without imposing specific constraints on the required hardware resources.
B.5 DEFENSE METHOSS
Early-stopping. The basic idea behind Early-stopping is truncate training before model starts to overfit. In our experiments, we save target models' checkpoints at varying numbers of training epochs and subsequently evaluate the attack AUC and test accuracy of each model check-
point. We set checkpoints at the following epochs: 225 50, 75, 100, 125, 150, 175, 200. 225, 250, 275] for CIFAR-10 and CIFAR- 100 datasets; [5, 10, 15. 20, 25, 30, 40, 50] for CH-MNIST dataset; [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110) for Texas 100 and Purchase 100 datasets.
Dropout. Dropout prevents co-adaptation of feature detectors by randomly masking out set of neurons in the networks, thereby alleviating model overfitting. In our experiments, we apply dropout to the last fully-connected layer of each target model and evaluate across a wide range of dropout rates (over (0.1, 0.3, 0.5, 0.7,).
Label-smoothing. habellssoothhigg prevents overconfident predictions by incorporating regular- ization term into the training objective that penalizes the distance (measured by the KL-divergence) between the model predictions and the uniform distribution. The objective is formularized as follows
-=a D
+ (1 - ()) (cF(0,z)
(14)
https://gg chub.ccmmsst. http:SPIINNMMMMMMMMMMP
izaiion
17

---

Published as a conference paper at ICLR 2022
where DKL is the KL-divergence; U4 denotes the uniform distribution; po(y) denotes the output prediction. a is a apper-parameter with range [0. 1] that balances the cross-entropy loss LCE and the regularization term. We vary the a across its full range for plotting the privacy-utility curves.
Confidence-penalle Confidence-perr pentity regularizes models by penalizing low entropy output distributions. This is achieved via an entropy regularization term in the objective:
L = aa. H(po(y)a)) (=) + LcB(0,
(15)
where H(po(ylz)) = Σ po (x) log(po
(16)
H. represents the entropy of the output prediction, and a is a hyper-parameter that controls the importance of the entropy regularization term. Consistent with the original paper (Pereyra et al., 2017), we vary the over [0.1, 0.3, 0.5, 1.0, 2.0, 4.0, 8.0.
Distillation. Knowledge distillation stands for a general process of transferring knowledge from a set of teacher model(s) to student model. To focus our investigation the effect of the distillation operation itself, we use self-distillation (Zhang et al., 2019) our experiments, i.e., we train the student model to match single teacher model with the same architecture. The objective for training the student model (i.c., the target model) is:
aT DKL (Fo, (g/(z) |1 Po. (0]=)) (1-a)) a). Lce(0,2) 2)
(17)
(0.
where Po (o(y)=) = E.
(18)
The KL-divergence term DRL targets minimizing discrepancy between the softened student Po, (y]æ) and teacher prediction Pe, (y]a)) T denotes the temperature scaling factor that controls the degree of softening. cross-entropy LCE term. To determine the that balances the KL-divergence and the normal that best describes the privacy-utile trade-off, we conduct preliminary experiments and investigate the effect of and T independently. By fixing one and changing the other, we observe similar results in terms of the privacy-utility trade-off. Following practical standards, we then fix the a=0.5 and vary the temperature over [1,2, 5, 10, 20, 50, 100] for plotting the privacy-utility curves.
DP-SGD. DP-SGD enforces privacy guarantees by modifying the optimization process. consists of two steps: (i) clipping the gradients to have a Z--norm upper-bounded by C' at each training step; (ūi injecting random noise to the gradients before performing update steps. We adopt the implementation provided by the Opacus library 13, We tune the clipping bound C' when fixing the noise scale to 0.1, as suggested by the official documents. To plot the privacy-utility curves, we vary the noise scale with fixed pre-selected clipping bound.
Memguard. Memguard modifies the output predictions of pre-trained target models during test-time, ie., output predictions are perturbed by adversarial noise to fool surrogate attack model. Following the official implementation 14 we adopt the same architecture for the surrogate and the real NN attack model, and use the complete logits prediction input the attack models. Each surrogate attack model is trained on the target model's predictions when inputting the target model's training data (used as member samples) and a separate hold-out set data (used as non-member samples). The privacy-utility trade-off is fully determined by the magnitude of the adversarial noise (measured by Ln-omm)) We plot the privacy-utility curves by increasing the noise magnitude from 0 until the NN attack has been defended (i.e., attack AUC x= 0.5).
Adv-Reg. Adv-Reg incorporates an adversarial objective when training the target model: the target model is trained to minimize a weighted sum of the cross-entropy loss and the adversarial loss (ob- tained from a surrogate attack). Same in Memguard, each surrogate attack model is trained on the target model's 's training data and a separate hold-out set data, We follow the official implementation 12 and vary the weight a (=1.0 by default) of the adversarial loss across [0.8, 1.0, 1.2, 1.4, 1.6. 1.8] for plotting the privacy-utility curves.
ceattorrchoopuuss /opaus
https://gh com/ jjy 994 /MmGGaadd
18

---

Published as a conference paper at ICLR 2022
C ADDITIONAL RESULTS AND DISCUSSION
C.I LIMITATIONS AND FUTURE WORKS
Although Loss is empirically proven effective for improving target models' utility, is generally hard to explain such improvement, as understanding the generalization ability is still open problem. As an attempt, we conduct experiments on toy datasets and attribute the improvement to a flat decision boundary (See Appendix C.I A thorough investigation into how each individual components of our approach affects generalization are left for our future works. In addition, the assumptions of model parameters (Sablayrolles al., 2019) which are made for the optimal attack demand further validation.
C.2 CORRELATION BETWEEN LOss VARIANCE AND ATTACK AUC
We plot the attack AUC values versus the training loss variance in Figure 6. Each point in the figure corresponds to one target model trained with different defense mechanisms. We indeed observe negative correlation between the loss variance and the AUC value of both the black-box and white-box attacks, which supports Section the main paper.
Black-bre
White- box
AUC
0.7
0.7
0.5
0.5
0.2 Variance
0.3
0.4
0.2
0.3
0.4
Variance
(a)
(b)
Figure 6: Correlation between the training set loss variance and the MIA performance on CIFAR-10 (ResNet20). The Pearson's correlation coefficients equal (a) -0.77 for black-box attacks; (b) -0.94 for white-box attacks; and -0.85 if considering black-box and white-box attacks together.
C.3 RELAXLOSS Vs. ATTACKS
Supplementary to Table in the main paper, Table 4 summarizes the top-1 training and test accuracy as well as the generalization gap (i.c., difference between the top-1 training and test accuracy) of target models with or without being defended via our method. We observe that our approach, as desired, reduces the generalization gap and is still able achieve high performance.
(ReeNetti:
CIFAR-10
CIFAR-TO (VGGL)
CIFAR-TOR
(ReeNer20)
CHFAR-TOO
(VGGHL)
CHUMNIST
Texas 100
Purchase 100
trin
1891
main
38
Trin
1891
Tain
1891
TRai
AR0
TRan
TEST
23p
tran
1691
No deleuse
100
M0.
100
733
26.2
1001
352
66.8
1031
41.4
38.6
7550
71.
419
9538
23.3
47.6
1001
29.1
10.9
with defense
$7.0
7388
13.2
99.5
24.4
25.1
52.7
35.1
17.6
99.7
414
56.3
90.1
78.4
11.
67.1
35.3
1188
99.8
89.1
10.7
A
-1.0
33
-83
288
86
-11
-7.3
1.0
493
-63
00
208
469
13
10.
-288
30
353
-02
0.0
202
velesial
84
113
0.2
2.5
13
Table 4: The top-1 training and test accuracy as well as the generalization gap (in %6) of the target models with without (wo) applying our defense. A corresponds the absolute difference after applying our defend method (in %). We also include the selected value of a. This is supplementary Table in the main paper.
In Table 5, we show the attack accuracy values on target models trained with and without (wo) applying our defense method, which is supplementary to Table and Figure 4 in the main paper. Same in previous work (Song & Mittal, 2020)), the attack's decision threshold is selected to be the one that yields the best attack accuracy on undefended shadow models. We observe that our approach effectively reduces the attack accuracy random- guessinng level (around 50%) for most cases. In
19

---

Published as a conference paper at ICLR 2022
Entropy 86.5
M-Entropy
Loss
NN
Grad-x &
Grad-x 62
Grad-w
Grad-w é2
wo defense defénse
87.3
86.9
82.3
87.3
87.3
87.8
87.8
CIFAR-10
(ResNel20)
-42.2
-2.77
-42.5
-39.5
42.99
-4.99
-43.2
-43.1
CIFAR-10
wo defense
80.1
80.7
80.6
76.
81.3
81.3
82.7
82.9
(VGG11)
defense
50.0
50.0
-37.6
-38.0
-38.0
-34.3
-38.7
-38.5
-39.5
-39,7
CIFAR-100
wo detense
91.8
92.1
92.6
87.0
93.7
93.7
94.6
94.7
(ResNer20)
defense
50.0
-55.5
-45.7
-46.0
-42.5
-46.6
-46.6
-47.1
-47.2
CIFAR-100
wo defense
97.1
97.5
97.4
98.
98.5
98.4
98.9
98.9
defense
50.4
(VGGL1)
-8..5
-48.7
-48.7
-48.4
-49.1
-49.1
-48.8
-49.0
wo defense
55.5
56.7
56.7
63.6
67.6
67.7
67.1
66.4
CH-MNIST
defense
51.7
-10.1
-7.8
-10.2
-20.9
-22.8
-23.6
-24.4
-24.8
wo detense
70.3
79.0
79.0
63.8
78.3
78.5
78.4
78.3
Texas100
defense
28.9
-36.7
-36.7
-21.6
-33.8
-32.5
-34.8
-36.1
wo defense
63.9
64.8
64.7
62.6
65.8
65.7
63.8
65.7
Purchase
52.2
-21.8
-22.8
-22.7
-20.8
-20.5
-20.5
-23.9
-23.9
Table 5: The attacker accuracy (in %) evaluated on the target models with and without (wo) applying our defense. A corresponds to the relative difference after applying our defend method (in %). All thresholds are selected with undefended shadow models trained with shadow dataset.
particular, we find that the selected decision thresholds is highly biased in certain cases s.t. all the queried samples are predicted to positive (or negative), which leads to exactly 50% accuracy.
C.4 ADAPTIVE ATTACK
Table 6, we show accuracy adaptive (a.) attacks on target models trained with (w/) applying defense method, which is supplementary to Section 6.4 in the main paper. For thresholding based attacks, the attack's ' decision threshold selected to the one that yields the best attack accuracy on the shadow models (which is trained with exactly the same configuration as defended target models in Table 1). And for the NN-based attack, we use the complete logits prediction from the pre-trained shadow models as features to train the adaptive attack models (modeled as NN). We observe that our approach consistently reduces the attack accuracy for all cases, though the reduction less significant compared to non- adaptive attacks shown Table 5,
Entropy 52,5
M-Entropy
NN
Grad-x E
Grad-x l
Grad-w
Grad-w
CIFARIO
w/ defense (a.)
56.0
56.0
54.2
53.5
53.3
54,0
53.8
(ResNer20)
-39.3
-35.9
-34.3
-38.9
-38.5
CIFARIO
defense
64.4
68.2
67.8
66.6
66.2
66.4
67.4
68.0
(VGG1I)
A
-19.6
-15.5
-15.9
-12.5
-18.8
-18.3
-18.5
-18.0
CIFART00
defense (a.)
52.2
37.8
53.6
50.1
50.0
(ResNet20)
A
-43.1
-37.2
-37.6
-38.4
-46.5
-6.55
-47.1
-47.1
CIFART00
W/ defense (a.)
78.9
84.2
84.0
83.4
78.2
78.2
81.6
82.2
(VGG11)
-1877
-17.5
CH-MNIST
defense
50.7
53.9
53.4
50.9
55.8
55.7
56.6
56.4
8.6
.4.9
-20.0
-17.5
-17.7
-15.6
-15.1
Texas100
defense
31.6
33.8
32.1
31.8
53.4
A
26.66
-31.9
-31.9
-18.3
-33.9
-33.9
-33.9
-31.8
w/ defense (a.)
54.0
54.8
54.8
54.5
55.4
55.4
55.6
56.0
10rchase100
-15.5
-15.4
-15.3
-12.9
-15.8
-15.7
-15.5
-14.8
Table 6: The accuracy (in %) of adaptive (a.) attacks evaluated on the target models with (w/) applying defense. A corresponds to the relative difference (in %) attack accuracy when applying our defend method compared to vanilla training. The selected target models are the same as in Table
C.5 GENERALIZATION GAP
We show the training and testing accuracy (and the generalization gap) when applying RelaxLoss with varying value of in Figure 7. We observe that increasing the value of a will reduce the generalization gap. Moreover, RelaxLoss with a reasonable value of a can even improves the test accuracy of vanilla training.
20

---

Published as a conference paper at ICLR 2022
(eas)
Than (so safena))
Tax dotera)
51
Tran
(oea)
denrr))
0.30
000
6:75
1.25
1.30 1.20
2.0 03 12
10
15
20
25
10
(a) CIFAR-I0
(b) CIFAR-100
Figure 7: Training and testing accuracy (y-axis) with varying value of (X-axis) CIFAR-10 (ResNe120) and CIFAR- 100 (ResNet20) datasets. We plot the training and testing accuracy of vanilla training (wo defense) in dashed lines for reference.
C.6 COMPATIBILITY WITH DATA AUGMENTATION
Additionally, we investigate the effectiveness of our approach when data augmentation is applied. Following practical standard, we apply random cropping and random flipping when training the target models on CIFAR-100 dataset. As illustrated in Figure 8, Relaxl Loss is compatible with standard data augmentation techniques: our approach enjoys the performance boost introduced by data augmentation while retaining its effectiveness in defending MIAs.
M
M
Gopoc
Gien
45
05
23
87
09
100
05
E3
E7
39
1.0
05
08
C7
12
13
02
02
07
63
13
35
05
03
08
05
18
35
35
07
08
05
Oue
Utal Schtn
Figure Effect of data augmentation (denoted as "aug") on CIFAR- 100 (ResNet20). When jointly applied with data augmentation, our approach shows consistent effectiveness in improving the MIA resistance and model utility.
C.7 COMPUTATIONAL COMPLEXITY
The additional computation cost of RelaxLoss scales O(BC) (B: batch size; C: number of classes), which includes: (i) softlabel construction of cost (BC)); and (ii) computation of the cross-entropy loss on the softlabel O Note that we reuse the prediction p: generated by the previous forward-pass and thus no additional forward (nor backward) pass is required. Compared to the forward and backward pass, which is of magnitude least O(BNL) (N: number of neurons per layer, L: number of layers), the additional costs of RelaxLoss are negligible as NL (roughly the total amount neurons of the whole network) much larger than C' (the number neurons of the last layer).
C.8 EFFECT ON DIFFERENT CLASSES OF INDIVIDUALS
To analyze the effect of Relax Loss on different individuals, we conduct additional experiments on Texas and Purchase datasets which consist of 100 classes with non-uniform class distribution (i.c., the proportion of each class ranges from 0.35% to 4.5% for Texas, and 0.05%-2.6% for Purchase) and evaluate the attack performance each class separately.
In Table 7, we show the 10 highest Attack AUC (in increasing order) among all the classes, which can be regarded the estimated worst-case privacy risk of different classes of individuals. As be seen from the tables, applying our defense method effectively reduces the Top- 10 Attack AUC (i.c., worst-case privacy risk), and the effectiveness is consistent each dataset across different attack
21

---

Published as a conference paper at ICLR 2022
methods, with which conclude that our method, despite the nonuniformity, does defend MIAs for different individuals.
(a) Texas
Atttack methods
with/wo defense
Top-10 Attack AUC
wo defense
0.985, 0.987, 0.988, 0,989, 0,994, 0.994, 0.995, 0.996, 0.998, 0.999
Loss
with defense
0.717 0.719, 0.721, 0.722. 0.724, 0.739. 0.741. 0.743. 0.752. 0.761
defense
0.846, 0.847, 0.851, 0,851, 0,854, 0.864, 0.865, 0.868, 0.879, 0.880
Entropy
with defense
0.617, 0.619, 0.625, 0.625, 0.636, 0.636, 0.648, 0.679, 0.733, 0.747
wo defense
0.985, 0,987, 0,989, 0.990, 0.992, 0.993, 0,994, 0.996, 0.997. 0,999
M-Entropy
with defense
0.717, 0.718, 0.722, 0.722, 0.724, 0.741, 0.742, 0.742, 0.754, 0.761
wo delense
0.837, 0.837, 0.844, 0,848, 0.850, 0.852, 0,859, 0,865, 0,903, 0.943
Grad-x 12
with defense
0,644, 0.650, 0.653, 0,655, 0.657, 0.666, 0.679, 0.691, 0,723, 0.744
defense
0.850, 0,852, 0.853, 0.856, 0,862, 0.862, 0.863, 0.866, 0.867, 0.874
Grad-w 12
with defense
0.627, 0.631, 0.632, 0.632, 0.633, 0.634, 0.643, 0.644, 0.661, 0.663
(b) Purchase
Atttack methods
with/wo defense
Top- 10 Attack AUC
wo detense
0.699, 0.709, 0.715, 0.719, 0.727, 0.731, 0.735, 0.738, 0.763, 0.875
Loss
with defense
0.663. 0,666, 0.671, 0.672, 0.684, 0.692, 0,694, 0.701, 0.705, 0.722
wo defense
0.692, 0.697, 0.714, 0.714, 0.718. 0.724, 0.725, 0.725, 0.747, 0.868
Entropy
with defense
0.651, 0.651, 0.654, 0.657, 0.658, 0.673, 0.678, 0.681, 0.695, 0.711
wo defense
0.699, 0.711, 0.716, 0.720, 0.727, 0.731, 0.734, 0.739, 0.765, 0.875
M-Entropy
with defense
0.663. 0.666. 0.671, 0.672, 0.684. 0.693, 0.694, 0.701, 0.705, 0.721
wo defense
0.708, 0.725, 0.730, 0.733, 0.736, 0.738, 0.742, 0.747, 0.777, 0.897
Grad-x 12
with defense
0.653, 0.662, 0.662. 0.667, 0.667. 0.669, 0.672, 0.684, 0.696. 0.697
wo delense
0.662. 0.664, 0.665, 0.666, 0.666. 0.668, 0.670, 0.67I. 0.68I, 0.710
Grad-w 12
with defense
0.629, 0.629, 0.632, 0.634, 0.634, 0.638, 0.639, 0.654, 0.655, 0.663
Table 7: Top- 10 Attack AUC among 100 label classes on (a) Texas and (b) Purchase with and without (wo) applying our defense. The AUC values are shown in increasing order.
C.9 PRIVACY-UTILITY CURVES
In this section, we include detailed results with regard to various datasets and different target models' architectures: we show results on CIFAR- dataset with VGG11 architecture Figure 9; CIFAR-100 dataset with VGG11 architecture in Figure 10; CH-MNIST with ResNet20 architecture in Figure 11; Texas100 with MLP architecutre Figure 12; Purchase 100 with MLP architecutre in Figure 13.
We observe that while baseline approaches are effective for at most one data modality, our approach is the only one that is consistently effective in defending MIAs, across all different datasets and model architectures.
Natural Images. See Figure 3(a)-3(b) in main paper, and Figure 9-10: for natural image datasets (CIFAR-10 and CIFAR-100), DP-SGD is the best baseline method in terms of defending MIAs and preserving model utility, but is inferior to RelaxLoss as our method consistently achieve better test accuracy (model utility) across the full range of achievable privacy level.
Among all regularization-bn -baedd defense methods, Early-stopping is the only one that exhibits notice- able effects reducing attack AUCs. In comparison, our approach can achieve the same level of defense effectiveness with much better model utility. Moreover, our approach can further decrease the attack AUC a random- random-guessing level, which is not achievable by Early-stopping.
Memguard and Adv-Reg, previous tfte--oftth-arr defense mechanisms specifically designed for MIAs, are highly effective in defending NN-based attack but generally lose their effectiveness for other types of attacks. In comparison, our approach shows much better defense effectiveness for all types attacks while achieving better model utility at the same time.
Gray-scale Medical Images See Figure 11: for gray-scale medical image (CH-MNIST), approach is comparable with the best baseline methods (i.e.. Confidence- Senfidenne-penalty), as both approaches are able reduce the MIAs random-guessing level while preserving the model utility.
22

---

Published as a conference paper at ICLR 2022
NN
60
Extrapy
10
M-Ertrert
60
892
33
10
30
Attock ABC
Ours
DP
Doopout
Dedi on Ade Beg
Menguand ideal dedease
deleese
Cooidence-peett peroby
Lahelamoothing
-
doopiig
(a) Black-hox attacks
Gradae
Gradin
Godden E
80
Gradier E
03
60
60
30
10
10
Attack ABC
Attack AUC
Actack ALC
Attack AlC
Drapout
Corridnnc-
Lebolancating
Early stopping
Adw Rox
X
dofans
(b) White-box attacks
Figure 9: Comparisons of all defense mechanisms on CIFAR-10 dataset (VGGII). We the clipping bound C = 0.5 and vary the noise scale over 0.01-0.45 for DP-SGD, We vary the noise magnitude across 0-20 for Memguard.
NN
Entrupy
M-Entrop
Attock AUC
Attack MJC
Artack
Despost
Mamguard (deal defiase
definae
Cooidence- peroby
Labelanoothing
Adw Reg
(a) Black- box attacks
God
Gradia ft
Godden ft
Gradier Ey
0.5
Attack AUC
Attack AUC
Antack AUC
Attack AUC
Dropoot
Corfidnnc
Labelancading
Early stopping
Adw Rox
X
dofans
(b) White-box attacks
Figure 10: Comparisons of all defense mechanisms on CIFAR- 100 dataset (VGGII). We set the clipping bound C=1.0 and vary the noise scale over 0.01-0.3 for DP-SGD, We vary the noise magnitude across 0-400 for Memguard.
In comparision, DP-SGD is significantly worse than our approach for this data modality, as inevitably degrades the model utility.
Memguard and Adv-Reg are still highly effective in defending NN-based attack. Moreover, Mem- guard is able defend black-box MIAs utility, but is not applicable to white-box attacks. In contrast, our approach applicable to all attacks, and achieves comparable (or better) effectiveness for blx attacks.
seddoo--uesssing level without degrading the model
23

---

Published as a conference paper at ICLR 2022
NK
93
60
Extropy
M-Ertrop
10
193
53
50
50
Attock ABC
Ours
DP SGB
Doopout
Deti on
Menguand ideal dedease
defeese
Confidence peroby
Labelamoothing
doopiig
Adw Beg
(a) Black-hox attacks
Gradie
Gradie fg
Gudden Eo
00
Gradier Ey
83
60
80
70
50
90
40
Attaak ABC
Attack AUC
Actack ALI
Attack AUC
Drapout
Corridenc-
Lebobancading
Early stopping
Adw Rox
dofans
(b) White-box attacks
Figure 11: Comparisons of all defense mechanisms on CH-MNIST dataset. We set the clipping bound C=5.0 and vary the noise scale over 0.001-0.5 for DP-SGD. We vary the noise magnitude across 0-50 for Memguard.
Binary Medical and Transaction Records. See Figure 12-13: for binary medical and transaction records (Texas 100 and Purchase 100), Label-smoothing performs generally the best among all baseline methods. Compared to our approach, Label-smoothing can achieve superior model utility when the Attack AUC is of range around 0.65-0.8 on Texas and 0.55-0.65 Purchase 100. However, our approach is able to reduce the attack AUC to around the random-guesse gunnsnggssiingg level, which is not always possible for Label-smoothing (white-box attacks on Purchase 100, black-box attacks on both datasets).
DP-SGD exhibits unexpected results for these datasets: small noise scale results in both lower Attack AUC and higher model utility, which is contradictory to the common belief that scall-scale noise only provides weak privacy guarantee and thus the Attack AUC will remain high. We conjecture that there exists a non-negligible gap between the worst-case privacy guarantee that provided by the theoretical privacy analysis and the real-world attack performance in practice. Especially for the small-scalc noise case, the privacy cost is meaninglessly large and cannot faithfully reflect the risk when facing practical MIAs. We tried varying the noise scale in the experiments: by decreasing the noise scale, find DP-SGD cannot reduce the Attack AUC random- -ueossing level, while by increasing the noise scale, the utility soon drops significantly. In comparison, our approach consistently yields better privacy-utility trade-off.
24

---

Published as a conference paper at ICLR 2022
73
70
70
63
60
69
100
0.7
Oun
DF-5GD
Diopoue
Dati bat on Ade Rog
Mempand ddal defense
X
defénan
Corfidancs peralty
Labelarocthing
Early Carlyatozziig
(a) Black- box attacks
Gradie
Gradi fg
Goader
70
Gradier Eg
60
00
Attuck AUC
MUC
Drapoue
36
Corridence peralty
Liblloonding
Early stopping
Adv Rog
X
delonee
(b) White-box attacks
Figure 12: Comparisons all defense mechanisms on Texas 100 dataset. We set the clipping bound C=1.0 and vary the noise scale over 0.001-0.5 for DP-SGD. We vary the noise magnitude across 0-500 for Memguard.
APM
Fetmey
W-
03
50
94
Tea
60
50
50
50
AUC
AUC
Oun
DF-SGD
Diopoct
Dotille on
Merquar deal defense
no dedinne
Labelonnotiing
Eariy-tosssis
Adl Rii
(a) Black- box attacks
Gradie
Gradia fg
Gradre E
Gradier Eg
90
00
70
83
60
60
50
Attock AUC
ALC
Artack AUC
Drapout
Dotilazion
X
Conrideseepeoo perohyy
Isbelencttingg
eapplinopping
Adw Bag
x
delense
(b) White-box attacks
Figure 13: Comparisons of defense mechanisms on Purchase 100 dataset. We set the clipping bound C=1.0 and vary the noise scale over 10-4-0.4 for DP-SGD, We vary the noise magnitude across 0-300 for Memguard.
25

---

Published as a conference paper at ICLR 2022
C.10 Loss HISTOGRAMS
To better understand the effect of each defense method, we additionally plot the loss histograms when applying different defense methods on target models with a ResNer20 architecture trained on CIFAR-10 dataset in Figure 14-20. In the parentheses of each subtitle, we show the hyper-parameter values corresponding to each subfigure from left to right.
We observe that: (i) Regularization techniques in general have limited effects in reducing the gap between the training and testing loss distributions. (ii) Unlike our approach (See Figure in the main paper), baseline methods are generally not able to increase the training loss variance nor closing the gaps between the member and non non-member distributions, which explains the superior performance of our approach in defending various types of MIAs. (iii) By setting relatively large noise scale, DP-SGD is able to increase the training loss variance and reducing the gap between the training and testing loss values (See last column of Figure 17). However, the large scale of noise dampen the learning signal in this case, leading to a non--nnnngeee negligglee utility drop.
10
noverenbe neeler
creeee nemba
wwwber member
menile
0.0
200
3.6
0..3
(teg
10
adi 00:45
2. unee
aC
Malfi
roreaad
-1.33
1.33
f- 1,42
80
0.0
3.5
105
Figure 14: Loss histograms when applying Label-smoothing (a 0.200,40.66600..8
rce-nender
bod
60
Vilff -7 1o:0
00
Vali Ce:45
Vie 02
Valli +4.09
-0.81
Vaal 18.59
Via 30.45
0.2
14
14
90
Figure 15: Loss histograms when applying Dropout (dropout rate 0.1,0.5,0.7.00.)
novmember nember
roa-rarber rwnieer
eoe-marder menn
20 44
Varf C7
-0.3
Vial 2.48
Vae
11
22
06
230
3.0
1.5
330
020
3.5
3.0
3.5
224
20
330
001
0.5
102
24
23
Figure 16: Loss histograms when applying Confidence-penalty (a 0.1.0..,1.0.2200
26

---

Published as a conference paper at ICLR 2022
100
novernber member
rom-rerbe nember
ParTT and feqquanco
Sregancy
Bl -0.1
+0.09
Viat
16
0.0
1041 125
3541
300
N01
325
25
3.0
Figure 17: Loss histograms when applying DP-SGD (noise scale 0.01,0005.00110055)
1.0
meeber wenber
0,
13 -2.023
Rd Lnnnddd 25
named col
Vie +845
Vadii
34
1.13 Ja
R Mouil 84
Soeff
22
Figure 18: Loss histograms when applying Adv-Reg (ax = 0...1.0,1.22.1.4)
1.0
membay mumber
300
novre member
FCB- vem-ee number
non-eo momber
90TTA and Iogu
6o.04
1o.04
7e-36
Naril
Ein
-0 Volf...-1
02
km
Figure 19: Loss histograms when applying Distillation (T' 1,00,50,100)
navrrrbor nember
1.0
nowtsrber member
1.0
riweber member
ron- mamóor memberr
Eli -0.13
ID 30-01 Caan- 33.6-7
Hir 11 -2
Ei 20-0
L6000
Xi 1.00
B0 1.77
Ky -5.1
Ve
12
62
60
Figure 20: Loss histograms when applying Distillation Early-stopping (ep = 25.50,75,100),
27

---

Published as a conference paper at ICLR 2022
C.11 ANALYSIS OF MODEL GENERALIZATION
As supplementary to Section 7 of our main paper, we show results of toy experiments that investigate the impact of approach model generalization. We visualize the prediction scores and the decision boundaries Figure 21. In contrast to vanilla training that assigns high confidence scores on hard examples near the decision boundary, our approach can soften the decision boundaries, leading to large area with low (and well-calibrated) predicted confidence scores. In line with Zhang et al. (2017); Pereyra al. (2017), we conjecture that the flatness of decision boundaries improves model generalization, while an in-depth analysis is left as future work.
Inpuc data
Vanilla training
Our
data
Vanila training
Ours
Vanila
traiting
Figure 21: Visualization of target models' prediction scores and decision boundaries. The training samples are shown in solid colors and testing points are semi-tre transpranpp.rent.
28


Reviews:
[{"review_id": "e0O-la-XgkO", "reviewer": "Reviewer_zx7i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive aspects:\n-\tThe proposed method outperforms similar defense strategies while preserving the accuracy of the classification model on test data.\n-\tEven with the full knowledge of the attacker on the defense mechanism, the defense algorithm is capable of reducing the attacker’s accuracy considerably.\n-\tTheir defense strategy is computationally efficient (in comparison with other similar methods) and blind to the attack algorithm.\n\nNegative aspects:\n-\tThe paper lacks intuition behind some assumptions and choices made in designing the algorithm (more below).\n-\tThe authors did not discuss limitations of the method (more below).\n-\tSome minor errors that can be easily fixed (see Minor Comments).\n\n---------------------------------------------------\nMajor Comments:\n-\tIn the set of defender assumptions in Section 3, the necessity for considering the first assumption is not clear. In practice, additional unlabeled data is normally available and if they can help with building a stronger defense mechanism, why should we ignore them? This assumption lacks justification.\n-\tI find the theoretical and intuitive justification behind the “posterior flattening” step to be lacking. By discarding the learned information of the network about the non-ground-truth classes and manually replacing the values, my intuition is that there is a good chance we are overfitting (i.e., we can't expect good performance on the test set). The poor testing results in Table 1 for purchase100 data, which has the largest number of training samples, might be hinting at this overfitting. So, it seems the method has its own limitations with regards to the training sample size. I believe authors should discuss this limitation (and any other possible ones) and address them.\n-\tFigure 4 would be more informative if the comparison was made against the best performing baseline instead.\n-----------------------------------------------------\nMinor Comments:\n-\tIn “Notations” (section 3), the sentences are confusing. There are no x, y , p in f(.;theta), and the notation for “1” is not consistent.\n-\tSimilarly, in \"Attacker’s Assumptions”, the definition of m and z should precede that of S.\n-\tIf the attack has full access to the model (as mentioned in Attacker’s Assumptions), it means the attacker has full knowledge on the classifier’s architecture and parameters. So, shouldn’t it be A(z,f(.;theta)) instead of A(z,theta)?\n-\tFigure 1-c, the (E,Val) pairs should be linked to the corresponding distribution.\n", "summary_of_the_paper": "The study tackles the problem of defense against membership inference attack, with a focus on (1) decreasing the performance of the attack, (2) maintaining the classifier’s performance, (3) assuming the blindness towards the attack model. They achieve (1) by closing the distance between the train and test distributions and maintain the utility of the model by flattening the posterior scores of the non-target classes. Their method is shown to be computationally efficient (i.e., the additional computational cost is negligible), close to optimal in maintaining the trade-off between the utility and attack performance, and effective in the face of attack’s countermeasures.", "summary_of_the_review": "The approach is clear, and the extensive experiments performed on variety of datasets against reliable baselines proves the merit of the proposed defense algorithm. However, the intuition behind some assumptions and steps in the algorithm is not clear and authors should certainly discuss the limitations of their method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "cWcNH7NiZ5l", "reviewer": "Reviewer_A7eb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper proposed a relax loss to defend privacy leakage, since the authors find that membership privacy risks can be reduced by narrowing the gap between the loss distributions. Also the gradient descent and gradient ascent are used alternatively to balance the member and non-member loss distributions. \n\nWeakness:\n1. The authors give an analysis for \"RelaxLoss increases the variance of the training loss distribution\" in appendix A. In Theorem A.1, the authors show that the variance of loss distribution increases after a gradient ascent step, with the condition $\\text{Cov}(l, |\\Delta l|)>0$. My questions here are why we need the absolute operator. The variance of loss distribution after a gradient ascent step should be $\\text{Var} (l+\\Delta l)$.  In this case, $\\Delta l$ can be negative, and how to make sure the condition $\\text{Cov}(l, |\\Delta l|)>0$ still holds. Also, the assumption of \"each sample within a batch exhibits the same gradient\" seems too strong and impractical. The common assumptions may be: each sample within a batch exhibits the same norm of gradient. \n\n2. In Figure 5, why does 'w/o gradient ascent' only have a single point? It should be a curve just like 'ours' and 'w/o posterior flattening'.\n\n#############after authors' response#############\nThanks for the responses from the authors. I will raise my score to 8.\n", "summary_of_the_paper": "To defend the common membership inference attacks, this paper proposes a relaxed loss with a more achievable learning target to reduce the attack accuracy and also help with the generalization gap of learning models.  Extensive evaluations on five datasets with diverse modalities show that the proposed method can achieve higher membership inference protection.", "summary_of_the_review": "This paper proposes a relaxed loss to narrow the loss gap and reduce the distinguishability between the training and testing loss distributions, and further prevent the privacy leakage. However, I have some questions about the theoretical analysis, i.e., why the \"RelaxLoss increases the variance of the training loss distribution\". ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "04pZ5bUxxKT", "reviewer": "Reviewer_sYYW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[strengths]\n1. The two techniques – relaxed target loss with gradient ascent and confidence flattening are well-motivated in Section 4 and 5.\n2. The paper thoroughly evaluates the defense method on 5 datasets, and compares it with 8 defense baselines, making the experimental results very convincing.\n3. The consideration of adaptive attacks in Section 6.4 is necessary and important.\n\n[weaknesses]\n1. Section 6.4 only considers adaptive attacks for NN attack, can you run adaptive attacks for other attack methods as well and report the best attack success among all methods in Table 2 (or put results in another new table)?\n2. A small presentation issue: subfigures in Figure 3 are too small. Maybe put some of them in Appendix and enlarge the main subfigures?\n", "summary_of_the_paper": "The paper proposes a new training algorithm to defend against membership inference attacks (MIA) in machine learning models. Motivated by the connection between MIA success and difference between training and test loss distributions, the proposed algorithm sets a positive target mean training loss value and applies gradient ascent if the average loss of current training batch is smaller than it. Furthermore, to avoid hurting model accuracy, the proposed algorithm also flattens the probabilities among incorrect labels during training steps. Extensive results on multiple datasets along with several defense baselines validate the effectiveness of the proposed defense idea.", "summary_of_the_review": "Overall, I like this paper. The used techniques are well-motivated. Also, the experimental evaluation is very thorough to include several datasets, multiple existing defense baselines, and include adaptive attacks. So, I recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach and specific training algorithm to defend against membership inference attacks (MIA) in machine learning models. Existing MIA attacks are relatively simple and rely on the test loss distribution at the query point and therefore the proposed algorithm sets a positive target mean training loss value and applies gradient ascent if the average loss of current training batch is smaller than it (in addition to the standard gradient descent step). The submission gives extensive experimental results demonstrating advantage over existing defense methods on several benchmarks. The primary limitation of the work is that it defends only against rather naive existing attacks which do not examine the model (but rely only on the loss functions).
Decision: Accept (Spotlight)