Exploring Connections Between Memorization And Membership Inference
Authors: Jihye Choi
Varun Chandrasekaran
Shruti Tople
Somesh Jha
Venue: Submitted to ICLR 2023 (2023)

Abstract: Membership inference (MI) allows privacy adversaries to query trained machine learning models to infer if a particular data sample was used in model training. Prior work has shown that the efficacy of MI is not the same for every sample in the training dataset; they broadly attribute this behavior to various data properties such as distributional difference. However, systematically analyzing the reasons for such disparate behavior has received little attention. In this work, we investigate the cause for such a discrepancy, and observe that the reason is more subtle and fundamental. We first provide empirical insight that an MI adversary is very successful with those samples that are highly $\textit{likely to be memorized}$, irrespective of whether the sample is from the same or a different distribution. Next, we provide a game-based formulation which lower-bounds the advantage of an adversary with the ability to determine if a sample is memorized or not, under certain assumptions made about the efficacy of the model on the memorized samples. Finally, based on our theoretical results, we present a practical instantiation of a highly effective MI attack on memorized samples.

Under review as a conference paper at ICLR 2023
EXPLORING CONNECTIONS BETWEEN MEMORIZATION AND MEMBERSHIP INFERENCE
Anonymous authors Paper under double-bling review
ABSTRACT
Membership inference (MI) allows adversaries to query trained machine learning models to infer if particular data sample was used in training. Prior work has shown that the efficacy of MI is not the same for every sample in the training dataset; they broadly attribute this behavior to various data properties such distributional difference. However, systematically analyzing the reasons for such disparate behavior has received little attention. In this work, we investigate the cause for such discrepancy, and observe that the reason is more subtle and fundamental. We first provide empirical insight that an MI adversary is very successful with those samples that are highly likely to be memorized, irrespective of whether the sample from the same or a different distribution. Next, we provide game-based formulation which lower-bounds the advantage of an adversary with the ability to determine if a sample is memorized or not, under certain assumptions made about the efficacy of the model on the memorized samples. Finally, based on our theoretical results, we present a practical instantiation of a highly effective MI attack memorized samples.
INTRODUCTION
Advances in machine learning (ML) are enabling wide variety of new tasks that were previously deemed complex for computerized systems. These tasks are powered by models which are trained on large volumes of data, that often fall under the category of being sensitive or private, as it is collected from a variety of sources. For example, data used to customize (or fine-tune) large language models can often be sensitive (Carlini et al., 2021 Zanella- Béggllln et al. 2020). Hence, understanding and explaining the privacy risks of the data used to train these models is an important problem that need to be solved before widespread adoption of these models.
Several prior works (Shokri et al. 2017 Yeom et al., 2018) have successfully established that such models are susceptible privacy attacks, such as membership inference (MI), that aim to infer if specific data-points are used during their training. Even more concerning is that they have shown that the efficacy of MI is not the same for every sample in the training dataset. Unfortunately, the problem of explaining this discrepancy has received less attention. Only recently, researchers have proposed techniques to measure the susceptibility of attack per sample (Carlini et al.) [022aa] Ye et al. (022), and attributed the behaviour of disparate risks with coarse relation to distributional difference (Kulynych al. 2019). Consequently, -uitt--ditt (OOD) samples which are part of the training dataset were deemed to be at a higher risk compared to other samples.
In this work. we first systematically analyse the correctness of the above reasoning using repre- sentative techniques from OOD detection (Hendrycks & Gimpel 2016 Liu et al.) (020) and MI literature (Carlini et al. 2022a). Our empirical observations reveal that the relationship between OOD samples and higher MI risk is not straightforward ($ 3) The reasoning is more subtle and fundamental. We bridge the gap in our understanding of this problem and provide reasons for varying MI risk among training data-p.ints. We demonstrate that MI and OOD samples are connected via the susceptibility of memorization of these samples Feldman & Zhang (2020); Brown et al. (20TT. That is, we show that an adversary is highly successful in predicting the membership of those samples that are likely to be memorized, irrespective of the distribution from which the data-point is sampled. Moreover, as shown in previous work by Feldman (2020), and demonstrated in our evaluation, is

---

Under review as a conference paper at ICLR 2023
the OOD samples that have higher tendency of being memorized by the model, thereby misleading previous work in concluding that distribution distance increases the susceptibility to MI.
Next, formalize rization bounds from Brown et al. (202T) and the advantage of an MI adversary. We show that the ability of an adversary predict whether given sample is memorized or not lower-bounds the advantage of any adversary in predicting the membership of that sample ($ ( Lastly, we propose a practical instantiation of a highly effective MI strategy on memorized samples that can be performed with existing attacks. Extensive evaluation on image recognition models using four popular MI
base- definition for MI, and present connections between memo-
attacks (Yoom et al. 2018 [Shokri et al. 2017; Song Mittal 202T] | Carlini al., (022a) confirm that the susceptibility of MI risk is maximum, at times 100% (Advantage = 1) for memorized samples
2 BACKGROUND & RELATED WORK
Notation. The focus of our discussion is on supervised machine learning (ML). Here, one wishes learn trained model using data-points of the form 2 = (z, y), where E X is the space of inputs and y E y is the space of outputs. A distribution D captures the space of inputs and outputs, and a dataset D of size T is sampled from it (i.e., D ~ D")) such that D = { f=1 Using this dataset, a learning algorithm I. can create trained model 0 ~ L(D) by minimizing suitable objective L.
2.1 MEMBERSHIP INFERENCE ATTACKS
Membership inference attacks (MIAs) are training distribution and aim to infer if a particular sample z was present in the training dataset given access to model One common theory for why such an attack exists relies on the belief that models overfit to their training data (Ycom et al.)
2018), leading to numerous follow-up works aiming to infer the advantage of MIAs (in determining
membership) (Mahloujifar et al. 2022) Humphries al. 2020 Erlingsson al.. 2019 Jayaraman et al. 20200; Thudi et al.) [2022] Prior MIAs approach this problem by training shadow models
(i.e., models trained on different subsets of the training data) and observing their outputs to infer membership outcomes (Shokri et al.
2017). ). Recent work by Carlini et al. (2022a) employs a similar approach. and combines the shadow model training with hypothesis testing (also performed in prior work (Murakonda et al. | Ye et al. 2022)) to yield success. Their work, however, remarks that attacks are most representative low false positive rate (FPR) regimes. In our work, we will use the MIA of Carlini et al.] (2022a); more details are presented in Appendix B. Other approaches towards estimating MI success involve measuring entropy (Song & Mittal 2021).
2.2 OOD DETECTION
The goal of -fittt-diitributiin (OOD) detection is to determine if an input is drawn from the same distribution as the training dataset (i.e., in-distribution) or not. In empirical settings, OOD samples realized as data samples from different datasets (usually with disjoint label sets) (Nguyen et al. 2015 Ming al. 2022). Most prior work formalizes the detection problem by designing scoring function and applying a threshold. Performance is measured in terms of the false positive rate (FPR95) of OOD examples the threshold when true positive rate (TPR) of in-distribution examples is high as 95%, along with AUROC and AUPR. Starting with a bascline for OOD detection which uses maximum softmax probability (MSP) (Hendryeks & Gimpel| 2016). recent studies have designed various scoring functions based on the outputs from the final or penultimate layer (Liu et al. 2020
DeVesss & Taylor 2018), or a combination of different intermediate layers of DNN model (Lee et 2018 Raghuram et al. 2021).
2.3 LABEL MEMORIZATION
Label memorization, noted by Feldman & Zhang (2000), is a concept where ML models remember information from training samples (likely) entirely to ensure good performance (when evaluated on those similar samples). Brown et al. (2021) note that for certain types of inputs (which shall define soon), memorization is essential to create a performant model. We begin by defining certain salient concepts related memorization in their original work.
2

---

Under review as a conference paper at ICLR 2023
1. Label Memorization (Feldman & Zhang 2020): For training algorithm L on dataset D = { where 2, = (Tz, v4), the amount label memorization by L for any 2 E D is
mem L, D) = (t) = yi] Pio-LcDsy)((0:)) = wi
Since estimating the aforementioned quality is computationally prohibitive, Feldman Zhang (020)) utilize estimator based on sampling (refer Lemma 2.1 in their work). Such sampling campling-based based
estimators are also used in prior work related to influence (Ilyas et al. 2022. Kwon et al. 2021).
2. Singletons: As noted by Feldman (2020), datasets can be viewed as long-tailed mixtures of sub- populations. Models learnt from such datasets are unable to predict accurately on a sub- aub-population until they are trained at least on one (representative) sample from the uub-oopulation. Such sample is called a singleton. Singletons may come from rare (or outlier) sub- sub-populations or atypical sub- populations; in the former there is no accuracy impact, while the latter may significantly impact accuracy. To better illustrate the difference between rare and atypical, consider a dataset formed by the following equation: y 2x E where (a) y is quantized the closest element in N, (b) e is noise, and (c) is a normally distributed element in [0, I]. Those values of æ which lie in the tail of the distribution are atypical, but values of x > are outliers. However, models are unable to discriminate between outlier and atypical (i.e., of frequency 1) sub-populations; they memorize the singletons regardless. [Brown et al. (2021) note that if dataset contains n < singletons, each represented by d bits, performant model (i.e. one whose error is close to optimal) learnt using the dataset will memorize 2 S2(n - d) bits. We utilize this insight in motivating our claims.
3 MOTIVATION: NOT ALL SAMPLES ARE EQUAL
Our work is motivated by observations by Kulynych et al. 2019) and Carlini et al. (2022a) (which we recreate in Appendix D.3) who notice that the efficacy of MIAs is different for different sub-groups of data (within dataset). While prior work notes that MI efficacy is higher for those samples that are OOD, we wish verify if there are other characteristics of dataset that may result in this behavior.
Experimental Setup: We visualize this effect by considering the following setting: we consider different datasets (say Do and D and mix them using a mixing ratio (e.g., to obtain D" a DI + (1 a)- - Djj for different values of E [0, 1] In such settings, one dataset is an over-represented population (i.e., Dğ when the mixing co-efficient is greater than 0.5), and the other an under-represented population (i.e., Dfj.. Note that Dÿ and DV share the common set of labels. In our experiment, the over-represented population is fixed as entire train set of MNIST (LeCun) ct al. 20T0) (60,000 labeled samples), while the choice of under-represented population varies. We consider three variants: (a) MNIST augmented by |Hendrycks et al. 2022) (henceforth referred to as
augMNIST), (b) cropped SVHN (Netzer et al., (001)), and (c) CIFAR- 10 (Krizhessky 2009). We
choose these variants based on their distance from MNIST (Alvarez-Melis & Fusi 2020 larger the
distance, more atypical is the variant w.r.t the chosen over-represented population.
To construct Djj (for given variant), choose subset of samples (more details follow) from the corresponding training set; care is taken to ensure the consistent label assignment to both DY and
Dy All samples are (a) resized to match the dimension of MNIST samples (28 x 28 x 1), and (b) converted to grayscale We adopt CNN models from the work of Carlini et al. (0022a) (details in Appendix A.I) and ensure that the models are trained to exhibit good accuracy on both populations.
3.1 CONNECTIONS TO OOD DETECTION
As stated earlier, we wish to better understand the implication of the "nature" of sample on its susceptibility to MI. Similar to Carlini et al. (022a)). we hypothesize that outlier (i.e., OOD) samples are more susceptible. To this end, we choose random subset of samples from each of the aforementioned variants as our redreseepee population (Di) and train models as described
'That is, a [og] |Dg]+|DE
2 Consistent and balanced label assignment across classes; Le., all "airplane" images CIFAR-10 10 are labeled as class 0 along with digit "0" images of MNIST
'Gray Image- = 2989 R + 0.5870 x G 0.1140 B.
3

---

Under review as a conference paper at ICLR 2023
earlier. To achieve reasonable training and test error, need at least 1000 samples for augmented MNIST and cropped SVHN, and at least 6000 CIFAR-10 10 samples (10% of the total dataset); all these samples are chosen such that there are equal number of samples for each class (details in Table I..
In all the experiments, we compute the privacy score proposed by Carlini et al. (022a) which captures the vulnerability of sample against MIA (higher is more vulnerable). We then plot the privacy score
against the score returned by the two representative OOD detectors by |Hendrycks & Gimpel| (2016)
and Liu al. (0200) (higher is more likely be OOD) measure the correlation between the two
Over-represented
Under-represented
a
# Samples
Distance (x110)
Model
Train Acc. (%)
Test Ace. (%)
mngimented MXIST
0.99
1000
1:26
CNNEZ
99.15
4 0.13
89.10
9843 0.19
63.88 at 1:39
MNIST
cropped SVHN
0.99
1000
3.67
CNN32
99 44
0.05
90.07 + 0.66
98 60 + 0.09
64.76 + 1.31
CIFAR-10
0.93
6000
3,27
CSN64
99.27 0.86
82.69
6.84
98.18 0.47
40.72 1.17
Table 1: Salient features of the datasets used in our experiments. Distance represents the distance between Dÿ and DJ (larger is more). Accuracy reported in 95% confidence interval for each iteration in bootstrapping, we include 70% of D as the training set, and repeat 4000 times (as done in Feldman & Zhang (2020)). We also ensure that each sample in D appears in the training set for half of the 4000 iterations, following the setup of Carlini et al. (002a)).
Undie rommeeete
Undte repeoorrie
Hracy Score
Princee Store
05
0F
5.5
04
8T.
08
05
33
03
04
85
03
08
35
33
03
04
08
07
08
05
30
M5
MS7
Figure 1: Correlation between privacy score and OOD score. MNIST+aUgMNIST (b) Mid: CNN32 trained on MNIST+SVHN (c) Right: CNN64 trained on MNIST+CIFAR-10.
(a) Left: CNN32 trained on
Results: We capture correlations using the Spearman's 's rank correlation coefficient with p-value less than 0.001 for statistical significance; absolute values less than 0.4 are considered to represent weak correlation, absolute values between 0.4 and 0.59 are considered represent moderate correlation, and absolute values greater than 0.6 are considered to represent a strong correlation; the sign dictates whether the correlation is positive or not (which is not very useful for our discussion). From Figure [] we observe that there is no strong correlation between OOD scores (measured by the approach of Hendrycks & Gimpel (2016)) and MI success; experiments with other OOD detector (Liu) et al. (2200) show similar trends and are presented in Appendix [.2] This also suggests that one can not definitively use OOD scores to detect a sample is a member or not. While some of the points with high privacy score (i.e., highly susceptible to MI) are OOD points, others are not (this is particularly true in the case of Figure[(c), This suggests while the observations in prior work are true (to some degree), they are rather coarse and more refinement is needed.
There is only a weak correlation between samples that are OOD (as detected by an 0OD detector) and those that are very susceptible to MIAs.
3.2 CONNECTIONS TO (LABEL) MEMORIZATION
From the carlier results, we can see that not all points which are OOD are highly susceptible to MI. We wish understand properties of those points that are susceptible. To this end, we borrow insight from the memorization literature, and identify those samples (from the under-representeed
For the complete description of estimating privacy score and memorization degree, see Algorithm in Appendix B.
4

---

Under review as a conference paper at ICLR 2023
population) that have high label memorization values (> 0.8, calculated based on the definition in 2.31 We then measure the correlation between the memorization value and the privacy score. Note
that by doing so, reduce the size of the under-represented population (details in Table 2
Samples
Train Acc. (%)
Test Acc. (%)
Over-represented
Under-represented
Model
[D51
[DE]
17
DX
1%
D2
augnaented MINSS
600000
219
CNN32
95.27 0.14
72.08
138::000 3.04
MNIST
cropped SVHN
60,000
209
CNN32
99.32 0.10
71.40
1:
98.48
+: 0.20
19.67 1: 2.82
CLFAR-10
60,000
2153
CNN64
99.43 0
72.66 + L.66
98.28 0.31
20.78
1.215
Table 2: Salient features of the datasets used in our experiments. Accuracy reported in 95%
confidence interval over 4000 iterations in Table
Unterppoorrdd Ouer munited
thar ongor undtr
02
36
08
50
24
33
02
34
OR
Momorrtaiion Feldman
Momortzaiin
Momorizadinn Foldman
0
0
O
6 5
0
D
0
1
1
1
4
1
3
2
)
1
5
2
2
2
2
3
3
3
8
8
8
8
3
3
3
3
3
3
3
3
7
4
4
4
4
4 7
M
4
4
4
Y
Y
4
4
4
5
5
5
5
3
'
5
5
5
5
5
S
5
5
5
6
6
6
b
6
6
6
4 6
6
6
6
6
6
6
6
6
6
6
7
7
2
7
5
7
7
7
4
7
7
7
7
8
8
8
8
8
8
8
8
3
8
8
8
8
9
7
9
7
e
9
9
9
9
9
9
9
Figure 2: (a) Left: CNN32 trained on MNIST and augmented MNIST (b) Mid: CNN32 trained on
MNIST and SVHN (c) Right: CNN64 trained on MNIST+CIFAR-I0; (i) Top: Privacy score vs. memorization, (ii) Bottom: Random examples whose memorization value is close to 0 (first three columns), 0.5 (next three columns), (last three columns). Each row shows images belonging to each of 10 classes.
Results: From Table observe that the train accuracy of the samples from the under-represented pop- ulation is reasonable, and the test accuracy associated to points from that class is better than random guessing (but not by much); this suggests that memorization has happened to some degree (Feldman & &Zhhang] 2020; Brown et al. 2021). Several observations can be made from Figure [ First, the samples that have the high memorization values (i.e., the red points) roughly correspond to those samples which have high privacy score in Figure [ this suggests that samples with high memoriza- tion values are indeed those that are highly susceptible MIAs. Next, we visualize the examples of memorization values from each dataset (see the bottom of Figure 2 It confirms our intuition; the samples from the over-represented population fall into the owwmmmorrization regime, atypical samples from the over-represented population (and few similar samples from the under-represented population) compose the mid-level memorization regime, and the under-represented samples belong to the high- mesorizzoozznion regime. Finally, we observe that the correlation in this case is better; this suggests that memorization captures better signals regarding susceptibility to MIAs.
'We observe the privacy onion effect (Carlini et al. 2022b) when recalculate memorization values the
chosen subset. More details are presented Appendix D.2
5

---

Under review as a conference paper at ICLR 2023
Remark: Highly memorized samples (i.e., singletons) are also highly susceptible MIAs.
Thus, based on the experiments have conducted, conclude that there are connections between label memorization and MI susceptibility. In the next section, we formalize this connection through the form of MI game, and calculate the adversary's advantage of success.
FORMALIZING THE CONNECTION
In 3 we see that the memorized samples are also very susceptible to MI. In this section, we formally establish the connection between (high values of) memorization and MI success. We begin by formalizing new MI game in 441 and describe instantiations of the game that are both ineffective
($ 4.2) and effective (S
4.1 THE MI GAME
We extend the formulation of Mahloujifar et al. (0022) to introduce a game called MI A, m) between the challenger (which runs learning algorithm and an adversary A. where m parameter which controls how many additional samples A can use. Note that this generalization of games presented in the literature before; roughly speaking, previous games correspond to
Step 1.
The adversary A picks dataset D (21,'- 2T) and DA {2t 2
Step 2. The challenger samples a bit uniformly at random and creates
D' DUDA for b 1 D
for b 0
Step The challenger learns model by running I.(D") and sends to the adversary.
Step 4. The adversary guesses bit b' and wins if 6' =
The output of the game MT(L,, A. m) = 1 iff the adversary A wins. The advantage of adversary is defined to be Adv (L, m) = 2 (L, m) = 1] - 1, which can also written as 2 Pr(6 6'] 11- Given class of adversaries C, one can define Adv(L, m) as sup Adv Adv A, A, m). The following lemma holds in such cases.
Lemma: Adv 1)
Adv(L,m)
Proof: This result follows by a simple hybrid argument. Essentially the game MI(L, A, m) can be viewed series of m games where the adversary only adds additional point.
4.2 A SIMPLE (YET INEFFECTIVE) ATTACK
Consider the game described above, but for m Here, DA (z',, where =' (x, y) is arbitrary sample. The adversary A' utilizes the following strategy to determine (non-)membership:
1/ = [1 if = 0 if -
Such strategy is not uncommon; it has been evaluated by Yeom al. 2018) and (Song & Mittal 2021) and shown to be ineffective. In such setting, it is easy to show how Adv(L, A', 1) mem 2,) (refer Appendix C for the full derivation). However, note that this connection between memorization and MI is for (a) particular MI strategy (as defined above) and not all (or even the best) of them, and (b) for any arbitrary sample and not the highly memorized samples. In the subsections that follows, bridge this gap.

---

Under review as a conference paper at ICLR 2023
4.3 MORE EFFECTIVE ADVERSARIES
Consider the game described above, but where the adversary chooses DA s.t. has m points which all singletons through interaction with some memorization oracle O We modify the game this way for two reasons.
Reason 1: We define the empirical risk over dataset D' is defined as RD (0) (( [[(((z) y)]. As noted by Feldman 2000 RD (0 L(D)) Rp, L(D DA)) for DA (comprised of singletons). We refer to this as the memorization-induuee risk gap (MIRG). Note that such gap more prevalent when Da a contains m > points (as captured in our game definition). Thus, an adversary Aace can utilize this information, and its advantage is calculated follows:
Adv( L, Ance, m) = Pr[Rn,, (0 L(D U DA)) < <1 - Pr[RD, (0 = L(D)) < <nl
for some pre pre-determeeed risk threshold 7.
Reason 2: As noted in S 2.3 for inputs of size d. rell-generalized models (i.e., ones with low error) memorize f2(m d) bits. is clear to see for any adversary Amem that uses this information,
Pr Adv(L, Amem, m) (~I(DUDX))
(0(x)) y]
Pr (0(z) y G-I(D)
Both the aforementioned adversaries utilize (a) the fact that DA was chosen in such a manner so (b) exploit specific properties associated with memorization. Let us generalize the findings from above. For ease of exposition, let us consider the scenario where m = (and D = (=')). In such a setting, the aforementioned adversaries can be abstracted by an adversarial strategy Strat which when given access to model and singleton 2' returns the probability of the singleton being memorized by the model. Such a strategy is also a particular instantiation of an MIA (i.e., the probability is higher than some threshold, the sample is memorized and is consequently a member), and is lower bound for sep Adv(L, A, 1). Thus,
Strat 2') Ad)
Conjecture: If z' is a singleton and is a well generalized model, there always exists a strategy Strat 2') 1.
The consequence of the above conjecture that there exists an effective test for checking if singleton is memorized, then it can be used for highly successful MIA. We use the term "if" because the mutual information bounds presented the work of Brown al. (202T) are a lower bound of the number of bits that are actually memorized.
We believe the aforementioned conjecture true because both Amem and Ance are abstractions for Strat (as noted earlier). When MIRG is true, then there exists a y such that Adv A Ancey m) = Pr[RD, (0 = L(D U DA)) < 7 - Pr[RD, (0 = L(D) < > + 1. Similarly, if D is fully memorized, then Pro-L(DUD, [0(z) y) LID-L(D)) ((<z) y) 1. We present more evidence in Appendix D.I Formally proving the aforementioned conjecture remains an open question.
However, to ensure practicality, two considerations remain. First, how does instantiate O (needed for choosing A Effectively designing such an oracle is crucial for the success a memorization-aware MIA. Next, and more importantly, how does one design test for memorization (i.e., instantiate the best Strat:)? Prior work related to label memorization aims to estimate the propensity with which sample is likely to be memorized by model during the training process. However, we require test that tells us the likelihood of a particular sample being memorized in the parameters of trained model. In the section that follows, we answer these questions.
MI ATTACKS ON HIGHLY MEMORIZED SAMPLES
Having discussed the connections between memorization and MI advantage, we wish to understand if they hold in practice. this section, we carry out experiments address the following question: are singletons highly vuinerable to MIAs? In [. we describe the setup of our experiments. In $5.2 we present results that affirmatively answers the aforementioned question.
'Refer Section 2.5 and Corollary 4.6 in (Feldman 2020)

---

Under review as a conference paper at ICLR 2023
5.1 EXPERIMENTAL SETUP
Goal. We wish to understand the efficacy of MIAs is higher on (a) OOD samples which are also singletons (i.e,, samples which are highly likely be memorized) compared to (b) mixture of different data-points (i.e,, singletons and samples from an over-represented population). We want to compare (a) with (c) an under-represented population filled with randomly chosen OOD samples.
MIAs. We consider four representative attacks from literature (Yoom et al. 2018 Shokri et al. 2017 Song & Mittal 2021 Carlini et al. 2022a) (see Appendix f.. parameters associated with each attack). As noted in | Carlini et al. (2022a) suggest that more emphasis should be placed on whether an adversary can reliably violate the privacy of even few samples; this is captured through the TPR at low FPR. Following this idea, for each attack, we report TPR when decision threshold chosen to maximize TPR at target FPR (e.g,, 8,, as low as 1% in our setting) along with AUROC values (i.e., the probability that a positive example would have higher value than negative one).
Datasets and Models. Similar to our setup in $ we evaluate MIAs against ML models trained on dataset D that is mixture of two different populations. We set DĀ as MNIST (60,000 samples) throughout the experiment. We then consider two ways for an adversary to instantiate Omem to
obtain Dif: (a) choosing random samples which are OOD (e.g. 1000 MNIST samples augmented by Hendrycks et al.) (2022). 1000 SVHN samples, or 6000 CIFAR-10 10 samples) as in $ [31 or (b) using samples that OOD but also highly memorized (i.e. singletons whose memorization degree are approximated by the approach of Feldman & Zhang (2020), and are above 0.8). As done earlier, care was taken ensure consistent labeling. We use the same models as earlier, and ensure that the accuracy (both test and train) on both the under and over-represented populations is reasonable.
AUROC
TPR @ 0.1% FPR Under-represented
Method
Dataset (D")
All
Under-sepresented
All
MNIIST+augMNIST (random)
0.51
0.65
0.0%
0.0%
MNIST=augMNIST (singletons)
0.50
0.91
0.0%
0.0%
MNIST+SVHN (random)
0.50
0.74
0,0%
0.0%
Ycom ct al. 2018
MNIST-SVIIN (singletons)
0.50
0.95
0.0%
2.61 %
MNIST-CIFAR-I (random)
0.51
0.81
0.0%
0.0%%
MNIST-CIIFAR 10 (singletons)
0.50
0.96
0.0%
0.0%
MNIST+augMNIS (random)
0.51
0.72
0.20%
0.70%
MNIST-augMNIST (singlctons)
0.51
0.99
0.15%
18.88 %
MNIST-SVHN (random)
0.52
0.78
0.19%
3.01 %
Shokri 2017
MNIST+SVHN (singletons)
0.51
0.98
0. 1.14%
15.69 %
MNIST-CIFAR -0 (random)
0.58
0.89
0.82%
2.49%
MNIST-CIFAR- (singlctons)
0.55
0.99
1.92%
17.21 %
MNIST+aUgMNNITT (random)
0.51
0.70
0.16%
0.53%
MNIST=augMNIST (singletons)
0.51
0.98
0.10%
16.12%
MNIST+SVHN (random)
0.50
0.77
0.17%
2.86%
Song & Mittal 2021
MNIST-SVIIN (singletons)
0.51
0.98
2.04%
14.11 %
MNIST-CIFAR 1 (random)
0.54
0.87
0.20%
2.066%
MNIST+CIFFAR 10 (singletons)
0.53
0.99
1.98%
15.53%
MNIST=augMNNIT (random)
0.53
0.87
1.45%
27.35%
MNIST-augMNIST (singlctons)
0.53
1.0
1.07%
100 %
MNIST+SVHN (random)
0.53
0.95
2.15%
51.5%
Carlini et al. 2022a
MNIST-SVHN (singletons)
0.53
1.0
1.24%
100 %
MNIST-CIFARR- (random)
0.59
0.96
5.7%
44.2%%
MNIST+CIFAR-- (singletons)
0.56
1.0
4.6%
96.5 %
Table 3: Comparison of representative MIAs. For each attack, we evalutate the performance on three different mixture of datasets: MNIST with (a) augmented MNIST generated by the technique of Hendrycks al. (2022), (b) SVHN, and (c) CIFAR-10. For each choice of MIA and mixture dataset, we direct readers to compare AUROC and TPR between (a) (All) vs. (Under-representedd) and (b) (random) vs. (singletons) to see the effect of utilizing singletons in MIAs. Best results are boldfaced.
5.2 RESULTS
The results of our experiments are presented in Table Overall, the attack of Carlini al. (2022a) outperforms other attacks in the low FPR regime (0.1%), as illustrated in their paper (also explaining the overall poor performance of the attack of Yeom al. 2018
8

---

Under review as a conference paper at ICLR 2023
We observe that every attack performs better with respect the samples in under- rnder-represented population, which are drawn from different population than the over-represented population (i.e. MNIST samples). For instance, given MNIST+SVHN (random), the attack of [Shokri et al. (2017) (the earliest known technique) yields an overall AUROC of 0.52 and TPR of 0. 0.19% when evaluated all samples. However, we see increase in the AUROC to 0.78 and TPR to 3.01 % when the attack is evaluated on under-represented SVHN samples.
The same observation applies when attacks are tested on highly memorized OOD samples. Con- sider the case of MNIST+CIFAR-II (singletons): by utilizing the highly memorized samples, the attack of (Song & Mittal (2021) can improve its efficacy from AUROC=0.58 and TPR=1.98%) to AUROC=0.96 and 1PR=15.53 This is even higher than when the same attack is evaluated on just (random) OOD samples (i.e. AUROC=0.87 and TPR=2.06%).
It is noteworthy that the attack of Carlini et al. (2022a) achieves near perrfct AUROC and TPR even the low-FPR regime on the reder-represented population comprising of highly memorized OOD samples in any of three dataset mixture settings. We believe this high accuracy is attributed to the commonalities between the attack and memorization estimation (refer Appendix B.2) i.e., the attack is estimating memorization as well.
Result: With the knowledge of memorization, performance of MIAs improves significantly.
DISCUSSION
1. The Privacy Onion Effect:The definition of label memorization is implicitly dependent on the dataset (refer $ (23)). In recent work, Carlini et al. (20226) note that if a set of highly memorized points is removed from dataset, points which previously had low memorization values now have high values (and vice versa). They term this the privacy onion effect (and we observe the same in Figure [ (bottom row) in Appendix D.2 While defenders against memorization-guee -ugiiodd MIAs may consider removing those samples that are likely to be memorized from the training dataset, two problems may emerge. The first is that the error of the model (on the sub-population associated with the deleted singleton) shoots up. Secondly, and more importantly, there will be new set of points which are highly likely to be memorized. This suggests that attackers will always win.
2. Connections to Privacy Auditing: When training models with differential privacy (DP) (Abadi jet al. 2016), the estimate for the privacy budget (e) obtained is very loose. To obtain tighter estimates, rich body of literature (termed privacy auditing) aims at empirically estimating (Nasr et al. 2021 Ganclla-Béguehle et al., 2022 Jagielski et al. 2020). The core technique used in most of these works to calculate estimate of using the DP-trained models susceptibility to highly-effecttvv MIAs (which are typically designed for any arbitrary data-point). We believe that better estimates can be obtained when auditing is performed using memorization-gue guided MIAs. A key consideration is to understand the implications of oratriini on memorization of data-points,
3. Data Reconstruction: Brown et al. (02I)) discuss approaches reconstruct data from memorized models. For a given sample z' and a trained model 0, such approaches may also be used to instantiate Strat 0); if reconstruction is successful, we can claim that the sample is memorized. Based on the results presented in $ 5 of (Brown et al., 2021). } we can see that such a Strat + 1, providing more motivation for the correctness of our conjecture.
CONCLUSIIO
In this work, we attempt explain the high efficacy of MIAS for specific data-points as a function of their susceptibility to be memorized. Our theoretical justification provides motivation to our claims, and notes that the efficacy of attacks on such samples can be very high (i.e., almost always accurate). We also experimentally verify our claims, and the results are in accordance with our expectations. More analysis required to (a) convert the claims associated with mutual information from Brown et al. (2021) to probability bounds (needed to insantiate attack strategies). and (b) understand the efficacy of MI for samples that are unlikely to be memorized.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428-21439) 2020.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-aceuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium Theory of Computing, pp. 123-132, 2021.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022a.
Nicholas Carlini, Matthew Jagielski, Nicolas Papernot, Andreas Terzis, Florian Tramer, and Chiyuan Zhang. The privacy onion effect: Memorization is relative. arXiv preprint arXiv:2206.1042 2022b.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint /802 2018.
Ülfar Erlingsson, Ilya Mironov, Ananth Raghunathan, and Shuang Song. That which we call private. arXiv preprint arX8vv/908.000
2019.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1610.02 2016.
Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16783-16792, 2022.
Thomas Humphries, Matthew Rafuse, Lindsey Tulloch, Simon Oya, Ian Goldberg, Urs Hengartner, and Florian Kerschbaum. Differentially private learning does not bound membership inference. arXiv preprint arXiv:2010. 12/11 2020.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamod- cls: Understanding predictions with data and data with predictions. International Conference on Machine Learning, pp. 9525-9587 PMLR, 2022.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33: 22205-22216 2020.
Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer. Quanquan Gu. and David Evans. Re- visiting membership inference under realistic assumptions. arXiv preprint 200. 2020.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
10

---

Under review as a conference paper at ICLR 2023
Bogdan Kulynych, Mohammad Yaghini, Giovanni Cherubin, Michael Veale, and Carmela Troncoso. Disparate vulnerability to membership inference attacks. arXiv e-prints, pp. 1eXiv-1906. 2019.
Yongchan Kwon, Manuel A Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In International Conference Artificial Intelligence and Statistics, pp. 793-801. PMLR, 2021.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecnn.cm////
2, 2010.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributio samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33, 2020.
Saeed Mahloujifar, Alexandre Sablayrolles, Graham Cormode, and Somesh Jha. Optimal membership inference bounds for adaptive composition of sampled gaussian mechanisms. arXiv preprint arXiv:2204.00 06 00600,620000 2022.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into ood detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022.
Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statistics, pp. 2287-2295. PMLR, 2021.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866 882. IEEE, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Jayaram Raghuram, Varun Chandrasekaran, Somesh Jha, and Suman Banerjee. A general framework for detecting anomalous inputs to dnn classifiers. In International Conference on Machine Learning, pp. 8764-8775. PMLR, 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, pp. 3-18, 2017. doi: 10. 1..1..........
Liwei Song and Prateck Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pp. 2615-2632, 2021.
Anvith Thudi, Ilia Shumailov, Franziska Boenisch, and Nicolas Papernot. Bounding membership inference. arXiv preprint arXiv:2202.122333 2022.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced membership inference attacks against machine learning models. CCS, 2022.
Samuel Yeom, Irene Giacomell,, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268-282 IEEF, 2018.
Santiago Zanella-Be -éggulln,, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Securit, pp. 313-3333775 2020.

---

Under review as a conference paper at ICLR 2023
Santiago Zanella-Bégee Béguelin, Lukas Wutschitz, Shruti Tople, Ahmed Salem, Victor Rühle, Andrew Paverd, Mohammad Naseri, and Boris Köpf. Bayesian estimation of differential privacy. arXiv preprint arXi::2206.055: 200::0999.00099 2022.
12

---

Under review as a conference paper at ICLR 2023
Appendix
A IMPLEMENTATION DETAILS
We run all experiments with Tensorflow, Keras, Jax and NVDIA GeForce RTX 2080Ti GPUs.
A.1 EXPERIMENTAL SETTING.
Datasets. Here we describe the datasets used throughout our paper.
1. MNIST (LeCun et al. 2010). The MNIST dataset consists of 60,000 handwritten digits (between 0 to 9) for train set, and 10,000 images for test set.
2. augMNIST. We generate the augmented variation the MNIST dataset by using the technique in Hendrycks al. (022). Original MNIST image is mixed with augmented versions itself and the grayseale version of provided 14248 fractal images. We use severity level of 2 for mixing, and severity level for base augmentation operations (such as normalization, random cropping, and rotation) for 4 iterations for each image. We use 1,000 images randomly chosen from the augmented train set, while ensuring the balance across 10 classes.
3.
SVHN (Netzer et al. 2011). SVHN is the color images of real world house numbers. We use the cropped version of the SVHN dataset, where images tightly cropped around each digit of 0-9. From the original train/test set, we randomly select 1,000 images that are equally balanced across 10 labels.
4. CIFAR-10 (Krizhevsky 2009). We use randomly sampled 6,000 images from the original train set of 60,000 CIFAR-1O images. We preserve their original labels so that images for class of CIFAR-I0 where E [0, 9] are assigned the same class as the MNIST images with label i.
Models. We adopt CNN architectures from Carlini al. (022a): CNN models with 32 and 64 convolutional filters (referred to CNN32 CNN64, respectively). We use them when training shadow models or target models for MIAs.
MIAs. We briefly describle the details of different membership inference attack methods.
1. Yeom et al. (2018). The attack is based on the observation that ML models are trained to minimize the loss of the training samples, and the loss values on them are more likely to be lower than the samples outside of the training set. They suggest applying threshold on the loss values from the MI. model to infer the membership of an input.
2. [Shokri et al. (017)- The attack uses trained MI. model to ascertain membership/non- membership. In experiments, fully-connecte neural network with one hidden layer size with ReLU (rectifier linear units) activation functions and SoftMax layer is used to distinguish feature vectors obtained from shadow models trained with and without data-pot Maiatp
3. Song & Mittal (021). The attack uses shadow models to approximate the distributions of entropy values, instead of cross-entropy loss. Given target model and sample, they conduct hypothesis test between the member and non-member distributions for each class.
4. Carlini et al. (002a)). Refer Algorithml in Appendix B.I provided by the authors for the attack.
We used the implementation
B DISCUSSION ON OOD DETECTION AND MI
B. ALGORITHM FOR MIA AND MEMORIZATION
We describe the algorithm proposed by Carlimi et al. (2022a) for MI (reproduced from their work), and Feldman & Zhang 2020) for label memorization below. Observe that both the MI success and
13

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Algorithm for estimating memorization and privacy score.
Require: model f. example 2 (z, y), data distribution D
1: confsin = 0
2: confspur = 0
3: Fm {}
4: Four = {
5:
for N times do
6:
Dattack & D
Sample a shudow dataset D train IN model
7: Om f- L(Dumeck U v))
8: Fi f- Fi U {0mm)
9: lour + L(Damnok ((z, v)}
b train OUT model
10: Four f Fout U (fout)
1: confs confs U (0((0((2))))
2: confsout + confsou U ()((am())g))
3: end for
14: Pin < mean (confs
15: Mout mean (confssonl
16: 2 < var (confon))
ouu f- var(confseu)
18: mem (A,, S, =) == Pro.er. (0)n(x) - E.. (0mm(() = D]
return
|A Poal and mem S)
label memorization can be obtained from the same algorithm by saving some additional state. As noted earlier, for a single sample, the algorithm requires 2 N training runs, making it prohibitively expensive for large values of N. For T samples, it requires 2 T N training runs.
B.2 COMMON SUBROUTINES
Notice that the privacy score estimation relies on training numerous models, some with the point under consideration, and some without (refer the grey box in Algorithm []. We refer to this as the leave-one-out (LOO) subroutine.
Observe that the exact (which is also used to measure algorithmic stability and influence). The attack by Carlini et al.) (202aa) couples this with hypothesis testing. leading high success rates for all samples. However, restrict our focus to those samples which are highly likely to be memorized, we can observe that the generalization error of models without these samples not close to optimal (Feldman) 2020).
I.O subroutine can be used to empirically estimate memorization
C PROOF FOR APPROACH IN
4.2
Adv(L, A,1) 2- Pr(p = bj -
2- (Pr(bb 1] 1b -
0.b 0b 0]}
Pr[b/ 110 L(DU=))++PP) - 0/0 000000(~00-1
Pr[b' 110 (1- = 1|00'=10~~00000 L(D)))
Pr 10(x) = v] - (~LLD))
Pr 0(x)= =
)~L(DUx))
mem D'='' =(x,8)
14

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL RESULTS
D.1 EVIDENCE: DISTRIBUTION OF SOFTMAX SCORES
model returns a vector when fed an input z, where ((x)); corresponds to the 1th element of the prediction vector. The maximum softmax score is defined as max; 0(x),
For the experiment that follows, we consider a dataset which is mixture of MNIST (over-representeed population) and singletons obtained from CIFAR-10 (under-represenneed population). In Figure 3 we plot 3 = max, (x), max, (x)); where 8. is obtained when the model is trained with the sample (a, y) and Putt obtained when the model is trained without the sample (I, y). Intuitively, 3 will be close to when the presence or absence of the model does not significantly impact the model's performance (i.e,, max; ii (z)); max; Ouu (x))))))) (x)) However, 8 O when the sample's presencelabsence influence performance (i.e., max, 0in(2) >> max, Oorr (z):). Observe that for the under-representee population, 3 > An adversary can exploit this information (i.e., decrease in model's 's performance) to perform MI.
repessereeddd
002
Difference
Figure 3: Distribution of softmax score gap. Observe that the score gap of the under-represented populations is larger than that of the over-represented populations.
D.2 ADDITIONAL RESULTS FROM 3
Figure 4 conveys the same information as Figure I but when the OOD-nes of a sample is calculated using the approach of Liu et al. (202)). Observe that there is low correlation between the OOD-ness of sample and its susceptibility to MIAs.
Couumin Commatton 0211
Speaman Cene M7bn-02477
Speeman Commis Co-
Uscempreeura
Unterpproerod
Undareproosee
-10
-20
-24 -10
Figure 4: (a) Left: CNN32 trained on MNNST+ (b) Mid: CNN32 trained on MNIST+SVHN (c) Right: CNN64 trained on MNIST+CIFAR-10 For the OOD detector by Liu et al. 2020), higher the score, more likely a sample to be OOD.
To obtain the red points in Figure 2 we first choose those samples from Figure which are from the redreseepee population and plot their memorization values (which are presented in the top row of Figure 5) Next, consider those red points whose memorization value is greater than 0.8 (from the top row of Figure 55) and include them in a dataset of over- re-calculate the memorization of those samples (to obtain the bottom row in Figure 5) Figure is
MNIST samples and
15

---

Under review as a conference paper at ICLR 2023
obtained by choosing those red points whose memorization value is greater than 0.8 in the bottom row Figure
our cosetted
Der en moniee
Dersiggrsennn
Merrottoaton by Enieman ef al
Hemortaaitnn by Feldman et
Hemortoniinn by Feidman et al.
Spnaman du1S2
Untermpeeeee boer perrepaat
Uuar stantte
Uobar reprerrrdd
Merrestoriinn Enidman
Memostowiin Enidman
Memorttriion Enidman
3.5
Figure 5: (a) Left: CNN32 trained MNIST+augMNIST (b) Mid: CNN32 trained
MNIST+SVHN (c) Right: CNN64 trained on MNIST+CIFARR-1,
Top Row: Memorization values of under-represented samples vs. privacy score.
Bottom Row: Recalibrated memorization values of under-represented samples from the top row
(whose original memorization value was greater than 0.8) vs. privacy score.
D.3 ADDITIONAL RESULTS: MOTIVATION
Figure 6 is our recreation of Figure 13 from Carlini et al. (2022a), which provides observation that OOD samples are more susceptible to MI than others. is obtained using a mixture of MNIST and CIFAR-10 (as the under-represented population). In Figure 6 see that the privacy score separated (into normal distributions) when the data for the under-represented population is comprised of singletons. This distinction is not clear when the under-representee population's data are not singletons.
800
represe
Over resresented
Uncer-iepresenttd
Un Undes-reoresen peeeaartdd
700
600
Frequency
500
Frequency
400-
300
200
100
0
1.5
Privacy
Privacy
Figure 6: Singletons are more effective than random OOD samples in enlarging the privacy score gap between data populations. Left: MIIST-CIFAR-10 (random), Right: MIIST+CIFAR-10 (singleton)
16


Reviews:
[{"review_id": "kWQ2zi3gTNA", "reviewer": "Reviewer_HK7T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study focuses on the existence of samples in which membership inference is more likely to succeed and samples in which it is less likely to succeed and discusses the causes of this difference. It was generally believed that out-of-distribution samples were more likely to succeed in membership inference, but the study argues that whether the sample is in-distribution or out-of-distribution has little to do with whether the sample is in-distribution or out-of-distribution. Instead, they argue that the larger the gap between the probability that a sample is identified as the target label when trained on a training data set that contains the target sample and the data set that does not, the more likely it is that membership inference will be successful. In addition, theoretical and experimental considerations regarding this hypothesis are provided.\n\n\n", "summary_of_the_review": "The author's argument could not be fully understood because it either assumes knowledge and understanding of existing research and assumptions contained in them or because there are gaps in the logic. The paper's claims must be structured in such a way that they can be understood on their own.\n\nPage 4\nThe explanation of the privacy score is too simple.\n\nPage 5\nWhy do you only include samples with large memorization values in your attempt to correlate memorization and privacy scores? Isn't it impossible to evaluate the correlation unless we also include samples with small memorization values?\n\nFigure 2, bottom\nHow were these image samples selected? Are they not cherry-picked?\n\nPage 6\nHow were samples in D_A  drawn? Sampled from out-of-distribution?\n\nProof of Lemma\nWhat does the hybrid argument mean? \n\nPage 7\nIn Reason 1, do you assume \\theta is optimal?\nIn Reason 2, do you assume the training procedure is probabilistic?\n\n\n\n\n\n", "strength_and_weaknesses": "Strength\nTheoretical considerations on the success of membership inference.\n\nWeakness\nContains unexplained or implicit assumptions\nContains gaps in the logic. Due to this gap, it s difficult to examine whether it is making a valid argument adequately", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "alrSG6GW6z", "reviewer": "Reviewer_j9KE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the relationship between label memorization and membership inference. It showed that examples with high memorization are more easily attacked. Based on this, it formulated membership inference attack where the attacker choose highly memorized examples, and show that the attack performance improved significantly.", "summary_of_the_review": "This paper studies the relationship between memorization and membership inference. My main concern over this paper is that the main empirical observations are essentially known, and the proposed MIA that focus on attacking the memorized examples is just a straightforward implication of the observation applied to existing MI attacks, and does not seem to bring new insights to the table.\n\n---------------------\nPost rebuttal: Thanks to the authors for the responses. After reading the response and other reviewers' comments, I'm keeping my current rating.", "strength_and_weaknesses": "**Strength**: This paper systematically studied the relationship between memorization and MI attack performance, and also evaluated common OOD detection algorithms.\n\n**Weakness**\n\n1. The connection between label memorization and membership inference attack performance is known. The attack algorithm that this paper primarily use (Carlini et al. 2022a) is directly built on this relationship: it performs hypothesis testing based on the distribution of two group of shadow models to make the MIA prediction. Since the (practical estimator of the) label memorization is just the difference between the mean of the two distributions, the larger the difference between the two distributions, the easier the hypothesis testing is, and subsequently the higher the MIA performance is. So this observation, while not formally characterized yet, is essentially known in the literature.\n\n2. This paper claims that OOD is not well correlated with MIA performance. But this is not well supported, and the main reason for the observation seems to be that the OOD detector used in the paper are not a good one. From Fig. 1, x-axis, we can see that the OOD score failed to make a clear separation between the over-represented data and under-represented (out of distribution) data.\n\n3. The paper propose to perform MIA in a MI Game setting where the attacker choose the points being attacked according to the susceptability of them being memorized. This ended up making the MIA performance higher. I'm not an expert in security, but showing that the attackers could have high success rate when they choose to only attack the most outlier examples does not seem surprising or concerning, comparing to, e.g., showing that an attacker can achieve high rate on average across randomly sampled training examples.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BCpgurgRax", "reviewer": "Reviewer_eP6o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the connection between label memorization and membership inference.\nThe paper shows that examples that are considered OOD according to some empirical metrics may not be more vulnerable to MI. Instead, examples with large label memorization (as defined by Feldman) are more likely to be vulnerable to MI attacks.", "summary_of_the_review": "The paper's main claim is that OOD points are not more susceptible to MI, but this depends entirely how one defines \"OOD\". In fact, an entirely natural way of defining OOD data is simply by using the result of an MI attack.\nWhat the paper actually shows (in Figure 1) is that some canonical OOD metrics (like MSP) are not aligned at all with privacy scores from MI attacks. But this can be interpreted in two ways: either privacy scores aren't impacted much by OOD data, or MSP is not actually a very good OOD metric!\nLooking at Figure 1, it seems that the latter conclusion is most at \"fault\" here: indeed, in this experiment you have ground truth on what data is OOD (i.e., the under-represented data). As we can see, MSP is terrible at distinguishing the under-represented and the over-represented data. So it seems like a poor OOD metric.\nIn contrast, the privacy score appears to distinguish the under- and over- represented data a lot better.\nThere is a prior work by Carlini et al (https://arxiv.org/abs/1910.13427) that looks at the agreement between different measures of what constitutes an outlier, including privacy metrics. It would be worth discussing the relation of this work to that paper.\n\nAs figure 2 shows, the privacy score of Carlini et al and the memorization score of Feldman are more closely correlated. This is not particularly surprising though: from the definition of label memorization (which should be credited to Feldman 2020, rather than Feldman & Zhang), it is obvious that high memorization of a given point implies the ability to do membership inference on that point well.\nIn fact, the two metrics are essentially measuring the same thing, just at different levels of granularity: the score of Carlini et al. measures the distinguishability of the loss distributions when training on D vs D \\ {i}. Label memorization takes a much coarser approach by just collapsing these distributions to a single scalar, the probability of outputting the correct label. Through this view, it is not clear why it should be surprising that these metrics are very strongly correlated.\n\nDue to this very strong correlation, the results from section 5 also seem somewhat circular. Of course, if we focus the MI attacks on those samples where the label probabilities for members and non-members are most different, then the MI attack will work better.\n\nSo overall, it is not clear to me what conclusions to draw from this paper. There are some interesting results that seem to show that MI attacks are better at identifying OOD data compared to other metrics. But these results are not at all interpreted this way, and rather taken to imply that prior work somehow drew incorrect conclusions regarding the success of MI attacks on OOD data.\n", "strength_and_weaknesses": "Strengths:\n- understanding what makes data vulnerable to privacy attacks is an important problem\n- comparing different ways of \"defining\" OOD data is valuable\n\n\nWeaknesses:\n- Very imprecise. Terms like \"singleton\" or \"OOD\" are never formally defined, and the conjecture in Section 4.3 and its consequences are stated in terms vague enough that the conjecture cannot meaningfully be falsified\n- Circular reasoning: ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "66gEsQbfpKs", "reviewer": "Reviewer_zop7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies in what situation training samples are vulnerable to membership inference attacks.\nFirstly, the authors empirically argue that membership inference performance may have a weak connection with the OOD property of training data.\nThen, they hypothesize that membership inference advantages may be related to how well the samples are memorized by models.\nThrough experiments, the authors find a clear correlation between the label memorization score of data and the corresponding membership inference attack accuracy, which justifies their claim.", "summary_of_the_review": "This paper studies an important and interesting problem, i.e., the relationship between data memorization and membership inference advantages. In general, the empirical results are strong enough to justify most of the claims in the paper. Therefore, I tend to accept this paper.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is written clearly and the studied topic is important and interesting.\n\n2. The empirical results are strong enough (especially Fig. 1 and Fig. 2) to justify the correlation between label memorization and membership inference.\n\n\nConcerns & Questions:\n\n1. According to the definition of label memorization score, one needs to access the learning algorithm to calculate such a score. However, in real-world attack scenarios, the learning algorithm (as well as hyperparameters) of a black-box model may be agnostic by adversaries, which would result in difficulties in obtaining memorization scores. Please discuss the practicality of leveraging memorization scores in membership inference attacks.\n\n2. Why the conjecture in Section 4.3 is appropriate? Please justify.\n\n3. In Table 1, every model is trained with MNIST and a subset of under-represented data. However, from Table 2, it seems that every model is instead trained with MNIST and the singletons from the under-represented data (which results in the size of the under-represented population being reduced). I think the comparison between Table 1 and Table 2 is unfair, and a more fair comparison may be that: (1) train model on MNIST + subset of under-represented data, and (2) compare the classification accuracy of the model on random under-represented data and under-represented singletons. Please comment.\n\n4. Only MNIST is used as the over-represented dataset. I suggest using more datasets as the over-represented datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject