How much of my dataset did you use? Quantitative Data Usage Inference in Machine Learning
Authors: Yao Tong
Jiayuan Ye
Sajjad Zarifzadeh
Reza Shokri
Venue: ICLR 2025 Oral (2025)
Keywords: Machine Learning
Privacy
Dataset Usage Inference
Dataset Ownership
Membership Inference Attack
Dataset Copyright
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: The first method to quantitatively and non-binarily answer the question ``How much has a dataset been used in the training of a given model?''

Abstract: How much of my data was used to train a machine learning model? This is a critical question for data owners assessing the risk of unauthorized usage of their data to train models. However, previous work mistakenly treats this as a binary problem—inferring whether all-or-none or any-or-none of the data was used—which is fragile when faced with real, non-binary data usage risks. To address this, we propose a fine-grained analysis called Dataset Usage Cardinality Inference (DUCI), which estimates the exact proportion of data used. Our algorithm, leveraging debiased membership guesses, matches the performance of the optimal MLE approach (with a maximum error <0.1) but with significantly lower (e.g., $300 \times$ less) computational cost.

Published as a conference paper at ICLR 2025
How MUCH OF MY DATASET DID YOU USE? QUANTITATIVE DATA USAGE INFERENCE IN MACHINE LEARNING
Yao Tong, Jiayuan Ye; Sajjad Zarifzadeh, Reza Shokri Department of Computer Science, National University of Singapore {yactong, jiayuan, s. s.zarif, reza]@come comp. nus. edu. ag
ABSTRACT
How much of my data was used to train a machine learning model? This is critical question for data owners assessing the risk of unauthorized usage of their data to train models. However, previous work mistakenly treats this as a binary problem problem-inferring whether all-or-none or any-or-none of the data used-which is fragile when faced with real, non-binary data usage risks. To address this, we propose fine-grained analysis called Dataset Usage Cardinality Inference (DUCI), which estimates the exact proportion of data used. Our algo- rithm, leveraging debiased membership guesses, matches the performance of the optimal MLE approach (with maximum error < 0.1) but with significantly lower (e.g., 300x less) computational cost
INTRODUCTION
The increasing legal conflicts over the unauthorized use of datasets to train artificial intelligence models (e.g., the New York Times' lawsuit against OpenAI (The New York Times Company 2023)
signal the increasingly critical issues about the boundary between intellectual property infringement
and fair use, well as compliance with data privacy regulations (e.g.. GDPR (European Parhament
and Council of the European Union, 2016) and CCPA (California Legislature 2018)). This has driven extensive research into data provenance and dataset ownership verification machine learning (Li
et al. 2023 2020; Tang et al. 2023 Li ct al. 2022b Sun et al. 2022 Sablayrolles et al. 2020 Hu
al. 2022 Zou al. 2022 Wenger al. 2022 Maini al.) 2021 Song & Shmatikov 2019)).
primarily focusing on implanting backdoor into target models by adding poisoning data to the protected dataset (Wei et al., [2024] Li al. [023] 20226 Tang et al.) 2023) or thresholding average
statistics over the dataset to conduct hypothesis testing (Maini al. [021] [Song & Shmatikov 2019).
whether an entire dataset were used in training specific model? Such binary methods are observed to be fragile under partial dataset utilization (i.e., a practical setting where models are trained on a combination of subsets from multiple data sources), In Figure I we plot the original binary prediction given by two representative works (Li et al. 2023 Maini et al. 202T and the continuous scores we retrieved from their methods (normalized to the range from O to I). (See the implementation details in Appendix B While both methods achieve perfect prediction when predicting binary utilization (either none or all), their decision fluctuates in cases of partial usage (regardless of the threshold). For example, both methods will neglect small proportion of usage, and might classify a model as "utilizing" the dataset when 50% of data included in the training set, yet inferring a model as "not utilizing" the data when usage increases to 60%. Such inconsistencies make these methods unreliable.
However, these methods are limited exploring a binary question:
In practical applications, we also need know the extent of dataset usage. For example, according to the Section 107 of the U.S. Copyright Act (United States Code 1976). determining whether use qualifies as fair use or copyright infringement must consider the "amount and substantiality of the portion used in relation to the copyrighted work" and "nature of the copyrighted work" (i.e., different
"Co-firt author
'Code is at / http://hhh..om//rivatttttttttttt http:///p.raaeeeeeeeeeeeee//aaaa
research

---

2. We develop novel debiasing procedure that is applicable to all membership identification methods (equation 4 Published as a conference paper at ICLR 2025
cases require different thresholds). However, none of the existing methods can be extended in a straightforward way to predict the proportion of dataset usage, due to their specific assumptions and objectives. Specifically,
goal is to identify suitable continuous score and inferring dataset usage without any modifications to the data itself. This later constraint is critical for broadening the applicability of our methods to scenarios where all prior binary watermarking approaches (Guo
et al. [222] Li et al., 2023; Tang et al., [0233] Hu et al., [2022] Li et al. 2022b) Wenger et al. 2022]
Sablayrolles et al., 2020) fall short: 1) scenarios where altering high-quality data is undesirable
or where data integrity must be preserved (e.g., in healthcare applications); 2) situations involving
previously released datasets that have likely already been incorporated in trained models.
Can we reliably and cost-effectively infer the proportion of the target dataset used in a model? We introduce the Dataset Usage Cardinality Inference (DUCI) problem in machine learning. via a paradigm shift from binary formulations granular assessment of dataset utilization, and propose efficient and accurate algorithm to precisely estimate the proportion of a dataset used in model training. Specifically,
... Dataset Inference [Maini et al., 2021]
Backdoor Watermarking [Li al. (2023)
aod 00000
Binary Proliction Danset
0.8
Normellee Proicia
0.4
1. We formally define the DUCI problem and its challenge from unavoidable membership guesses errors (Section 3).
0.2
0.2
0..
0.8
3. We design an unbiased estimator for data usage by de- biasing individual membership guesses (Section 4) and quantify uncertainty in DUCI by constructing confidence intervals via Lyapunov Theorem (Billingsley 2017 and validate them through extensive experiments (Figure 3)
Proportion of Used Data
Figure 1: Prior (binary) data infer- ence methods are unreliable and fluc- tuate between and 1.
Our experiments confirm the accuracy and cost-effectiveness of our method (Section 5.3 provides reliable proportion estimation maximum error < 0.1 with as few as one reference model (Figure 2) and requires 630x less computational budget than idealized methods while maintaining comparable performance (Table
5. Our debiasing method can be extended to group-level (Table 2 and effectively address the challenge of practical quality-dependent dataset sampling schemes. Our methods can also be applied to language models for practical tasks such as quantifying book copyright infringement 44..
2 PRELIMINARIES
Notations. We use T to denote the training algorithm that, in addition to the model architecture, may have access to auxiliary public information, c.g., a population data pool p. We consider the standard supervised learning setting. where trained model fe processes input to produce prediction fo( (e.g., the predicted probability on target class in image classification or the inverse perplexity in text generation). Membership m; of data point x; for the training set D of model fo is m; = 1 if x; E D (a member), and m; = 0 if x; g D (a non-member). We define the term membership guess vi the predicted membership status of data point x; by membership inference algorithm.
2.1 MEMBERSHIP INFERENCE (MI)
MI test whether data point & is part of the (unknown) training set of a target model fo. We briefly present the state-of-the-art MI algorithm that enables the best performance in our experiments
Robust Membership Inference Attack (RMIA) (Zarifzadeh et al., 2024). The method combines multiple likelihood ratio tests, each testing whether the model output fa is more likely when x is in the training set than when it is replaced by random data point from the population (i.e., x is not in the training set).
m = MIA(x; 0) = 1 (L)o...)))) for threshold B(x),
(1)
We present the results of other MI algorithms in Appendix H.4 for comparison.
2

---

Published as a conference paper at ICLR 2025
where p is the population distribution, and (2, PiAE2 is the pairwise likelihood ratio be- tween x and 2, which can be computed as ((a)))/))))((()))))/((()))) and P(x) is normalization constant for æ calculated by integrating over some reference models trained with the same structure and training data distribution (but not the same training dataset) as the target model fa. Additional discussions of related works are deferred Appendix
P(Ola)
Here, P(x(0) (x)
DATASET USAGE CARDINALITY INFERENCE PROBLEM
In this section, formulate the problem of Dataset Usage Cardinality Inference (DUCT) in machine learning and discuss the challenges of solving this problem.
3.1 DATASET USAGE AS A NON-BINARY INFERENCE PROBLEM
Problem formulation. Given model and a target dataset X, our objective is to infer what fraction of data points in X is used in the training of t.
The model f is trained training algorithm T. Since the training set may include data from other sources, we model this by introducing population pool within T, from which additional data can be sampled. The Dataset Usage Cardinality Inference (DUCI) algorithm A acting as an agent for the dataset owner with full access to X-aims to estimate (denoted as p) the overall usage proportion p, given black-box access to and knowledge of the training algorithm T.
proportion p |x]
(m, L1, selected) of the dataset X, using
In this work, we assume each data point in X is independently included in the training set of 0 with probability %/- Let the vector hxll be the sampling probabilities for all records in X. The pipeline is as follows:
Dataset X [ara]]
X [=
x
Independently include æ, with probubility 77 +
Selected p proportion of X
Training Algorithm T
Black- Box
A +
(2)
We aim to design DUCI algorithm A that, for any possible dataset usage proportion value, achieves low prediction gap. That is, minimizing the maximum mean absolute erro across usage proportions, where the mean ensures robustness algorithmic randomness, i.e.,
(
min max A
[Ip 211,
where p = ト Σ!A my denotes the ground-truth proportion, p = A(O, X) denotes the estimated proportion, and the expectation is over random trials of the DUCI pipeline (Equation (2) i.e., over data sampling m. Bernoulli (y), = I,..., XX| and the random coins of the training algorithm.
3.2 CHALLENGES IN DATASET USAGE CARDINALITY INFERENCE
natural strategy for inferring dataset usage is to sum the membership inference guesses over individual training data. However, we observe that this method suffers from poor performance (Table due to the inherent errors in inferring membership of individual data points.
Errors in optimal point-wise membership inference: Membership inference the level of individ- ual training data has high error when there are inherent randomness in the training pipelines (Ye) al. 2022) (such as data sampling and randomness of the training algorithm). When the training algorithm satisfies certain constraints in terms of output indistinguishability, one can prove that even the theoretically most informed and optimal membership estimate (m,) has an error that increases with the level of indistinguisaabiliy (Kairouz et al.) 2015 Steinke et al. 2024). In practice, the error of existing membership inference methods not only due to these inherent errors but also due to their inability to extract all membership information from model outputs.
See Appendix D and discussions on Figure 3 for why additive error is more appropriate measure and why multiplicative guarantee for the DUCI problem impossible.
3

---

Published as a conference paper at ICLR 2025
Challenge: biases arising from the accumulation of errors: The aforementioned per-poiint point mem- bership identification errois accumulate and introduce a bias in solving the DUCI task. This bias is clearly illustrated in Table | In the next section, will discuss strategies to mitigate this bias, and show that aggregating debiased membership guesses can serve as a promising proportion estimator.
4 UNBIASED DATASET USAGE INFERENCE FROM AGGREGATION OF INDIVIDUAL GUESSES
Given the unavoidable errors in membership identification methods, how can we achieve reliable (e.g,, unbiased) estimation of the proportion?
Denote X as the dataset of interest, and M as the distribution of the ground-truth membership vector. We have M = Bernoulli
Bernoull((y)x)- i.e., the probability of each data record 2; being used is (Section 3.1 For each trial, a target model A is trained by randomly drawing a membership vector m (m) "",,)"",,"),))) ~
M, and including the i-th data record of X (denoted as æ;)) if m, for each As the proportion p(m) 지
x=l my is value that varies
across trials due to the randomness in m, our goal is to construct a reliable estimator j(m) that is accurate (in expectation) across random trials. To do so, we design estimator p(m) that satisfies (t(.mM(((mm)) [m]]. i.c., is unbiased over trials. For brevity, when the context is clear, we use p and p refer to p(m) and p(m). respectively.
Em~M (wp.Mppp(mmmm m]] Em>M[pi
Construct ý from debiased individual MIA guesses. By definition, the expected inclusion propor- tion p the sum of inclusion probability for each sample Thus. by averaging per-sample estimate for the inclusion probability of each data record, we design the below estimator:
X
1
m, Pr(m- 1 1mm = 0)
p =
Σ
Ps where P. = Pr(m; = 1 my = 1) - Pr(m; = 1|m, = 0)
(4)
Here, Pi estimates the inclusion probability " for data record z;; m E {00 1} is the membership guess (given by an arbitrary membership inference method) for data record I; (with ground membership E {0.1)) and Pr(m; 1[m, = 0) and Pr(m; 1mm = 1) are the per-record
False Positive Rate (FPR) and True Positive Rate (TPR) for inferring the membership of each 2, X
(constants across trials).
We refer to the computation of ú; as debiasing process and assume TPR A FPR, which is reasonable given that membership identification algorithms are designed to discriminate between member and non non-member data points, not to act randomly. In our experiment, we use the state-of-the- art MIA methods to compute the membership guesses m MIA (z 0) (as introduced in Section 2p. but note that our approach is flexible and applicable to any membership prediction technique.
Unbiasedness of و and its intuition. How is, and estimator P satisfying Em MM [p(m)] = (p Note that as m, ~ Bernoulli(ys). E[m,]] 74 for any 1,, ..., [X| over the random trials. By the additivity of expectation, have E(p] = E x Σ m+] = |x _ 7/-- Thus, estimating 7 and aggregating across all data ideal solution. Since 7/4 = Pr(m: 1) Pr(m, 0). the rule of total
estimator b be unbiased across trials? That
points serves probability ensures the following relationship:
B [m]] = Pr(m = 1) 0 Pr(m, 0)
Pr 1|m 0) Pr(m: 0) + Pr(m:: 1/m = =) Pr(m: =
(5)
Rearranging terms in Equation 5
yields úE in Equation (4) Consequently,
X
[X|
1 X
1
E m - Pr(m; = m, = 0)
P =
E
X
Pr(m, m,
Pr(m; m; 0)*
which simplifies to Elp)
Elp). Therefore, f is an unbiased estimator of the expected inclusion proportion. We next discuss how compute the TPR and FPR in Equation (
X
Σ
i=1 74
[X|
Efficient and accurate estimation of TPR and FPR. To compute the estimators in Equation (4) we require the probabilities Pr(m, Im; 0) (FPR;) and Pr(m: 1|m =1) (TPR.). A direct, but
4

---

Published as a conference paper at ICLR 2025
computationally expensive, approach is to estimate these probabilities using N empirically trained reference models, where the training datasets are known. However, this method incurs a sampling error of order 52 small). To reduce error and computational cost, we propose estimating TPR and FPR across the entire dataset X, which requires as few as a single reference model and can reuse models from the MIA process. This simplification is justified because the proportion p is a dataset-level statistic, i.e., Elp] - E Penote .I.....0, ON N reference models (where N can be 1), each trained on randomly sampled halves of the dataset X, we can compute:
(i.e., the approximation error of TPR; and FPR; can be notable when N is
X
Σ
MIA(0 an m;
FPR
FPR;
=
(6)
X
X
j=
N
X
MIAO0 2.) and iy m0
TPR
TPR; R 2
X
X
j=1
The confidence interval analysis below reveals that this empirical average is only subject to sampling error (i.e., the standard deviation of the empirical estimation of FPR and TPR over the randomness of 0 and MIA algorithm). See Appendix G the proofs. Thus, employ- ing larger datasets and more reference models enhances the accuracy of our debiasing method.
O( / /
Confidence interval and uncertainty estimates for Dataset Usage Cardinality Inference Modern machine learning algorithms and their data sampling processes are highly randomized. To capture the intrinsic uncertainty and confidence in data usage inference, we resort to the Lyapunov Central Limit Theorem (CLT) (Billingsley 2017 to compute confidence intervals for our estimation method via aggregate statisties. Specifically, if the individual statistics $1, ,/|x in Equation (4) are independent and the Lyapunov condition is satisfied (see the proof's in Appendix (222.. then their aggregation converge to a Gaussian distribution as |X| increases, By applying Lyapunov CLT and using the sample variance estimate the average variance (see details Appendix F). we obtain the following approximate 95% confidence interval for estimating Elp)
|
s2
i - 1
Eppl
ta/21
(8)
X
X
Here P [X I A. 0.05, ta/2 is the critical value, and s2
|xi-1 (f:
1 E
2
5 EXPERIMENTAL. SETUP AND RESULTS
5.1 EVALUATION SETTINGS
We considered the general uniformly random sampling setting and special data selection setting, ensuring that p is controllable over the random trials (as described in equation The comparison experiments in Section 5.3 are conducted under the uniformly random sampling, and Section [.4 extends the discussion to the special sampling scenarios. For uniformly random sampling. examine all methods under dataset usage proportion E [o. 1] with a granularity of 5%. i.e., p 0.00, 0.05,... 1.00. For each p. evaluate the (mean absolute) error E[[p ~l] of proportion estimation over 30 random trials. In each evaluation trial, we train the target model on freshly sampled random p proportion of the protected dataset X, combined with randomly sampled remaining data from population pool (that does not overlap with X). For settings with specialized data sampling, we constructed the data selection probability vector p for each ground-truth p using data
"To clarify further, under datasct-level estimates, we have + Σ, Fljd Elp) + Corr, (1PR 372
TPR-FFX
(See derivation in Appendix E). This suggests that for many practical samplings (e.g., i..d. sampling), this simplification yield unbiased estimator for p. For special sampling methods, can debias subgroups achieve the unbiasedness (as discussed in Table [
'ee experimentally validate that P1.... . approximately independent (Figure and use > 30 individual observations compute valid confidence intervals via Lyapunov CI.T.
5

---

Published as a conference paper at ICLR 2025
selection strategies (Paul et al. 2021) and sample data from X for model training accordingly. We then compute the mean absolute error (MAE) in the same way.
Considering the evaluation cost of training target models various proportions p. mainly assess all methods in Section [.3 across three model architectures: a standard 5-layer fully connected network (FC-5), wide ResNet with width 2 (WRN28-2) 2) (Zagoruyko & Komodakis (016), and ResNet-34 (He et al. 2016), as well as three benchmark datasets: CIFAR-10 10 (Krizhevsky et al. 2009), CIFAR- 100 (Krizhevsky et al. 2009), and Tiny-ImageNet (Le & Yang 2015). We further validated the applicability of our methods to language models through a book copyright infringement case study on GPT-2 (Radford et al. 2019). using the BookMIA dataset (Shi et al. 2023). To implement our unbiased proportion estimator, we use MIA to predict the membership guesses, i.c., and defer discussions
rit1, ... x- We report the best- pers-perfonnnng MIA (i.e., RMIA Section 2.1p
on other MIAs to Appendix [[add Appendix H.44 See Algorithm 1 for pseudocode of our method
and see Appendix H for more implementation details.
5.2 BASELINES
Vanilla Membership Inference Attacks (MTA) baselines To demonstrate the importance of debias- ing, we compare our method against two intuitive baselines: direct averaging of binary MIA guesses and real-valued MIA scores outputted by existing MIA methods (Zarifzadeh et al., 2024 Carlini We assign N reference models with each trained on half of dataset X to MIA baselines:
et al. 2022).
MIA Guess: This involves directly counting MIA membership predictions, where the proportion is calculated the fraction of member predictions: ý |x] | m, For fair comparison,
use the same MIA algorithms that were employed our method.
MIA Score: This takes the average of MIA confidence scores across all data points: X
Σ [X| è, where is the confidence score of RMIA (i.e., probability before thresholding).
[X]
Maximum Likelihood Estimate (MLE) baselines When computation cost is not concern, the below computationally expensive MLE estimators can serve as an idcalized baseline:
(M) arg max Pr(olp= = 4)2
where Pr(olp- = 9) is the conditional probability of observations o (generated from the trained model 0) given that proportion of the records in the target dataset X is used in training. This probability Pr(alp- = q) can be empirically approximated via training many (N) reference models on freshly sampled random q-fraction of the target dataset X for each of 21 possible proportion value q. employing the same algorithm T that was used for training the target models (in total 21 N reference models). Regarding the observation o in the MLE baselines, consider two choices follows.
Joint logits (Joint-Logit) of all records in target dataset (bigh-dimensional): Building on prior research (Carlini et al. 2022) which demonstrated that scaled logits for individual data points can serve as effective observations and empirically follow a Gaussian distribution, we naturally extend this approach to a dataset- wide scale. We concatenate these point-wise logits into high-dimensional joint logits and model their distribution using a multivariate Gaussian.
Pr(olp = g) = Pr(logit(X; (X;)) Eg)):
(10)
where logit(X: () is the the logits of model 9 on all records in X, and My and Eg are its empirical mean and covariance matrix across N reference models trained with q proportion of X. Averaged logits (Avg-Logit) of all records in the farget dataset (one-dimensional): To enable more accurate empirical mean and covariance estimation, we reduce observation dimensionality by using average statisties (i.e., the average of logits) over the dataset X, akin to Maini al. (202T).
The likelihood is then modeled as one-dimensional Gaussian.
Pr(op= = g) =
logit(r; 0) (Pgg ( #EX
Σ
X
where My and o, are the empirical mean and variance of the averaged logits of all records in X across N reference models trained with y proportion of X.

---

Published as a conference paper at ICLR 2025
Table Maximum mean absolute error (MAE) (maxpe[o.1) Elpp pl) over 21 proportions p. each derived from 30 trials, of all methods across various datasets and model architectures within the same computational budget (# reference models). The lowest error in each test is highlighted in bold. Grey rows are the idealized computationally expensive MLE baselines.
#Ref
Methods
CIFAR-10
CIFAR-I00
Tiny-ImageNet
FC-5
WRN28-2
ResNet34
WRN28-2
ResNet34
WRN28-2
MIA Guess
0.3291
0.4819
0.1611
0.1676
0.0889
0.1478
MIA Score
0.3619
0.4486
0.2950
0.3220
0.2909
0.3040
Ours
0.0678
0.1836
0.0534
0.0873
0.0879
0.0580
2
MIA Guess
0.2617
0.4801
0.1637
0.1458
0.0859
0.1131
MIA Score
0.3587
0.3865
0.2687
0.3074
0.2822
0.2952
Ours
0.0380
0.1787
0.0421
0.0851
0.0723
0.0463
4
MIA Guess
0.2797
0.4350
0.1468
0.1401
0.0746
0.0889
MIA Score
0.3571
0.3822
0.2571
0.2836
0.2593
0.2739
Ours
0.0345
0.1634
0.0325
0.0774
0.0564
0.0242
8
MIA Guess
0.2831
0.3862
0.1493
0.119
0.0768
0.1099
MIA Score
0.3022
0.3621
0.2552
0.2824
0.2588
0.2759
Ours
0.0296
0.1127
0.0180
0.0554
0.0368
0.0178
42
MLE (Joint-Logit)
0.2450
0.2230
0.1983
0,2667
0.2050
0.2067
MLE (Avg-Logit)
0.0533
0.1383
0.0567
0.0617
0.0383
0.0583
MIA Guess
0.2516
0.2889
0.086
0.0797
0.0377
0.0867
MIA Score
0,2948
0.3692
0,2530
0.2920
0.2576
0.2867
Ours
0.0271
0.0722
0.015
0.0339
0.0124
0.0179
630
MLE (Joint-Logit)
0.1400
MLE (Avg-Logit)
0.0283
5.3
MAIN RESULTS
In this section, we compare the performance of our method (Algorithm D with the baselines in Section 5.2 in terms of estimation quality and efficiency.
Comparison under the same computation budget Table T shows an overview of the comparison results under different datasets and model architectures while keeping the total computational budget fixed. Note that for our debiasing process, single reference model is enough and those reference models for launching MIA can be reused. For the MIA Guess baseline, achieved across candidate MIA methods. We first observe that our method consistently outperform all baselines under all settings under 42 reference models (which is the minimal number of reference models required for MLE baselines), whether on well-generalized setting such as WRN28-2 on CIFAR-10 10 more challenging datasets like CIFAR- 100 and T'nny-lmagenet. These latter datasets have fewer data records per class, thereby increasing their susceptibility to memorization and generally making them easier tasks for DUCI. We then tested the case with extremely limited computational budget (i.e., 1-8 reference models). This is challenging, as the scarcity of empirical samples for estimating FPR and TPR will lead less accurate debiasing. Additionally, existing MIAs themselves also suffer from reduced strength when the number of reference models is small (Carlini et al. 2022 Zarifzadeh et al. 2024). Remarkably, our method achieves significantly smaller estimation error than vanilla MIA baselines even with as few as 1 model, i.e., our method can provide reliable proportion estimates even under a severely restricted computational budget.
report the best performance
Figure 2 further shows that, for all methods, prediction еттоr generally decreases as the number of reference models increases, under the setting of FC-5 trained on CIFAR-10 10.. This improvement is expected, as more reference models provide more accurate estimation of the likelihood Pr(olp) for MLE methods, the distribution for the MIAs, and TPR/FPR for our debiasing methods (Section 4 thereby reducing prediction errors. Notably.
method outperforms MLE methods more under less computation budgets This is because the accuracy of MLE methods relies heavily on the quality of approximating the Pr(olp) for all possible p E [0, 1). task that becomes challenging with limited
7

---

Published as a conference paper at ICLR 2025
1.00
Prequency
0.60
maxxx (0.)) Elpp [0]
0.3
0.20
MLE Joint Logit
0.2
Aug-Logit
MIA Ciocss
0.14
MIA
0.1
Our Method
Longth
0.12
0.08
0.08
20
27
Number of Reference Models
0.2
0.4
0.6
Tree proportion of Used Data =p
Figure 2: Maximum MAE of all methods under an increasing number of reference models. The grey dotted line represents the minimum compu- tational budget required for MLE methods. Our method yield meaningful predictions even using single reference model.
Figure 3: (Confidence Interval) Top: Frequency of the true proportion p falling within the pre- dicted 95% confidence interval for different p. The frequency was computed across 30 interval predictions for each p. Bottom: Average width of the predicted interval over 30 tests for each p.
computation budget. Our results indicate that, under such constraints, the distribution estimation process required for MLE introduces more significant bias compared to our method.
We also allocate a substantial budget-a total of 30 x 21 630 reference models- to the strongest baseline, the MLE methods (grey rows Table D fully exploit their potential. Note that MLE methods, given extra information candidate proportions (i.e.. hypotheses) and accurate distribution estimation, can serve as idealized baselines according to Neyman & Pearson (1933). The comparison demonstrates that our method can achieve comparable estimation quality to MLE methods while incurring significantly lower computational costs, ic., 100 smaller. Due to the huge computational costs, only presents the results of FC-5 models trained on CIFAR-10 for this comparison.
Confidence interval for dataset usage inference Our methodology also aims to capture the un- certainty inherent in dataset usage inference. To evaluate the efficacy of our interval estimation, we conducted tests on the same series of target models- 30 trials for each propotion-eemploying significance level of
0.05. The confidence intervals Equation 22 where the degrees of freedom is X = 499, leading to a t-value of la/2 1.96.
computed in accordance with
Figure 3 (Top) displays the frequency of the true proportion p falling within the predicted 95% confidence interval. There is a consistently high frequency (over 0.9), which demonstrates the high coverage of our interval predictions. Moreover, the intervals generally have small length, as shown in Figure 3 (Bottom), suggesting that the overall uncertainty of our proportion prediction is small (around 0.1). Note that here we derive the additive confidence interval (absolute uncertainty) instead of the multiplicative confidence interval (relative uncertainty) because the eтror in Dataset Usage Inference task not continuous. (e.g., For a small dataset in size 100, the unit of error is 1%.) Additionally, we observe that our confidence intervals exhibit greater length when reasonable as the number of possible p- p-proportion subsets of the target dataset is the largest when p 0.5, thus making dataset usage inference inherently more complex at 0.5.
0.5. This is
5.4 DISCUSSIONS
We next discuss the properties, further improvement and potential application our method.
Dataset usage inference under special sampling challenges. While our dataset-level debiasing methods (Equation (6) and Equation (7) are statistically guaranteed to be unbiased under common
Table 2: Relative stability of methods under special sampling, measured by Maximum mean absolute error (MAE) (max,e(o.1) E[pp Pl]) over 21 proportions.
Sampling
MLE (Best)
MIA Guess
Ours
Random
0.0533
0.2933
0.0619
Significant First
0.6117
0.3876
0.1089
Insignificant First
0.6301
0.3787
0.1077
8

---

Published as a conference paper at ICLR 2025
Table 3: Dataset usage estimation error decreases as the (relative) size of the target dataset increases for all methods. Errors are measured by Maximum Mean Absolute Error (MAE) (maxpe(o.1) E[p 511) across 21 proportions using WRN28-2 trained on CIFAR-10
|
MLE (Best)
MIA Guess
Ours
500
0.1383
0.2889
0.0722
5000
0.0849
0.268
0.0528
25000
0.08
0.2659
0.0442
sampling scenarios like i.i.d. and uniformly random sampling (see Footnote for clarification), the question remains: can they also achieve near-unbiasedness under special or adversarial sampling? To the best of our knowledge, all previous work considers uniformly random (or i.i.d.) sampling scenario in method design (or evaluation) (Li et al. [023]] Tang et al.) [202]] Maini et al. [02T]) This means that when the target model employs special but unknown sampling, the sampling mismatch presents a significant challenge for dataset usage inference techniques.
As noted in Footnote 4 the unbiasedness of our methods hinges on the correlation term Com: ( - .r)
(Under uniformly random sampling, this term zero since 7/- -p for all i = L....... [X])) For settings with special data sampling, we construct subgroups where TPR; FPR; ~ TPR' - FPR' and debias within each to obtain an unbiased estimator of 7/ (see Figure@for empirical validation).
TPR FPR
To evaluate the performance of all methods under the special sampling challenge, we conduct experiments where the model trainer applies dataset selection techniques (Guo al. [2022] Sorscher et al. 2022) to select p-percentage of the target dataset to include in training. Specifically, we adopt an error-based coreset selection method, EL2N (Paul et al. 202T}, to rank the target dataset by "data significance | using 10 trained models that have not seen the target dataset. Then, constructed the probability vector (P) over the target dataset X according to this "data significance" for each p E {0.00, 0.05, ..., 1.003. The data in the target dataset that used in the training of target models are therefore sampled according to each (P))) By contrast, the reference models are trained under uniformly random p% subset of the target datase These scenarios represent the least favor samplings for our methods, as the EL2N is closely related to TPR; FPR,, which introduces strong correlation between and the errors.
Table 2 shows that all other methods degrade in performance under special sampling challenges and exhibit similar performance across the two symmetric sampling methods. This confirms earlier discussion on the distribution mismatch challenge from special sampling: they make mistakes occur early or late in the process. Nevertheless, the effectiveness of our debiasing method largely remains and yields better performance than all other methods. By contrast, the MLE method suffers the most significantly under the mismatch, since its superiority comes from fully recover the possible model distribution when data sampling scheme is known.
Dataset usage estimation error decreases as the ratio Drein) increases. Previously, we focused on a challenging scenario where the owner's dataset (size 500) constitutes only small portion of the target model's training set Dirain- i.e., the ratio D was small. We now examine the impact of the (relative) dataset size on the DUCI performance. To do this, we evaluated our method using target dataset sizes from 500 to 25,000 (i.e., relative ratios from 0.02 to 1.00), while keeping the full training set fixed at 25,000 examples (half of CIFAR-10's training set).
that all methods benefit from the increasing relative dataset size. This is because, as the ratio |X|/| increases, the randomness from the remaining data in Drrain decreases, making dataset usage inference easier. For instance, in the extreme case where | = 1, there is no randomness at 100% utilization. This observation highlights that prior evaluations in the literature (Li et al. [023 Maini et al. [027]. which test whether a model was trained on private dataset A versus an alternative dataset are oversimplified and offer limited insights into real-world scenarios. Additionally, we noticed that while the improvement for MIA baselines is modest (as
AA term we use cautiously, as EI.2N can only be considered as reflection data value. Since the sampling method used in the target model is inaccessible, assuming random sampling is the reasonable (if not the best) strategy dataset owner can take for simulating the model trainer.

---

Published as a conference paper at ICLR 2025
MIA does not leverage dataset-level statistics and benefits only from reduced randomness at larger ratios), our debiasing process sees a more substantial gain. This is because the sampling error in our | (see formal proof in Appendix Go, meaning that as the target
debiasing is proportional to
dataset size increases, our debiasing procedure becomes more accurate.
On (the possibility of) leveraging correlation in membership prediction process Our method implicitly assumes that the prediction of membership probability for one record i does not affect that of another record j. i.e. Blp:p,] El [[pp]] Will ignoring "correlations" between P; and Pj make our method sub-optimal? We derive pairwise debiasing approach in Appendix [.2] but observe only minimal gains. To understand this, we measure the covariances between the predictions for pairs of records in the target dataset (no duplication) using the error metric in Equation (34 and find that most pairs show near-zero covariance (Figure 4). This aligns with the prior theoretical findings of small correlation between records (Pillutla et al. 2023). This suggests limited room to improve our method by modeling correlations natural dataset.
Case study: quantifying book copyright infringement can be seen as a specific case of data usage unference-giverd model likely infringing on book copyrights, the authors want determine the extent of content usage. We conduct experiments on GPT-2 (Radford et al. 2019) and 50 new books with first editions in 2023, unseen by GPT-2, from the BookMIA dataset (Shi al., 2023). We fine-tune pre- pre-trained GPT-2 on varying proportions of these books to create target models. The sequences in books are sampled contiguously. rather than randomly and uniformly. from the copyrighted books in order to simulate more realistic scenarios. Results in Table Table@pste show Shole that both MIA Guess and MIA Score baselines struggle with varying extents of infringement, particularly when infringement is less significant. In contrast, our method estimates the extent of infringement much more reliably, with a maximum MAE (Equation 33 of 168, compared to 0.335 for MIA Guess and 0.5 for MIA Score. However, observed that, compared to image data, the eπror variance in text data is generally larger. We defer the discussion and additional experimental details to Appendix [J]
107
Table 4: (Book copyright infringement) Mean Absolute Error (MAE) Bllp, - pl] for all methods under different proportion p.
Oouns
p
MIA Guess
MIA Score
Our Method
0.0
0.3350
0.5019
0.1684
0.2
0,2396
0,4029
0.1573
1023
0.5
0.4
0.1202
0.2381
0.0833
Paiswis Covatiance Co
0.6
0,0439
0.1028
0.0316
Figure 4: Histogram of the covari- ance Cov(p,, P;) between data us- age predictions of all data pairs and in the target dataset.
0.8
0.1286
0.0720
0.0453
1.0
0.2448
0.2189
0.0568
maxp MAE
0.3350
0.5019
0.1684
6 CONCLUSIONS
We identify and formally define the problem of Dataset Usage Cardinality Inference (DUCI), which quantifies dataset utilization- significant issue that has been overlooked or misinterpreted in prior literature. To solve this, we propose low-cost, reliable, and unbiased estimator for DUCI, along with confidence intervals for uncertainty. Our extensive experiments demonstrate the effectiveness of our approach, including its practical application in detecting book copyright infringement.
Limitations and future works We have thoroughly discussed the special sampling challenges in DUCI in Table 2 and demonstrated improved robustness of our methods compared to baselines. However, there 18 still room for further enhancement, particularly in achieving statistical guarantees under adversarial samplings. Another promising avenue applying our methods to related tasks, such as debiasing and aggregating user-level inference (Li et al.) 2022a Chen et al.) 2023 Kandpal ctt al. 2023) for finer-grained neighborhood inference. Given that whether or not datasets similar to copyrighted data, but not directly used in training the target models, should be considered members still an open question, we leave these extensions for future work. Lastly, while our method is independent of specific MIA and dataset choices, and quantifies uncertainty in proportion prediction, dataset crafting methods that enable FPR guarantees for MIAS could be a vital direction for stronger MIAs and further reducing our method's uncertainty.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The authors would like to thank Martin Strobel for his valuable feedback and discussions that contributed to this work, as well the anonymous reviewers for their insightful comments. Jiayuan Ye is supported by the Apple Scholars in AI/ML PhD fellowship. This research is supported by research awards from Google, and the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (MOE-12EP20223-00)
REFERENCES
Martin Bertran, Shuai Tang, Michael Kearns, Jamie Morgenstern, Aaron Roth, and Zhiwei Steven Wu. Scalable membership inference attacks via quantile regression. arXiv preprint arXiv:2307.03694
2023.
Patrick Billingsley. Probability and measure. John Wiley Sons, 2017.
California Legislature. Califormia consummrr privacy act of 2018. Assembly Bill No. 375, Chapter 55, 2018. URL ht htt.::::://egg................. ca. wttp:///....e..eeee/eeeeeeeeeee.
xhtm1 (ttp1ll_d=20072111000000
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Guangke Chen, Yedi Zhang, and Fu Song. Slmia-sr: Speaker-level membership inference attacks against speaker recognition systems. arXiv preprint arXiv:2309.07983, 2023.
European Parliament and Council the European Union. Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation). Official Journal of the European Union, L119, 1-88, 2016.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion deep learning. International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Junfeng Guo, Yiming Li, Lixu Wang, Shu-Tao Xia, Heng Huang, Cong Liu, and Bo Li. Domain watermark: Effective and harmless dataset copyright protection closed hand. arXiv preprint arXiv:2310. 14942, 2023.
Moritz Hardt and Kunal Talwar. On the geometry of differential privacy. In Proceedings of the forty-second ACM symposium on Theory of computing, pp. 705-714, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muchling, John Pearson, Dietrich A Stephan, Stanley F Nelson, and David Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS genetics, ((8)e1000166 2008.
Hongsheng Hu, Zoran Salcie, Gillian Dobbie, Jinjun Chen, Lichao Sun, and Xuyun Zhang. Member- ship inference via backdooring. arXiv preprint arXiv:2206.04822 2022.
Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer, Quanquan Gu, and David Evans. Re- visiting membership inference under realistic assumptions. arXiv preprint a0Xiv:2005.10881. 2020.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning, pp. 1376-1385. PMLR, 2015.
Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher A Sloquette-Choo and Zheng Xu. User inference attacks on large language models. arXiv preprint 230.0663,. 2023.
11

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
Guoyao Li, Shahbaz Rezaei, and Xin Liu. User-level membership inference attack against metric embedding learning. arXiv preprint arXiv:2203. 02077, 2022a.
Yiming Li, Ziqi Zhang, Jiawang Bai, Baoyuan Wu, Yong Jiang, and Shu-Tao Xia. Open-sourced dataset protection via backdoor watermarking, arXiv preprint arXiv:2010.05822 2020.
Yiming Li, Yang Bai, Yong Jiang, Yong Yang, Shu-Tao Xia, and Bo Li. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. Advances in Neural Information Processing Systems, 35: 111113338122555 2022b.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transac- fions on Neural Networks and Learning Systems, 2022c.
Yiming Li, Mingyan Zhu, Xue Yang. Yong Jiang. Tao Wei, and Shu-Tao Xia. Black-box dataset ownership verification via backdoor watermarking, IEEE Transactions Information Forensics and Security, 2023.
Zheng Li and Yang Zhang. Membership leakage in label-only exposures. Proceedings of the 2021 ACM SIGSAC Conference Computer and Communications Security, pp. 880-895, 2021.
Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang. Membership inference attacks by exploiting loss trajectory. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 2095-2098, 2022.
Yunhui Long, Vincent Bindschaedler, Lei Wang. Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well-generalized learning models. arXiv preprint arXiv:1802.0488%) 2018.
Yunhui Long, Lei Wang. Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 521-534. IEEE, 2020.
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. arXiv preprint arXiv:2104.10706 2021.
Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06444,, 2024.
Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-Béguelin, Shruti Tople, and Reza Shokri. The canary's 's echo: Auditing privacy risks of Ilm-generated synthetic text. arXiv preprint arXiv:2502.14921, 2025.
Yuantian Miao, Xue Minhui, Chao Chen, Lei Pan, Jun Zhang, Benjamin Zi Hao Zhao, Dali Kaafar, and Yang Xiang. The audio auditor: user-level membership inference in internet of things voice services. Proceedings on Privacy Enhancing Technologies, 2021:209-22 228, 2021.
Jerzy Neyman and Egon Sharpe Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231((94-706):289 337, 1933.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances in neural information processing systems, 34: 20607, 2021.
Krishna Pillutla, Galen Andrew, Peter Kairouz, H Brendan McMahan, Alina Oprea, and Sewoong Oh. Unleashing the power of randomization in auditing differentially private ml. arXiv preprint arXiv:2305. 18447, 18447 2023.
Saurav Prat. Fine-tuning gpt-2 for text generation.
http:://debugeerfffe... Fire-ttuiing-gpt22 -tt.eeettteeeeaiiooo 2023. Accessed: 2024-05-21.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. International Conference on Machine Learning. pp. 5558-5561 PMLR, 2019.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. In International Conference on Machine Learning, pp. 8326-8335.
PMIR, 2020.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint 1806.00000.11222 2018.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zett Detecting pretraining data from large language models. arXiv preprini 0rXiv:2310.16789 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 196-206, 2019.
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pp. 2615-26333 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19523-19538 2022.
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36, 2024.
Zhensu Sun, Xiaoning Du, Fu Song, Mingze Ni, and Li Li. Coprotector: Protect open-source code against unauthorized training usage with data poisoning. Proceedings of the ACM Web Conference 2022, pp. 652-660, 2022.
Ruixiang Tang, Qizhang Feng, Ninghao Liu, Fan Yang, and Xia Hu. Did you train on my dataset? towards public dataset protection with clean-label backdoor watermarking. arXiv preprint arXiv:2303.1 1 1470, 2023.
The New York Times Company. Complaint. https://nytcon awtoos-ssesss... nytimest com/ pdf] December 2023. Civil Action No. 1:23-cv-
2023/12/NYT, 11122//TTCooppl___ 11195, United States District Court for the Southern District of New York.
United States Code. U.s. copyright act, title 17, section 107. https: wtwp::///wwc.yycggttccccc Fitle17/92chapl.. html 107 1976. Fair Use.
Nikhil Vyas, Sham M Kakade, and Boaz Barak. On provable copyright protection for generative models. In International Conference Machine Learning, pp. 35277-35299, PMLR, 2023.
Cheng Wei, Yang Wang, Kuofeng Gao, Shuo Shao, Yiming Li, Zhibo Wang, and Zhan Qin. Point- nebw: Towards dataset ownership verification for point clouds via negative clean-label backdoor watermark. IEEE Transactions on Information Forensics and Security, 2024.
Yuxin Wen, Arpit Bansal, Hamid Kazemi, Eitan Borgnia, Micah Goldblum. Jonas Geiping, and Tom Goldstein. Canary in a coalmine: Better membership inference with ensembled adversarial queries. arXiv preprint 22. 2022.
13

---

Published as a conference paper at ICLR 2025
Emily Wenger, Xiuyu Li, Ben Y Zhao, and Vitaly Shmatikov. Data isotopes for data provenance in dnns. arxiv preprint arXiv:2208. 1388:23,...1393,, 2022.
Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei. Neil Zhenqiang Gong, and Bhuwan Dhingra. Recall: Membership inference via relative conditional log-likelihoods. arXiv preprint arXiv:2406. 15968, 2024.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 3093-3106, 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 3 computer security foundations symposium (CSF), pp. 268 282. IEEE, 2018.
William Youden. Index for rating diagnostic tests. Cancer, 3(1):32-35, 1950.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07/ 2016.
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost power membership inference attacks. In JCML,, 2024.
Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership inference attacks cannot prove that model was trained on your data. arXiv preprint arXiv:2409.19798, 2024.
Zihang Zou, Boqing Gong, and Liqiang Wang. Anti-neuron watermarking: Protecting personal data against unauthorized neural networks. In European Conference on Computer Vision, pp. 465. Springer, 2022.
14

---

Published as a conference paper at ICLR 2025
Appendices Contents
Related works
16
Implementation details for watermarking and dataset inference
18
B. I Watermarking
18
2 Dataset inference
18
Implementation details of our method
19
Selection of membership inference algorithms
19
LiRA implementation
21
RMIA implementation
21
C,4 4
Selection of thresholds
22
Effect of MIA choices
23
D Impossibility of multiplicative guarantee for DUCI
23
Analysis of potential errors from simplifying record- debiasing to dataset-Level debiasing
24
Derivation of the confidence interval
26
Proof of overestimation E[s²]
o²
26
Proof of the Lyapunov condition
28
Proof of the sampling error of dataset-level debiasing
30
Experiment details and additional results
31
H.I
Experiment settings
................................
31
H.2
Implementation details of baselines
31
H.3
Forming subgroups by TPR, FPR
................................
32
H.4
Results using different MIA
33
H.5
Performance of method in determining binary dataset usage
34
On (the possibility of) improving Dataset Usage Inference via pair- "correlation"
35
On the possibility of covariance reduction
35
On the possibility of pair-wise bias reduction (Second-Order Debiasing)
35
Case study: quantifying book copyright infringement
39
15

---

Published as a conference paper at ICLR 2025
Notation
Description
T
Training algorithm, may have access to auxiliary public data. Population data pool.
P
D
Training dataset.
X
Target dataset (from data owner). A data record i.
m
Membership indicator (1 xy is in training set D, otherwise 0). Membership guess from an inference algorithm.
m,
A
Membership inference algorithm. Target model with parameter 0.
fo
fe (z)
Output of the target model on æ.
Proportion of dataset X usage in training 0.
Estimated dataset usage proportion.
7/
Probability of sample z; E being used in training fo- Estimated probability of sample æ; E X being used in training fa- Parameters of reference models (j = 1,..., , used for estimation.
p.
Table 5: Summary of notations used in the paper.
A RELATED WORKS
Membership inference attack Membership inference attacks (MIAs) Ye al. (202)) Shokri et al. (2017) aim to determine whether a specific data entry is part of the training dataset of target model. Initially gained traction in medical research Homer et al. (2008) MIAs were first introduced into machine learning by (Shokri et al. (2017), subsequently inspiring numerous shadow model attacks LLng et al. (020): Sablayrolles et al. (2019); [Snn Mittal (202T); Carlini et al. (2022). These shadow model attacks train a collection of shadow models (or reference models) to simulate the target model's 's behavior. All shadow models are trained with the same architecture as the target model using random subsets of data that follow the same distribution the target model's training data, including or excluding the target data point, to estimate a statistical distribution of certain function (such but not limited to loss (Sbllyrolles al. 2019); Yeom al. (2018). 8),
confidence Salem et al. (2018), entropy [Song & Mittal (2021). or loss trajectory Liu et al. 2022))
reflecting the data point's ss presence or absence in the training set. Aside from shadow model attacks, an alternative line of research eliminates the need to train multiple shadow models. Instead, they
queries the target model with population data that related to, or follows the same distribution as, the target point Wen et al. ); Bertran et al. (2023); Zarifzadeh et al.] (2024); Jayaraman al. (2020); & Zhang (202)):[ono Long al. (2018); Xie al. (2024). These data are utilized as reference for inferring membership. Individual MIAS cannot be directly adapted to solve the dataset utilization cardinality inference problem because they often cannot achieve perfect distinguishability. and their predictions vary drastically depending the specific threshold or test used. Therefore, it necessary to consider the uncertainty in the attack decision to reliably infer the extent of dataset usage. Moreover, the cost of conducting point- wise MIA tests for the entire dataset is another concern, which calls for an efficient adaptation.
A.0.1 DATASET OWNERSHIP VERIFICATION AND DATASET PROVENANCE TRACING
Dataset Ownership Verification (DOV) and Dataset Provenance Tracing are two related research area to our Dataset Usage Inference problem. Currently, all known dataset ownership verification techniques utilize dataset watermarking, which involves altering subset of the data to embed a detectable signature Li et al. (2023 2020); Tang et al. (2023): Li al. (202bb); Sun et al. (2022); Sablayrolles et al. (0220) Hu et al. (0022): y Zou et al. (2022 Wenger et al. (2022) y Maini et al. (2021) ; Song & Shmatikov (201)). In addition to these. handful of non watermarkkk based dataset
tracing techniques, although not specifically designed for the dataset ownership problem, can be adapted to address the dataset ownership problem related to this issue. These methods primarily
16

---

Published as a conference paper at ICLR 2025
encompass variants of membership inference techniques that operate beyond the individual level Song & Shmatikov (2019): Maini et al. (2021): ): Li et al. (022a): Miao et al. (0021). Thi
Dataset watermarking Dataset watermarking Li et (003) Tang al. (2023); Li al. (2020); Sun et al. (2022), predominantly utilizing poison-only backdoor attacks (that manipulate only the training data while keeping training components, model parameters, and structures intact Li al. (2022c)), currently leads in the field of dataset ownership inference. Generally, these methods capitalize on the memorization capabilities of machine learning models during training to embed spurious features (information divergent from the true data distribution) into the model. This embedded knowledge is then used for subsequent verification: the detection of specific backdoor behaviors (targeted untargeted) in the model's output during the verification phase indicates the model was trained with the protected dataset.
While the earlier watermarking techniques modified the target label of poisoned samples Li ct al. (2020); Hu et al. (2022), contemporary and more sophisticated methods employ untargeted backdoor attacks Li et al. (2022b);; (Sun et al. (2022) clean-label backdoor attacks (Sallyrrlllss al. (2020); ; Tang et al. )023); Zou et al. (2022). .. which are significantly more stealthy due to their preservation of
label consisteney. Several works have extended their focus to different application scenarios, such as protecting open-source code Sun et al. (2022) or performing classification tasks without knowledge of labels Wenger al. (2022). Additionally, recent works have attempted to minimize the impact on model utility when modilying samples Guo et al. (2023) However, all these modification- based methods are not suitable for scenarios where the goal to protect the ownership of already released datasets, nor do they guarantee data utility in sensitive scenarios, such as with medical data.
Dataset Inference While several class-level membership inferences Li et al. 2002a Miao et al. (202I) exist for object detection, such as voice or face recognition from an individual, these works are limited to their specific use cases and are not applicable to the broader scenarios considered in this paper: they require that the data group has unique pseudo-label. For text data that lacks clear boundary between records, [Song& & Shmatikov (2019) utilizes the long tail of the text data distribution within user's document to carry out lser- MIA. The work most closely related to ours is Dataset Inference Maini et al. (001)). which hypothesizes that private data in general will have a larger distance to the decision boundary than public data, since the model always attempts maximize this distance during training. It trains a linear binary classifier that takes in the distance-to to--tondbbbory for each data record to predict its membership. The underlying idea of this method to use membership inference based on the average of loss-like signal over the data group metric. However, all these methods still focus on solving the all-or-none binary question, and thus also fail to address the problem of inferring the dataset usage extent.
Another area of research in copyright infringement, such as Vyas et al. (003), focuses on protecting generative models from producing copyrighted content, regardless of whether the model was trained on copyrighted data. Although this task may appear similar to ours, focuses on different aspect of copyright and can be addressed quite differently-for for example, by adding distance constraints to prevent outputs from closely resembling protected content. In contrast, our goal is to infer properties of target model's training based its outputs. As this line of work is greatly distinct from ours, we did not discuss them in the scope of this paper.
17

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS FOR WATERMARKING AND DATASET INFERENCE
For both implementations, the target protected dataset X (for which infer the usage) of size 500. All models are Wide Residual Networks trained on 25,000 records from CIFAR-10 (that contains a randomly sampled fraction of the target dataset). This setting ensures that the target protected dataset is only small subset of the target model's training set, making it a more practical scenario.
B.1 WATERMARKING
The watermarking baseline depicted in Figure D is an implementation of Algorithm in (Li et al. 2023). Our objective is merely to demonstrate the limitations of watermarking methods in addressing the Dataset Usage Cardinality Inference (DUCI) problem. Consequently, we selected the most robust and straightffrward backdoor watermarking techniques, disregarding stealthiness. We employed the targeted poisoned-label attack with simple white square as the trigger pattern (examples are shown in Figure 5 Considering the sub-sampling inherent in the DUCI problem, we opted for a poison rate of 30% for the protected dataset with a size of 500 (lower poison rates were tested, but the likelihood of a successful backdoor attack diminishes with reduced poison rates). We designated class O as the target label.
The following steps were undertaken:
1. We randomly sampled m 2000 instances from the CIFAR10 test set, adhering to the methodology outlined in the original paper. Specifically, we only chose instances correctly classified by the benign model to minimize the impact of the model's 's intrinsie accuracy the test. If the model exhibits low accuracy, it might erroneously categorize benign instances poisoned, thereby influencing the confidence value.
2. Following Li et al. (2023).
obtained the confidence score at the ground-truth label for each benign sample x and its poisoned counterpart x' from the target model, denoted as P. f(x) and P, = f(x'), respectively.
3. We conducted pairwise T-tests: For each pair of samples, the difference in posterior proba- bilities P. P, was calculated. Utilizing these differences, a pairwise left tail T-test was performed to test the null hypothesis Ho P. Pp + where is hyperparameter. We evaluated performance across different values of for T E {0.0, 0.005,... 0.02). The results are reported under the best pest-performing T.
B.2 DATASET INFERENCE
We implemented simplified yet effective version of the approach described by Maini al. (2021). replacing the random walk distance (to the decision boundary) with logits queried on the target model for each data point. Empirically, simplification, and the essential principle remains intac( This approach is based on a hypothesis that private data in general will have a larger distance (smaller loss and higher prediction confidence) to the decision boundary than public data, since the model always attempts to maximize this distance during training. Therefore, comparing the average distance/logits score across the entire data group with that of public data can help to make a decision. All subsequent steps adhere strictly to those outlined in the original paper:
observed no significant performance degradation with this
1. Randomly sample data points each time from both the public dataset X and private dataset, and query the target model to compute their confidence scores, (public) and co (private).
2. We repeated Step I for [X/m iterations, calculating the mean values μ ē. and p4, C.-
3. We conducted the pairwise T-test on the distributions of and calculating the p-value to test the null hypothesis (the dataset is not being used) Hg : p Pa-
For example, the distance-to-houndary in linear model equivalent to the loss
18

---

Published as a conference paper at ICLR 2025
Figure 5: Visualization of poisoned data. Following standard method, add the white square with random transparency as the trigger pattern.
C IMPLEMENTATION DETAILS OF OUR METHOD
We next introduce the implementation details of our method.
Algorithm provides the pipeline of our method, where we use the existing membership inference algorithm, denoted as MIA, as the membership identification method to provide membership guesses vi, for all 2; X. We first trained N reference models, O,,..., 0N. such that each reference models are trained on half the dataset X. Knowing the membership status of all æ; E X for these reference run MIA them to get membership guesses for each Tú and then compute the TPR and FPR as defined in Equation (6) and Equation (7). The aggregation debiased membership guesses gives
models,
unbiased estimator of the expected inclusion proportion. To reduce potential errors from correlations between the MIA score and membership predictions in certain MIA methods, we recommend using separate reference models the target model and attack models during the debiasing process.
C.1 SELECTION OF MEMBERSHIP INFERENCE ALGORITHMS
While any membership identification method capable of providing membership guesses can, in principle, be applied our approach, we use Membership Inference Attacks (MIA) in our exper- iments. Specifically, we consider two state-of -ftee--ffth-attte the-artt art MIA algorithms (and their variants): the Likelihood Ratio Attack (LiRA) (Carlim al.) 2022) and the Robust Membership Inference Attack (RMIA) (Zarifzadeh et al. 2024 6 Given that both methods perform similarly with practical computa- tional budget, and RMIA shows greater robustness under extremely limited computational resources, present the RMIA results in the main paper and include results for both methods in the appendix.
'RMIA has two simplified versions with lower computational costs: the population attack (which omits the normalization denominator) and the reference-mode attack (which does not use population data). For the text data experiment in Appendix[] [ we use the reference-mee model attack due to the high inference cost of querying population data language models, done in Meeus et al. (2025). For DUCT with a single reference model, the population attack avoid strong membership-scone correlations introduced by reference models.
19

---

Published as a conference paper at ICLR 2025
Algorithm Dataset Usage Inference: Infer the proportion p of data in dataset X being used in the training of a given target model 0
Require:
Dataset X, Training algorithm T (including the data distribution D), Membership inference algorithm MIA. Target model (black-box access)
Train reference models for debiasing (as few as one model)
for to N do D - D
2:
3:
mn + Bernoullif |)|
4:
0j - +TT(DU u (m6) @ OX) Select data from X by m
5: end for
6: Save m I.....N = [mm.....,N]]
Debias
7: Run MIA for x4 E X on O for j E 1.... ON to estimate Pr(m = 1/m 0) and Pr(m
Im = 1):
|X|
aI and (0==0
Pr(m 1/m
N |X]
j=1
N XX
MIA(O, and m 01_1
Pr(m= 1m(m=1m=11 2
N X
Launch datase usage inference
8: for 1 to |X|oo do
9: mi MIA(z;; 0) Get individual membership guess on the target model
Debiasing each individual membership guess my;
m Pr(m 1/m 0)
p Pr(m 1 I 1) Pr(m 1/m 0)
11: end for
12: Output the proportion estimation p = - p.
20

---

Published as a conference paper at ICLR 2025
Likelihood Ratio Attack (LiRA) Define the scaled logits for point of model 0 as logit(2; 0)
log 1-fo(z) where fa(x) is the output probability at the ground -oound-truth truth class. LiRA (Carlini et al.
(ar
2022) generates a membership guess ri for a point I; by conducting a likelihood ratio test, utilizing
scaled logits the test signal, follows:
Pr (logit(z,; 0) WHun:: Pin) > 3(x..)
m, MIA(T;;0) 0)
(11)
Pr (logit(3r,; N) (Pout: )our))
where A (Htnn )n) and N (Hout; duut) are the Gaussian distributions of logit(æ;; 0) estimated on the N reference models trained with (in) and without (out) F;, respectively.
The essential idea LiRA is to compare the likelihood of observing the target model's signal given the distribution of models trained with the target point z against the distribution of models trained without z, where both distributions are estimated as Gaussian.
Robust Membership Inference Attack (RMIA)
m4 = (IA 0) =
2) ((z)) Pr((2]0)
(R 2) = (P((z:(0)
-1
(12)
Pr(x.))
Pr(z)
RMIA Zarifzadeh et al. 2024) compares the target point x; with a number of population data z from the same distribution, using the ratio Pr(z.]0)
Tr(x;;) as signal, where Pr(x:(0) is the prediction score (SoftMax) of output of the target model and Pr(x;) is the normalization term which is calculated by averaging Pr(x:]03 (0)) over all reference models 03 for 1, ,, 2N trained with or without wi. Σ> ii=1 Pr(zz) 0,z + Di=N Pr(z,10,) where 02,02.... 1N-1 are models trained ie.. Pr(x;) B BNN are models not trained with 24. For the offline setting, the Pr(x;) with 24 and fN..
be estimated only using models not trained with 24, i.e., 0y. ON+4..... B2N. It can be extended to launch membership inference using single reference model.
C.2 LIRA IMPLEMENTATION
Given the target points x E X and target model we implement the LiRA following Carlini et al. (2022) as follows:
1. Train 2N reference models. Each are trained on uniformly and randomly sampled half of X together with uniformly and randomly sampled half of the population pool. Therefore, each E X will have N reference models trained with it and the remaining N reference models trained without
2. For each z4. estimate two Gaussian distributions on reference models: one distribution on models trained with data record x; (denoted as W (Minn o )) and another distribution on models trained without v; (denoted as A (MMout 22 ( )). The mean μ and variance r² each distributions are estimated on the scaled logits of each data record E X over N reference models (trained with x or without æ, respectively).
3. For each data record I;, querying the target model get the scaled logit logit(z;; 0) and then compute the likelihood of observing on two distributions: Pr (Iogit(z:;; O) (pmm, o and Pr (logit(z;; ( (Hout: o d.
Pr (I) o.n) for each data record a;. Pe(logat( (0))) :8)] o...))
4. Compute the likelihood ratio LR(x;)
5. m; 1(LR(x.) > 3).
C.3 RMIA IMPLEMENTATION
In the original RMIA paper ZZarizaadh al. 2024). the population data utilized in Equation 12 could be other target data. Integrating RMIA scores directly into our aggregation approach would encounter issues of the differential rate of change as the true proportion of utilized data rises. To
21

---

Published as a conference paper at ICLR 2025
address this, we implemented the following adaptive version of RMIA focusing fixed target dataset: Similarly, given the target models & and the target points 2, E X, we implement the RMIA as follows:
1. Train N reference models. Each are trained on uniformly and randomly sampled half of X together with uniformly and randomly sampled half of the population pool. Therefore, each E X will have N₁ reference models trained with it and the remaining N2 = N Ni reference models trained without it. (When N 2 2, we can ensure Ni = N2. When = 1, we can do approximation as described below.)
2. For each E X, denote the reference models 01, 02......0. ON, as the models trained with Fy, and N +1, +1, 02, +2.............. NN +N₂ as the models trained without T2.
(a) (N >= 2) Compute Pr(a; Pr(ms)
N1 E fo (x++ N. Σ/ 12 (w.) where fa(z;) are the model's confidence value at the ground-truth class.
fo(2;)
(b) (N 1) We can use an linear approximation to compute the denominator. Assume I; not in the training set of the only reference model f,. we can estimate the probability of model trained with it as afo, (z;) + 1 - a. A similar approach applies in the reverse case. We select a 0.3 according to Figure in (Zaritzadeh et al. (2024). In this case, only N reference models are required for inference. (Note that the same N models can be shared among all x; as the models trained without 2;.)
3. (Optional) For t ...
T:
(a) Select z from the population pool such that it is in the training of N reference models and not in the training set of the remaining N reference models. Note that
z' & X since it is selected from population pool.
(b) Compute Pr(z118) Pr(z.)
fo(z")
N EN (=")) + & EN So_,e (=')a."
(c) Compute LR 2*) =
Pr(or10)) Pr(x)
Pr(z'100
for (ar,, pair. Pt(a*)
-1
4. For each E X, compute P2 2) >> 1) =1)=(=,=")>11]
T
5. m; = (P2((L((,, 2) > 1) > B).
C.4 SELECTION OF THRESHOLDS |
For the choice of threshold in both LiRA and RMIA, we sweep over all possible threshold to select the 3 as the optimal threshold maximizing the Youden's 's index (J = TPR -PR) Youden| (1950)
on reference modes, which provides the best balance between sensitivity (TPR) and specificity (1
FPR). Depends the computation of TPR and FPR in dataset usage cardinality inference problem, the threshold /3 can be computed either in the individual-level or dataset-level.
Individual-level threshold:
8. = arg max Pr 0) = 1/m- = 1) Pr(mp(2;0) = 1/m = 0)]
(13)
Here, (r, 0) = 1 = 1) is the True Positive Rate (TPR) computed under threshold / for each data point I;, and Pr(ma(z; (z; 0) 1/mm 0) is the False Positive Rate (FPR), similarly computed. Each of them is estimated on reference models.
Detaset-level threshold: The individual threshold may not be optimal when the number of samples is limited. In such cases, we can adopt dataset-level thresholds:
3 arg max((P(mii)) 1/m- = 1) Pr(má 1/m = 0)]
(14)
B
X
where Prx (mg 1 Im threshold f8, and Pr(mm (z; 0) = 1 m = 0) is the False Positive Rate (FPR) in Equation (6) computed in the same setting.
1) is the True Positive Rate (TPR) as depicted in Equation 7 under
Our all results presented in the main paper are based on dataset-level threshold and dataset-level debiasing due to the smaller sampling emor under limited reference models.
22

---

Published as a conference paper at ICLR 2025
C.5 EFFECT OF MIA CHOICES
While any membership identification method that provides membership guesses can be integrated into our approach, we empirically found that methods with stable performance against training randomness (e.g., less sensitivity to initialization and data variability) tend to perform better under our debiasing framework.
For example, RMIA combined with debiasing outperforms LiRA combined with debiasing on image classification models, particularly when the number of reference models is small (Figure This may be because LiRA relies solely on reference models for membership inference, making more susceptible to variability across trials. In contrast, RMIA explicitly incorporates population data in its test, in addition to reference models, leading to greater stability and lower prediction errors especilspy when the number reference models limited.
However, for text generation models, RMIA without population data (i.e., the naive reference model attack) performs better than RMIA with population data probably due to the normalization effect. This suggests that a more powerful attack does not necessarity yield better performance under debiasing. Instead, weaker yet more stable attacks (i.e., those producing comparable scores across models) benefit more from DUCI.
D IMPOSSIBILITY MULTIPLICATIVE GUARANTEE FOR DUCI
Why additive error instead of multiplicatiie error? Based on the standard packing argument (Hardt & Talwar 2010), providing a multiplicative guarantee for DUCI fundamentally impossible, regard- less of the method used. A simple example illustrates this: in the extreme case where the ground truth p = O, the estimate ğ must also be to keep the multiplicative ratio bounded. Below, we provide a more detailed explanation.
Let the ground-truth membership probability for record i in the training dataset be 7/- By definition, dataset sampled under probabilities (ri,-.- 7'. and (71 + To Tn + Tow)) can be identical 9 by the union bound. Consequently, with constant probability, with at least constant probability TO
a DUCI algorithm cannot reliably distinguish between datasets sampled with (7..... .r) and (71 2, ..., 7n + H), leading to an unavoidable additive error of J. Thus, for any fixed 2 1, as 7i + 0, either the multiplicative error Pi diverges to infinity or the multiplicative error P shrinks T
to zero, falling outside the range ())
On the other hand, as diseussed in Figure 3 additive enror provides consistent and meaningful metric for DUCI. Since DUCI is inherently a discrete counting problem, the focus is on incorrect counts, making additive error a more appropriate measure. For instance, protected dataset of size 10, an additive error of 0.1 consistently represents a single misprediction. In contrast, multiplicative error leads misleading results: single misprediction when p 0.1 produces the same ratio as mispredicting all points when 1.0, which is clearly unreasonable.
23

---

Published as a conference paper at ICLR 2025
E ANALYSIS OF POTENTIAL ERRORS FROM SIMPLIFYING RECORD-LEVEL DEBIASING TO DATASET-LEVEL DEBIASING
In Equation demonstrated that applying record-fevel debiasing process membership predictions before aggregation yields an unbiased estimator, X - [ E: P; for the ground-truth proportion p. However, due to the computational cost and notable sampling error when the number of reference models is small, instead estimate the TPR and FPR across the entire dataset X. We next show that the simplification in the debiasing process will not introduce errors in many practical sampling scenarios, such as uniform i.i.d. sampling (which are the common scenario considered in the long line of prior works in binary dataset inference literature listed in the Introduction section). Only in special cases where there strong correlation between the probability of sampling i-th point 7/s and its TPR; FPR,, there would be error. However, this can be effectively mitigated by subgroup debiasing. as shown in Table
In Equation 6 and Equation (7). we leverage dataset-level TPR and FPR (i.e., TPR = [X] Σ; TPR;
and FPR = [XT Σ; FPR,) to replace the individual TPR; and FPR, This simplification avoids the computationally cost (or large sampling errors) of debiasing each Pú- This simplification is justified because the proportion p is dataset-level statistic, as analyzed below.
To avoid confusion, we introduce and ç. the estimators under dataset-level debiasing defined by
m, FPR
Σ
is where P;
(15)
X
TPR FPR
We next prove I is an unbiased estimator of whenever correlation term between TPR; FPR; and "/j is zero: by definition (Equation 15) I we compute that
m, FPR
Ep
E
P1
Epp
E
(16)
X
X
X
TPR FPR
Plugging Equation 5
into Equation 16
we can get:
pp
TPR FPR X Σ (% TPR; (1 ()) -PR; FPR)
(17)
X E, b4 (TPR; - FPR.)]
(18)
x Σ, [TPR,- FPR,
Given Elp] = 1 E, Yss note that
[X]
(M1 E TPRR - -PP.))
Σ bts (TPR; - -FP::]]]
X Σ 71)
1
1 X
71
Σ W
TPR.
FPR;
(TPR. FPR,)
X
CTP: FPR,
(Here, use the term "correlation" informally to refer to empirical alignment between TPR; FPR, and 74. This is not statistical correlation between random variables, but rather a covariance- like quantity computed over deterministic values.) By further plugging definition for TPR and FPR in Equation (5) into equation 19 and Equation (16). we compute that:
Corr, (PR;; FPR,, 7R)
Epp
Epp]
(19)
TPR - FPR
The correlation term suggests that for many practical sampling methods (e.g., sampling without replacement, i.i.d. Poisson sampling), this simplification results in an unbiased estimator for expected utilization proportion because the correlation is 0. For special sampling methods, we can estimate TPR and FPR over subgroups of (rather than the entirety of) dataset X this ensures (empirical)
24

---

Published as a conference paper at ICLR 2025
unbiasedness, as discussed in Table as long TPR; FPR; is constant within each subgroup. This ensures that the correlation term for each subgroup is 0. providing a group-level debiasing approach. Note that the term "correlation" (slightly abused here) in the context describes how the value of a set of prefixed constants in the DUCI pipeline, determined. This is distinct from the correlation between membership predictions in Figure ovver random trials of the DUCI pipeline.
25

---

Published as a conference paper at ICLR 2025
F DERIVATION OF THE CONFIDENCE INTERVAL
Modern machine learning algorithms and their data sampling processes are highly randomized. To capture the intrinsic uncertainty and confidence in dataset usage inference, resort to the Lyapunov Central Limit Theorem (CLT) Billingsley (0017) to compute confidence intervals for our estimation method via aggregate statisties. We first introduce the Lyapunov CLT.
Definition F.1 (Lyapunov condition Billingsley (017) ). If a sequence of independent random variables (X:)== satisfy E[X;] = 14 00, [X; p.) 2 = of co, and for some d > 0:
lim 2-8 Sm
[[x; 44 2+ 0,
(20)
7-000
where 3/2 Ei= of O, the Lyapunov condition is said hold for {-,,, 1, L....,
Theorem F.2 (Lyapunov Central Limit Theorem Billingsley (2017). Let [Xii)= be sequence of
independent random variables with means μ4 = [[ and variances of = Var(X;). .. Ihe Lyapunov condition is satisfied, the following converges in distribution standard normal distribution:
(X, S 1(0.
8n
where Sn VE and S denotes the convergence in distribution.
Theorem
F22 states that if the individual statistics p.. : |||X| in Equation 4 are independent (without needing to be sampled from the same distribution), and the Lyapunov condition is satisfied,
their aggregation Σ|* | Pr converge to a Gaussian distribution as |X] increases. Therefore, by
applying Lyapunov CLT, we obtain the approximate 95% confidence interval for estimating Ely)..
x X
P p-ta/2
S Epp Sppl
ta/2
(21)
|X]
Here, p- p Σ!* Pr. 0.05, la/2 is the / critical value, and o X
Σ of. where of is
the population variance of p. which can be empirically estimated over the random trials.
However, considering the cost of launching this confidence interval inference, we porpose to use the sample variance to overestimate eix1 XXT (see proof in Appendix F.1. This gives us the following
approximate 95% confidence interval for estimating p.
s2
s2
P ta/2
<
Elpl
S P +all//2
(22)
X
X
Similarly, again here
X 1=1
Pi.
0.05, t./2 is the t critical value, and s²
I E.I (fr 2) where P
Pi-
F.1 PROOF OF OVERESTIMATION E[S?]
Lemma F.3. Given a set of independent random variables pr,... | with variances of.
(which may not be identically distributed), the sample variance 5² defined
X
[X|
1
Σ
D)
where
p
1
p
X
Σ o7 X
at least as large as the average variance
Proof. We start from the expectation of the squared differences:
Ei
-
B
(23)
26

---

Published as a conference paper at ICLR 2025
We next analyze each term.
For the first term, we have
E
+
1
(24)
where 14 = Epp for any i.
For the second term, we expand it as
XX|
[X|
1
E p
=E
Ps
Pj
[p.p11
(25)
X
X
Elp
+ Σ
(26)
1
0 + 1 + 14 Σ
Hj
(27)
X
j
X
1
0 11
2 2
+ 14
Σ
p4
(28)
X
/=1
[
1
0
2
14
(29)
XX
where equation 26 is by computing Elppýy as follows.
If Ep Elpe] of + x
if j. = = paزfij
For the third term, by definition, we have
X
X
2
1
1
E Epp Var + (E(p))²
of
(30)
|X| 2
|X|
By plugging equation 24
equation
29
and equation
30
into equation
23
we prove that
2
X
X
2
El(β: p)) of + 1
of
14
+
Σ
03
(31)
X
X
X
j=1
Thus
[
X
[X|
IX
[X
E[ -) =
2
+
Σ
2 +
b4
X
1=1
1=1
1=1
2
XX|
IXI Σ
|
03
+
Σ
|X
X
1=1
1=1
j=1
X
|
XX
1
1
(32)
X
X
27

---

be lower bounded as Published as a conference paper at ICLR 2025
Recognize that by Cauchy- Schwwar inequality, the terms involving μs follows.
x
X
X
X
1
1
Σ
X
M
X
==1
i=1
Thus Equation 32 can be immediately written as
Σ El - D) z I
Thus by the definition of 2 and by the additivity of expectation, we compute expected value of the sample variance as follows.
X
B[s²]
E Bi
- 1
X
i=1
(11
X
|
of
x
-
X
Therefore, s² is at least larger the average variance
Σ 7
[X
F.2 PROOF OF THE LYAPUNOV CONDITION
We next prove that pı, PPxi satisfy Lyapunov condition (Definition F.I.
Lemma E.4 (Boundedness). The random variables P1. 01.P2......7 x have bounded third moments.
X Σ FPR, are constants over the random trials with respect to the membership identification algorithm and T;, and by the assumption that TPR FPR (as stipulated in Equation (4)), we prove that Pr upper-bounded and lower-bounded by constant. Specifically,
Proof. Observe that the estimated global TPR [X| E TPR, and FPR
FPR
when t. 0,
P
TPR FPR
when nn 1.
TPR FPR
Since 9ú is bounded, i.e., Li /41= & C for some constant C, all moments are automatically bounded. Specifically, the third-order moments Ellè, 11131 < C3.
Lemma F.5 (Limit Property). The sequence P1, D2....... xa1i satisfies the following:
X
lim +
[la - m1)20] = 0 =1
E
2+5 [X|
where μ. = Elp..] = 7/ the mean of pi, of Var(p.) the variance of Pi, and sixi -!--| of is the sum of the variances.
Proof. Assume 8 for simplicity (a common choice). Denote M, as the third moments of Pr, i.e.,
M; = E [A: n4 3] From Lemma F.4 we have that there exists C; such that M, S C₁ for all i. Without loss of generality,
assume C₁ C2 =.. C = C. (A simple choice is to let
C' max(C;, C2.C...... .(.)))) Then, we have
X
X Σ
C|
Σ
M
<
C=
s3 X
I
|X
i=1
28

---

Published as a conference paper at ICLR 2025
Since six grows linearly with | we have s|x| of big-O notation, we have lxlg Ol O(v(X)) Thus,
O((X]). Therefore, from the algebraic properties
X
C|xI
C|XI C
C 0 |X
3
(O(Vix))3
O((V((99) 3(1))12
O(vIX)
co.
X
Since the independent random variables pr, p2...... paxd satisfy E[A] = % < 00 and E
< 00
for any 1,2..... [X] Lemma [.4). and have the limit property when
Eqquation (20 we finish the proof that {X,, -==1 satisfies Lyapunov condition. Therefore, by applying
[eemma
R.5
the Lyapunov CLT Theorem F.2 the sum Em converges in distribution a normal distribution as |X| increases. This ensures that the confidence interval derived remains valid even when the individual D; are not identically distributed but are independent.
29

---

Published as a conference paper at ICLR 2025
G PROOF OF THE SAMPLING ERROR OF DATASET-LEVEL DEBIASING
Under the mild assumption in Appendix F that membership predictions on different target data records are independent (experimentally validated Figure 4 that M M and M AA(O), Ty) are weakly correlated random variables for any i + i', where the randomness is over 0j and the randommess of the MIA algorithm), we next provide the formal proof:
Observe that Equation 66 and Equation 7 are empirical average of the membership prediction MIA(0 æz) over reference models f -.... .N and target data records T,, 1, | and that the reference models are independently trained on random subsets of the target dataset. Thenn under the independent assumption and by the additive property of variance for independent random variables, we compute the variance of Equation (6) and Equation 7 as follows.
1
Varl Equation
Var :)=1 .g9.)
X
1
Varl Equation
N 1
[X]
uni form
uniform
where 0j
{0
ON} and T;
Thus the sampling error of is asymptotically upper bounded by O /N.ixT
{.1,-.., Txx]
estimates Equation
66 and Equation
30

---

Published as a conference paper at ICLR 2025
H EXPERIMENT DETAILS AND ADDITIONAL RESULTS
H.1 EXPERIMENT SETTINGS
Evaluation setting: We consider an evaluation granularity of 5%, which is small enough for relatively accurate and meaningful dataset usage inference (DUI), To measure the DUI perfor- mance, we compute mean absolute errorElpp ~l] over 30 random trials for each proportion E {0.00, 0.05, 1.00}, and then evaluate the maximum error among all proportions, denoted as max, P Pll In each trial, we train the target model 0 on a freshly sampled p proportion of the protected dataset X (using uniformly random sampling special sampling), combined with randomly sampled remaining training data from data population pool (that does not overlap with X). The size of X in the experiments presented in Section 5.3 is 500, to maintain challenging and practical relative size. (In practice, the model is usually trained on a mixture of datasets, and the overall size of these additional datasets is typically much larger than the target dataset.) We then test the performance under size 5,000 and 25,000 and show in Section 5.4 that dataset usage estimation error decreases as the dataset size increases.
Model training details: We consider three model architectures: a standard 5-layer fully connected
network (FC-5), wide ResNet with width 2 (WRN28-2) 2) (Zagoruyko & Komodakis (2016), and ResNet-34 (He et al. | (2016). We use three benchmark datasets: CIFAR- (Krizhevsky et al. [2009). CIFAR- (Krizhevsky et al., 2009), and a scaled-down version of ImageNet & Yang 2013). We sample the protected dataset X from the training set of each benchmark dataset and use the remaining training set as the population pool.
(Le)
For target models, we train them on a freshly sampled random proportion of the protected dataset X, combined with randomly sampled remaining training data from the population pool (that does not overlap with X), such that the complete training dataset size for the target model is always half of the training set of the benchmark dataset. For example, the CIFAR- 10 training set consists of 50,000 images. Therefore, the training set size for the target model is 25,000.
For the reference models used for our method and MIA baselines, we train each one on half of X combined with remaining data sampled from the population pool, ensuring the complete training set of the reference model has the same size as the target models.
For all target models and reference models, trained them to achieve the following test accuracies: for CIFAR-10 10, 53% for FC-5, and 92% for WRN28-2; for CIFAR- 100, 68% for WRN28-2, and 65% for ResNet-34; for Tiny-Im 58% for WRN28-2 and 54% for ResNet-34.
H.2 IMPLEMENTATION DETAILS OF BASELINES
MILE baselines The MLE baselines require to estimate the distribution for the observation of models trained on each proportion of X. To approximate the distribution of olp, empirically, we train N reference models on freshly sampled random p-fraction of the target dataset X combined with remaining data sampled from the population pool for each of the 21 possible proportion values (i.e., p 0.00, 0.05... 1. 1.00). employing the same algorithm T that was used for training the target models (in total 2 reference models). The complete training set size of the reference model the same as the reference models used for our method. Regarding the two choices of observation in the MLE baselines:
Joint logits of all records in target dataset (bigh-dimensional):: Building prior research (Car lini et al.) (022) which demonstrated that scaled logits for individual data points can serve as effective observations and empirically follow Gaussian distribution, naturally extend this approach to a dataset-wide scale. We concatenate these point-wise logits into high-dimensional joint logits and model their distribution using a multivariate Gaussian.
Pr(olp = g) = Pr(logit (X; ()) Eq)):
where (ogit(X;0) 0) is the the logits of model 0 on all records in X, and Ma and E, are its empirical mean and covariance matrix across reference models trained with q proportion of X.
Averaged logits of all records in the target dataset dimensiiin:::::: To reduce the dimension- ality of the observation (and thus enable more accurate empirical mean and covariance estimation), we additionally consider the average of the logits over the dataset X, following the approach
31

---

Published as a conference paper at ICLR 2025
in Maini et al. (2021). The likelihood is then modelled as the following one-dimensional Gaussian distribution.
logit(x; (()
Pr(olp -) Pr
X
ZEX
where My and og are the empirical mean and variance of the averaged logits of all records in X across reference models trained with proportion of X.
The step-by-step procedure as follows:
1. For each {0.00, ...00.................... 1.00}:
(a) Train N reference models a proportion q of X combined with data randomly sampled from the population pool.
(b) Query the reference models using X to obtain observations de.g., g., the average logits)
for all data in X and estimate the Gaussian distribution on these observations.
(c) Query the target model using X obtain the target observation Otar-
(d) Calculate the likelihood of observing Our given the distribution for q.
2. Select the q with the highest likelihood of observing Our-
MIA baselines To demonstrate the importance of debiasing, also compare methods against two intuitive baselines: the direct aggregation of MIA guesses and MIA scores. (Some might think that MIA scores indicate the probability confidence that data point is member, which could be used to estimate dataset usage.) We employ the same 2.N reference models as our method, with each model trained on half of dataset X:
MIA Guess: This involves directly counting MIA membership predictions, where the proportion is calculated as the fraction of member predictions: ý = μ E!* 2 m.s For a fair comparison, we
use the same MIA algorithms that were employed in our methods.
MIA Score: This takes the average of MIA confidence scores across all data points: ń
XXT Σ è, where è; is the confidence score of RMIA (i.c., the probability of observing that i-th record's s likelihood to be member is greater than randomly sampled population data points).
The step-by-step implementation of the MIA algorithms is provided in Appendix pendix C.3 These implementations follow the standard procedures described in the original papers.
+ and Ap-
H.3 FORMING SUBGROUPS BY TPR, FPR.
subgroup
20.0
subgroup
subgroup
17.5
subgroup
subgroup
15.0
subgroup
frequency
10.0
TPR, FPR.
Figure 6: Samples of TPR; FPR, value for 6 subgroups.
32

---

Published as a conference paper at ICLR 2025
H.4 RESULTS USING DIFFERENT MIA
In the main paper, we present the results of our method using RMIA (Zarifzadeh et al., 2024) due to its superior performance at limited number of reference models. Additionally, we have implemented another state-of-the-an MIA algorithm, LiRA (Carlini et al., 2022). (Comparisons between RMIA All results in the main paper, when including the performance
and LiRA can be found Appendix CD of LiRA, are provided in this section.
Comparison under fixed computation budget Figure 7 presents the complete version of Figure including the performance of the MIA baseline using LiRA and our method using LiRA. Our methods (red and pink) are clearly more accurate in proportion prediction than all baselines. The comparisons between method using LiRA and direct predictions from LiRA further confirm the effectivenesss of our debiasing method. While our method using RMIA is more robust with a smaller number of reference models compared using LiRA (analyzed in Appendix C.5), the MIA Baseline (RMIA) and MIA Score (RMIA) show less sensitivity to the number of reference models, possibly due to their inclusion of population data. Overall, they perform better with fewer reference models than the MIA Baseline (LiRA), which is consistent with the results reported in Zarifzadeh et al. (2024..
0.6
MLE Join-LLgit
MI.F Aug-Togi
Aggrogation
max,ejuu El pill
Aggreguzion RMIA
MIA Baseline (LiRA)
0.4
Boweline (RMIA)
MIA Sove
0.2
600
120
140
Number of Reference Models
Figure 7: Maximum mean absolute error (MAE) of all methods under an increasing number of reference models. The grey dotted line represents the minimum computational budget required for MLE methods (2 for each proportion). Notably, our method can yield meaningful predictions even under limited computational budget (e.g.. single pair of reference models).
Confidence interval Figure gladds the confidence interval of our method using LiRA (correspond- ing Figure the main paper). The top figure demonstrates the effectiveness of our methods using both membership identification methods, while the lower figure illustrates that our 95% confidence interval predictions are consistently tight.
1.00
Freuuecy
0.60
Aggregation RMIA Aggregation LiRA
0.20
0.14
Longth
0.10
0.08
0.06
True Percentage of Used Data
Figure 8: (Confidence Interval) Top: Frequency of the true proportion p falling within the predicted 95% confidence interval for different p. The frequency was computed across 30 interval predictions for each p. Bottom: Average width of the predicted interval over 30 tests for each p.
33

---

Published as a conference paper at ICLR 2025
H.5 PERFORMANCE OF OUR METHOD IN DETERMINING BINARY DATASET USAGE
While methods restricted to binary predictions under an all-or-none dataset usage scenario cannot ensure consistent predictions for partial utilization (Figure D. method providing fine-grained estimates can naturally be reduced to solve the binary problem.
To illustrate this, consider the null hypothesis (Ho == =' +) used in prior binary dataset inference literature. For different contexts, s and s' can take the following forms:
Dataset Inference Maini et al. (2021): s and s' represent the distances to the decision boundary measured private dataset and population dataset, respectively. This hypoth- esis assumes that if the model was trained on the private dataset, the distance measured on the private dataset would exceed that on the public dataset.
2. LLM Dataset Inference Maini et al. 20024): and s' are the weighted aggregations of 52 MIA scores over the private and population datasets. This hypothesis assumes that merged MIA scores would be significantly higher for members than non-members over enough samples.
Backdoor Watermarks Guo et al. (0033;; Li al. (202)); Tang al. (202 ; Hu al. (2022); Li et al. (202bb); Wenger et al. (2022); Sablayrolles et al. (2020) and s' are the confidence score on the target label given backdoored inputs and given clean inputs. This hypothesis assumes that model trained on poisoned dataset (if successfully backdoored) will assign higher confidence to the target class when triggered, but not for clean inputs.
For DUCI, straightforward simplification the dataset inference problem can be made by set s' = 0, and serves as threshold, which may vary depending on the data type. Table report the performance of our method adapted for solving the binary dataset inference problem.
Table 6: Comparison of p-values between DUCI and binary dataset usage algorithms for determining whether a dataset X (size 500) has been used. The complete training dataset of the target model has size of 25,000. For p-values, a smaller value for Dataset Used is better, while larger value for Dataset Not Used is better.
Methods
p-value
Dataset Used +
Dataset Not Used t
Backdoor Watermark (poison 30% of X)
7.10 10-5
0.334
Backdoor Watermark (poison 100% of X)
6.18 X 10-54
1.000
Dataset Inference
7.27 x 10 10-10 10
0.937
Ours
3.15 10-51
1.000
Consistent with the performance shown in Figure I all methods can perfectly solve the binary dataset usage problem when the significance level is set to common thresholds such as 0.05 or 0.01. This challenge becomes especially critical and significantly impacts performance when the dataset is not fully sampled or when the protected dataset's relative size is small. Our method performs exceptionally well, achieving comparable results to backdoor watermarking when the entire dataset is poisoned. In principle, the performance of Dataset Inference should be close to our method; however, the slight drop in performance may be attributed the choice of signal, as the loss-based score is less distinguishable in distribution than the likelihood ratio-based score. We did not compare with (Maini al. 2024) as combining multiple MIAs is orthogonal to our approach. Our method can debias any number of MIAs using the same reference models without retraining, with combination possible after debiasing needed.
Finally, it is important to note that directly comparing the reported error of DUCI y 0 and p = to that of dataset inference is inherently unfair. DUCI predicts a continuous value, whereas dataset. inference is simple binary classification task.
34

---

Published as a conference paper at ICLR 2025
1 ON (THE POSSIBILITY OF) IMPROVING DATASET USAGE INFERENCE VIA FAIR-WISB "CORRELATION"
Our method in Section treat individual records in the target dataset separately, and thus implicitly require "independence" among records to perform well. Will the ignorance of "correlations" between records predictions make our method sub-optimal for the problem of dataset usage inference? If the "correlations" between records predictions exists, how can we debias mem- bership predictions? In this section, we explore the room for improving our aggregation method via exploiting correlations among different records in the target dataset. We first show that under natural datasets (without manually crafted data), the room for improvement is small as the correlation between records prediction is almost zero. However, in Appendix [.2] we demonstrate that with a dataset containing specially designed canaries (i.e., correlation exists), our method can be extended to second-ord -odder debiasing given pairwise membership predictions, enabling more accurate dataset usage inference.
1.1 ON THE POSSIBILITY OF COVARIANCE REDUCTION
For the sake of understanding the role of correlation, let now focus the simplest scenario of inferring the usage of a dataset with two records. Under such settings, the mean squared error (MSE) for our proportion estimator p = 지 Σ [X|
Li=l P. in Section 3.1 is given by:
MSE(2) =E (6 + pm) -p)
27
(33)
Var(pi) 1 Var( + Cov($1,
1
(34)
where the last equality is by the unbiasedness property of 91 and p2z This naturally motivates us to ask: Can we reduce the correlation (i.e., covariance term defined Equation (34) between data pairs in the target dataset, thereby reducing the error of dataset usage inference estimator? To answer this question, we evaluate the magnitude of correlations among data records in the target dataset, and plot the histogram in Figure 4 We observe that majority of data pairings incur near-zero covariance values. This is in line with prior works Pillutla et al. (2033) that observe small correlation between membership guesses on different data records. Only a small number of outliers (less than 0.01 %) show positive covariance values larger than 0.05 these pairs sometimes consist of records very similar to each other (examples are provided in Figure 10). These trends imply that the room for improving our aggregation methods via reducing covariance between data pairs generally very small. It is an intriguing open problem to whether our method could be further improved if the data owner allowed to enforce high correlations among different data records, e.g., by modifying the target dataset.
I.2 ON THE POSSIBILITY OF PAIR-WISE BLAS REDUCTION "SECOND-ORDER DEBIASING)
- Ep,, * 0), is it still possible to aggregate group-level statistics unbiasedly to predict dataset usage? In this section, we provide an example of second-order debiasing (when all records in the target dataset are uniformly randomly sampled, i.e, 7/ = for any record when the ground-truth proportion is p).
If correlations between membership predictions exist (i.e.. -
Debiasing pair-wise membership guesses to estimate dataset usage proportion Suppose a mem- bership identification method is able to predict pair-wise memberships m,m for any pair of points (z), where i, E 1,2, 2,..... [X]. J. We now apply it to randomly sampled data record pairs from the target dataset. For each data pair (x,, 2y)) denote m, and m, as their ground trouhd-truth memberships, and denote m; and m, their pair-wa wise membership guesses (by definition m, mj, m, m, E {0. 1)). Let Pm, = 00), P(m,m, = 01), P(m.m = 10) and P( = 00) be the frequency of each pair-wise guesses.
35

---

Published as a conference paper at ICLR 2025
To estimate the dataset usage proportion, need to perform debiasing operations (similar to Section on the observed pair-wise membership guesses. For this, we first estimate the following matrix M (i.c., all off-diagonal values in this matrix represent enror types, analogous TPR and FPR) of conditional distribution of the pair-wise membership guesses m.mi given different ground-truth pair-wise membership values m,mn, empirically trained reference models (of which we know the training datasets).
P()0)00) P0)))00) P(01)01)
P(00)01) P(00)10) P()0
M
P(11)00)
P(1
where for brevity we denoted P(a b) to P(m,m = almam = b). and the randomness is over the training algorithm and the random data pairs.
Therefore, by the total probability, we have
P Pmmgm 00)
P(ingin 00)
P Pimany 01)
Pimmn 01)
M
(35)
P(m.m = 10) P(m,m, 10)
P P(mmm 11)
P(mmmn 11)
To estimate the dataset usage proportion, one only need to observe that when a randomly sampled
q g-proportion of the target dataset used for training, the ground-truth pair-wise membership values
follow the following distribution: P(m,m, = 11) = 4. P Pm,,m
01) P(m.m = 10) =
(1 - q)q, and P(m,m, = 00) = (1 - 2)². By plugging them to Equation (35) we can estimate the dataset usage proportion by finding a y that enables the closest observations the empirically observed pair-wise membership guesses, as follows.
(1 9) 27 (1 - 1)
P(m,m 00)
P(mmmn 01)
2
p = arg min
M
q(1 9)
Plmaing 10)
q²
P(immn 11).
1.2.1 EXAMPLE OF PAIR-WISE DEBIASING GIVEN LIKELTHOOD-BASED MEMBERSHIP PREDICTION
We begin by selecting the most basic pair-wise statistic wise likelihood (analogous to loss and considered the weakest MIA signal)-as as the basis for generating pair-wise membership predictions. As baselines, we include likelihood-ratio-basee
methods (a stronger likelihood-basse signal), per- point MIA (LiRA Carlini et al. (2022)) ), and the best signal (RMIA). Our results demonstrate that, under second-orde -order debiasing, even the weakest signal performs comparably to the aggregation of membership predictions using the best first-order signal.
Pair-wise membership inference The idea of pair-wise membership inference is to design test that can simultaneously guess the membership status of pair of points (z,, .ry) for any i, We can use the general maximum likelihood estimation to design wair-wise test, as follows.
2, |X
mam arg S (00,01,10,11)
P(olm,my
(36)
where we consider the observation o as averaged logits of the data pair (I; xT) and model the distribution of this observation o when m.mj s for each E {00,01 10, 11} as a multivariate Gaussian, similarly as Equation [TO..
10ehe second-order debiasing is natural extension of straightforward debiasing since under uniformly
[1 g)]
P(m, = 0)
random sampling, method in Section is solving arg min, E M principle.
in
P(m. = 1)
36

---

Published as a conference paper at ICLR 2025
Performance of the second-order debiasing We refer method presented in the main paper, which is based on the aggregation of debiased individual membership guesses, as the aggregating individual statistics method (Agg Individual) distinguish it from the pair- wise test. In Figure observe that the pair-wise test enables better performance (smaller bias) for proportion estimation compared to the Agg Individual method using LiRA. This suggests (1) the second-order debiasing is accurate, and (2) that pair-wise statistics (naive likelihood value) may capture more information than the corresponding individual statistics (stronger likelihood ratio).
However, we observe that the pair-wise test does not outperform the RMIA-based Agg Individual method (which is stronger than the LiRA-based method). We hypothesize that this is because the score (likelihood based rank score) used in RMIA is much stronger than the score used in LiRA and our pair-wise wise test. The RMIA score captures more data information, bridging the gap between first-order and second-order statistics. It is an intriguing open question whether we can design more powerful pair-wise tests using stronger MIA scores like those used in RMIA II
Additionally, it is important to note that the current pair-wise test requires much higher computational cost than the aggregation of individual statistics due to the computation of MLE for each sampled data pair. The computational cost for inference of the pair-wise test is ((X)), while our Agg Individual method is ([) Therefore, identifying ways to reduce this cost could be another direction for future research.
1.0
Inferred Percentage of Used Data
0.8
mm
0.6
0.2
Pair-wise modelss
Individual LIRA x models)
0.0
Agg individual RMIA 12 models)
0.0
0.05
0.15
0.2
0.3 0.35
0.45 0.5 0.55 0.65 0.75
0.8 0.85
0.35 1.0
True Percentage of Used Data
p
Figure 9: Comparison between the dataset usage inference using pair-wise statistics and individual statisties (red and pink) when only 4 pairs of reference models are used. For the pair-wise test, follow Appendix I.2 and aggregate the debiased pair-wise membership guesses over 20000 randomly sampled data pairs the target dataset.
Uheere is straightforward method for adapting RMIA scores for pair-wise testing.
37

---

Published as a conference paper at ICLR 2025
Figure 10: Examples of pairs with large covariance.
38

---

Published as a conference paper at ICLR 2025
J CASE STUDY: QUANTIFYING BOOK COPYRIGHT INFRINGEMENT
The issue of copyright infringement in books can be considered specific case of data usage inference: given a model that likely infringes on book copyrights, the author or publishing house would want to determine the extent to which their content is used. We conducted
experiment using pre-trained GPT-2 (Radford et al., 2019). For a rigorous setting, we selected books from the BookMIA dataset constructed by [Shi al. (202)). This dataset contains 50 new books with first editions in 2023, which could not have been included in the pre-training data of GPT-2, and 50 old books- be memorized by ChatGPT-whoe memorization status by GPT-2 uncertain. To prevent data contamination and distribution shifts between members and non-members Zhang et al. (2024), we only use the 50 new books from the BookMIA dataset [Shi et al. (2023) for evaluation. These books are partitioned into a protected pool of 30 books and population pool of 20 books, with each sentence treated as a separate entry.
Target Model: We consider six different infringement proportions E {0.0, 0.2........ 1.0) and trained m 4 target models, each fine-tuned on different proportions of protected books.
Specifically, for each of the m target models, we randomly partition the protected pool into six disjoint sets of books So, So.2, S0.4, So.6, So.s, and S where each S, contains n = 5 target books. The fine-tuning dataset for each target model is then constructed as follows:
1. Protected books: For each protected book X E Sp for p E {0.0, 0.2, 0.4, 0.6, 0.8). we select the beginning p proportion of the sentences from the protected book X.
2. Outsourced books:We add another 5 randomly sampled books from the 50 old books (as the remaining dataset from unknown sources).
Therefore, for each these m = 4 target models, we have n = 5 target books with p proportion of content included in the training for each possible proportion value p E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0). This setup results in a total of 20 trials for each p.
Note that in this experiment, the used sentences are sequentially chosen rather than uniformly randomly chosen as in Section 5.1 to better reflect practical scenarios where text continuity in a book is preserved.
Reference model: We train two reference models, each with a fine-tuning set that includes two parts:
1. Protected books: Each model trained on a random half of each book in the protected pool of 30 books. More specifically, both models trained on all 30 books, but each model uses only half amount of the sentences from each book without overlap.
2. Population books: Each model also includes half of the sentences from 2n 10 randomly selected books from the population pool of 20 books as the remaining data.
This ensures that 1) for each sentence used in training the target model, one reference model is fine-tuned on it while the other is not, and 2) among the remaining data (exclusive the target books), we can select sentences that are present in only one of the reference models as the population data (used for computing the MIA scores only).
All target and references models incur training loss of around 3.2 and test loss of around 4.0, which is close the loss reported by Prat (2023). The time for fine-tuning single model (both target and reference models) for 3 epochs on single NVIDIA GeForce RTX 3090 GPU is approximately 20 minutes.
Experiment setting We use the RMIA without population data as the underlying MIA algorithm (following the implementations in Meeus et al. (2025); Zarifzadeh al. (2024) for our methods and MIA Guess baseline due to the significant costs of population data inference, and use the RMIA with the above-mentionee population data only for the MIA Score baseline. We select the optimal MIA threshold as described in Appendix C.4 on reference models! We consider MIA Guess and MIA Score as our baselines:
12 "SSpccificllly, among the two reference models, the one used as the target model for threshold determination debiasing will not be used the attack model.
39

---

Published as a conference paper at ICLR 2025
1. MIA Guess: For each book X with sentences {c]][1 we perform MIA on each chunk
Ci to obtain an individual membership prediction mi. We then compute the ratio
X 1= rig.
2. MIA Score: Similar to MIA Guess, we conduct the same process but, instead of obtaining binary predictions, directly aggregate the MIA scores described in Section 5.2
For our methods, we repeat the debiasing process described in Algorithm I given by B(m/mm = 0) and E(m/mm = 1) estimated on top of the membership guesses provided by the MIA.
The inference time over 30 target books for all methods, including the estimation of P(m = 0)
and ( m 1), on a single NVIDIA GeForce RTX 3090 GPU takes 408 seconds.
Results and analysis Although MIA on text generation models has been explored since (Song Shmatikov 2019), previous research typically conducts MIA by querying individual texts and gathering single statistic to infer whether the entire collection is part of the training data. A few discussions on book copyright infringement issue in prior works (Radford et al. 2019) also focus on this Yes or NO question, which is just the binary version of our dataset usage inference problem.
Our experiment shows that the naive aggregation of membership inference methods struggle to distinguish between different levels of infringement, particularly when the infringement minimal. As shown in 7 the MIA baseline tends to over-rely on the behavior of the half-half split reference model, leading to large errors when the true usage proportion deviates from p 0.5. In contrast, method effectively mitigates this issue. For example, MIA Score and MIA Guess baselines incur maximum
absolute errors of 0.5 and 0.335, respectively, across tested proportions, whereas method achieves a much lower eтrоr of 0. 061.8. Notably, while our estimator is provably unbiased under independent sampling-a condition violated in this correlated, contiguous sampling experiment- still performs reliably, demonstrating its robustness in estimating the extent of copyright infringement under special sampling conditions.
Interestingly, we also observe generally higher error variance for our method across different propor- tions on text data compared to image data. This is because the variance our estimator increases as the gap TPR- FPR decreases. Just for illustration, assuming each record is independently sampled with equal probability, the variance of the estimator
Varlm]
p(1 p)
Varlp)
(TPR - FPR)2 (TPR FPR)2
(TPR - FPR)2
Given that memberships text data are typically less distinguishable due larger training corpora and more severe model forgetting, it tends to have a smaller TPR FPR, and thus higher estimation variance, as reflected in the larger MAE variance.
More surprisingly, we observed that RMIA with only reference models more stable than complete- version of RMIA (with population data). The broad distribution of population data makes RMIA scores less stable across models. Instead, the normalization term computed over reference models plays key role in improving the effectiveness of debiasing. That's saying, more powerful attack does not necessarily yield better performance under debiasing. Instead, weaker yet more stable attacks (i.e., those producing comparable scores across models) benefit more from DUCI.
Table 7: Mean Absolute Error Ellè: pl] for different values of p. The lowest error under each proportion p is highlighted.
proportion p
MIA Guess
MIA Score
DUCI (Our Method)
0.0
0.3350
0.5019
0.1684
0.2
0.2396
0.4029
0.1573
0,4
0.1202
0.2381
0,0833
0.6
0.0439
0.1028
0.0316
0.8
0.1286
0,0720
0.0453
1.0
0.2448
0.2189
0.0568
maxpe|0.1] Bli pl]
0.3350
0.5019
0.1684
40


Reviews:
[{"review_id": "a4cAoFhPwo", "reviewer": "Reviewer_mJ1c", "summary": "The paper formalizes the problem of dataset cardinality inference, which aims to measure _how much_ of a given dataset has been used in model training. The paper shows how existing out-of-the-box membership inference methods fail to solve this problem and show how that can be remedied with de-biasing. Experimental results show the benefits of the proposed approach.", "questions": "- **Poor results around $p=0$**: The results of Table 4 show that the method is not very reliable around $p=0$. This would make it unsuitable to answer the question of _if_ a dataset has been used. Are any modifications possible to adapt the proposed method to [dataset inference](https://arxiv.org/abs/2406.06443)?\n\n- Further, how does the proposed method work if our goal is to provide a multiplicative guarantee of the form that $\\hat p / p \\in (1/c, c)$? These would be more realistic in the small $p$ regime.\n\n- Like differential privacy is designed to protect against membership inference, are there any provable protections against DUCI?\n- Why do you think MIA Guess fails to work?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper introduces on a very important problem and gives some solid baselines to tackle it. \n\nThe method, error metrics, experimental settings, baselines, and evaluations are thoughtfully designed (in general). For example, I particularly appreciate:\n- the extra mile effort in deriving confidence intervals;\n- the use of dataset selection methods in experimental evaluations;\n- an analysis of why the confidence intervals are large around $p=1/2$;\n- the experimental setting of book copyright infringement;", "weaknesses": "The main drawback, in my opinion, is that there are approximations involved in deriving the confidence estimates, making them potentially incorrect. There appear to be two approximations (please correct me if I'm mistaken): \n1. replacing $TPR_i$, $FPR_i$ with a single TPR/FPR across all samples, so the de-biasing is not exact;\n2. assuming independence of $\\hat p_i$'s to compute the confidence intervals.\n\nWhile the authors empirically show in Fig 4 that the correlations in item 2 above are small, it would be nice to see that the bias induced by item 1 is also not too large.\n\nFinally, I would have liked to see some approaches for rigorously correct (asymptotic or non-asymptotic) confidence intervals in addition to the heuristic ones used here. I believe that the XBern confidence intervals given by [Pillutla et al](https://arxiv.org/abs/2305.18447) can be used (XBern confidence intervals for $TPR_i$ and $FPR_i$ can automatically adapt to the correlation, leading to better intervals for $\\hat p$).\n\n**Other comments**: \n- I do not understand the derivation of footnote 1. It would be nice to expand on it (possibly in the supplement). \n- Figure 2 can be clearer if the x axis is in log scale\n- Missing relevant refs: [Kandpal et al](https://arxiv.org/pdf/2310.09266) for membership inference of users (groups of data) and is related to dataset inference, [Vyas et al](https://arxiv.org/pdf/2302.10870) for copyright protection, [Zhang et al.](https://arxiv.org/pdf/2406.15968) for a recent MIA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DARtpBdRGs", "reviewer": "Reviewer_THsi", "summary": "In this manuscript the authors identify key issues of current techniques that aim to ascertain if a dataset was used to train a Machine Learning model. To alleviate these problems:\n\n1. The authors formally define the concept of Data Usage Cardinality Inference (DUCI). The authors state that, compared to other binary types of inference, DUCI better reflects real world scenarios, where models are trained on fractions of different datasets. \n2. The authors propose a way of de-biasing current models that estimate individual membership, i.e. if one individual sample was part of the training dataset. Then, they propose to use these unbiased estimators to compute the overall proportion of the dataset used for training. They also present an asymptotic method to design a confidence interval for this overall proportion used for training. \n\nFinally, the authors provide some numerical experiments where they compare the proposed procedure with four other adapted techniques that also perform DUCI. Throughout their experiments, the authors' de-biased method outperforms the other four techniques.", "questions": "What is the the sampling error mentioned in Line 228 in this particular setting?. \nWhat is the definition of weak independence in Line 269?.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Identifying the dataset used to train a Machine Learning model could have a direct impact on privacy rights or copyright infringement, as mentioned in the Introduction. Hence, I think that this article deals with a relevant problem. I also appreciate that their proposed procedure is cost-effective and intuitive. In my opinion, there is a lot of merit in noticing that Member Identification methods suffer from biases and then presenting an straight-forward tool to address this issue.", "weaknesses": "I think that the main weakness here is the presentation. A lot of times the authors describe mathematical objects by vaguely saying what they are or make rushed arguments. However, this approach is not intuitive enough to give any insight about the matter nor formal enough to have any actual meaning. This overshadows the interesting contributions made in this paper.\n\nFrom Lines 108-115, I wonder what is \"a number of population data\", what is $\\theta(x)_y$ (as this is the first time they use this notation with y as a subscript; in fact, what is y?). What is the reference model modelling, i.e. are these models for membership inference or are these models that represent a real world classifier or regressor?. In Line 285-286, it is difficult to understand what \"the probability of observing that i-th record’s likelihood to be a member is greater than randomly sampled population data points\" means. This is not even relevant to understand the paper main contributions, so it should be remove it the authors are not willing to explain it clearly or should be rewritten, if they prefer to do so. \n\nDependence/correlation of records is handled in a confusing manner. In particular, the authors pose the question \"Will the ignorance of “correlations” between records make our method sub-optimal?\" in Lines 490-491. The answer here is clearly \"Yes\", as the authors themselves have stated in Lines 444-448 that under special sampling one should divide the dataset into subgroups and then de-bias using the TPR and FPR within each subgroup. However, this additional step, which accounts for possible high-correlation, is not carefully mentioned in Section 4, so I would not assume that this is a fundamental part of their proposed technique. However, it reads as if Lines 489-497 argue that correlation between records is not an issue and that there seems to be limited potential to improve their method in this regard. Maybe the authors here are considering different methods of sampling or different settings but this is not clearly stated in Lines 489-497. This is something that should be addressed. \n\nRegarding the numerical experiments, there are two things to consider: two methods were adapted from Individual Membership Attacks and the other two baseline estimators were inspired by maximum likelihood estimation but rely on additional modeling decisions, like assuming some joint/mean logits follow a normal distribution. Although this choice is based on a theoretical result, as mentioned in Line 298, it is not clear to me that this assumption would not hinder the performance of these baselines. The authors do mention in Line 276 that they use MIA Guess and MIA Score \"To demonstrate the importance of debiasing [...]\", but I think they would need to address why the MLE with joint logits is presented as an idealized baseline. At least the MLE with average logits has good performance in various experiments, so there is evidence in favor of presenting it as an idealized baseline. However, I feel like the experiments in this paper do indicate that the proposed method performs well, under the scenarios considered here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3H8nvDJW6h", "reviewer": "Reviewer_H5r7", "summary": "This paper presents a new variation of membership attack: estimating the fraction of data from training set. Unlike normal MIA, which determine individual membership, the proposed task estimate the data usage directly. The proposed algorithm is based on the fact that the  unbiased data usage estimator can be written as a function of FPR and TPR (Eq. 6) of MIA. In other words, the proposed estimator can adapt any existing MIA attack with FPR and TPR evaluation. The experiments demonstrate the effectiveness.", "questions": "1. In Figure 3, the length of confidence interval seems to be large compared to the absolute error. Is this true?\n\n2. Would this debiasing method downgrade the test power?\n\n3. Is there any connection with auditing differential privacy?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a new task in privacy attacks. The main contribution is a practical and scalable data usage estimator that could encourage further research in this area.", "weaknesses": "My main concern is that this method requires known training set for estimating FPR and TPR. In practice, the train set is usually private (see following reference Zhang at el 2024).\n\n\n\nZhang, Jie, Debeshee Das, Gautam Kamath, and Florian Tramèr. \"Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data.\" arXiv preprint arXiv:2409.19798 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fBQ05a967J", "reviewer": "Reviewer_Tzw7", "summary": "Given a dataset, this paper presents an algorithm (DUCI) which estimates the proportion of that dataset used in the training of a model. The algorithm estimates the false positive rate (FPR) and true positive rate (TPR) of the membership inference guess across the entire dataset to avoid the accumulation of errors that occurs when estimating the FPR and TPR of each individual in the dataset. They conduct experiments to compare the performance of their algorithm (DUCI) against traditional membership inference baselines and an idealized, computationally inefficient MLE baseline. They also analyze the performance of DUCI and membership inference baselines under special sampling conditions and varying dataset sizes.", "questions": "If TPR=FPR, how does this affect the debiasing results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well motivated and addresses a gap in the literature by taking a fine grained approach to the data usage problem.\n\nThe proposed approach (DUCI) is significantly more computationally efficient compared to previous approaches.", "weaknesses": "I don't see any major weaknesses. It would be nice to show a comparison between DUCI and SOTA “binary” data usage algorithms for the specific case for when p=1 and p=0 to demonstrate that DUCI still has comparable performance to “binary” data usage algorithms in these specific cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xwwMCCq5ia", "reviewer": "Reviewer_BehN", "summary": "This paper proposes an algorithm framework that can figure out whether data points in the data sets are used to train a model. This problem seems interesting, and there are some works on a similar problem called membership inference. The authors propose that instead of predicting each member by {0,1}, a better way is to predict a probability in [0,1] and design an algorithm for DUCI based on this idea. The authors also did many experiments comparing their method with several baselines on errors and confidence intervals.", "questions": "The authors mention two possible improvements in Appendix G. I think this paper would be really strong if the authors could be more concrete on how to apply one of the ideas to their algorithm and show some results.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper studies an interesting problem and proposes a new algorithm to predict a probability in [0,1] instead of {0,1} for the problem. Though the authors proposed a specific algorithm, the technique here can be used for any algorithm that serves the purpose of membership query. This paper is well-written and easy to read. The authors also did experiments thoroughly by comparing with baselines and on different datasets.", "weaknesses": "Though this paper has some novelty, the technique here seems to be quite simple and straightforward. It is hard for me to say this paper has good contribution confidently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)