On the Robustness of Dataset Inference
Authors: Sebastian Szyller
Rui Zhang
Jian Liu
N Asokan
Venue: Submitted to ICLR 2023 (2023)
Keywords: ownership verification
model extraction
model stealing
fingerprinting

Abstract: Machine learning (ML) models are costly to train as they can require a significant amount of data, computational resources and technical expertise. Thus, they constitute valuable intellectual property that needs protection from adversaries wanting to steal them. $\textit{Ownership verification}$ techniques allow the victims of model stealing attacks to demonstrate that a suspect model was in fact stolen from theirs.
Although a number of ownership verification techniques based on watermarking or fingerprinting have been proposed, most of them fall short either in terms of security guarantees (well-equipped adversaries can evade verification) or computational cost. A fingerprinting technique introduced at ICLR '21, $\textit{Dataset Inference}$ (DI), has been shown to offer better robustness and efficiency than prior methods.
The authors of DI provided a correctness proof for linear (suspect) models. However, in the same setting, we prove that DI suffers from high false positives (FPs) -- it can incorrectly identify an independent model trained with non-overlapping data from the same distribution as stolen. We further prove that DI also triggers FPs in realistic, non-linear suspect models. We then confirm empirically that DI leads to FPs, with high confidence.
Second, we show that DI also suffers from false negatives (FNs) -- an adversary can fool DI by regularising a stolen model's decision boundaries using adversarial training, thereby leading to an FN. To this end, we demonstrate that DI fails to identify a model adversarially trained from a stolen dataset -- the setting where DI is the hardest to evade.
Finally, we discuss the implications of our findings, the viability of fingerprinting-based ownership verification in general, and suggest directions for future work.

Under review as a conference paper at ICLR 2023
ON THE ROBUSTNESS OF DATASET INFERENCE
Anonymous authors
Paper under double-blind review
ABSTRACT
Machine learning (ML) models are costly to train as they can require a significant amount of data, computational resources and technical expertise. Thus, they con- stitute valuable intellectual property that needs protection from adversaries want- ing steal them. Ownership verification techniques allow the victims of model stealing attacks to demonstrate that suspect model was in fact stolen from theirs,
A Although number of ownership verification techniques based watermarking or fingerprinting have been proposed, most of them fall short either in terms of se- curity guarantees (well-equipped adversaries can evade verification) or computa- tional cost. A fingerprinting technique introduced at ICLR '21, Dataset Inference (DI), has been shown to offer better robustness and efficiency than prior methods.
The authors of DI provided correctness proof for linear (suspect) models, How- ever, in subspace of the same setting, we prove that DI suffers from high false positives (FPs) it can incorrectly identify an independent model trained with non-overlapping data from the same distribution as stolen. We further prove that DI also triggers FPs in realistic, non-linear suspect models. We then confirm em- pirically that DI the black-box setting leads to FPs, with high confidence.
Second, we show that DI also suffers from false negatives (FNs) an adversary can fool DI by regularising a stolen model's decision boundaries using adversarial training. thereby leading to an FN. To this end, we demonstrate that blxk-box DI fails to identify a model adversarially trained from a stolen dataset the setting where DI is the hardest to evade.
Finally, we discuss the implications of our findings, the viability of fingerprinting- based ownership verification in general, and suggest directions for future work.
INTRODUCTION
Machine learning (ML) models are being developed and deployed at an increasingly faster rate and in several application domains. For many companies, they are not just part of the technological stack that offers an edge over the competitors but a core business offering. Hence, ML models constitute valuable intellectual property that needs to be protected.
Model stealing is considered one of the most serious attack vectors against ML models (Kumar et al., 2019). The goal of model stealing attack is to obtain a functionally equivalent copy of victim model that can be used, for example, to offer a competing service, or avoid having to pay for the use of the model.
In the white-box attack, the adversary obtains the exact copy of the victim model, for example by reverse engineering an application containing an embedded model (Deng et al., 2022). In contrast, in black-box attacks (known as model extruction attacks) (Papernot et al. 2017 Orekondy et al.) Tramèr et al. 2016) the adversary gleans information about victim model via its predictive interface. Two possible approaches to defend against model extraction are 1) detection (Juuti al.) 2019 Atli et al., 2020; Zheng et al. 2022) and 2) prevention (Orekondy et al.) 20200] Mazeika al. 2022, Dziedzic et 2022). However, powerful, yet realistic attacker can circumvent these defenses (Atli et al. 2020).
2019
An alternative defense applicable to both white-box and black-box model theft is based on deter- rence. It concedes that the model will eventually get stolen. Therefore, an ownership verification technique that can identify and demonstrate suspect model as having been stolen can serve

---

Under review as a conference paper at ICLR 2023
deterrent against model theft. Early research in this field focused watermarking based embed-
ding triggers or backdoors (Zhang et al., 2018)) [Chida et al. 2017 Adi et al. 2018) into the weights of the model. Unfortunately, all watermarking schemes were shown to be brittle (Lukas et al.) 2022) in that an attacker can successfully remove the watermark from a protected stolen model without incurring substantial loss in model utility.
An alternative approach to ownership verification is fingerprinting. Instead of embedding a trigger or backdoor in the model, one can extract a fingerprint that matches only the victim model, and models derived from it. Fingerprinting works both against white-box and black-box attacks, and does not affect the performance of the model. Although several fingerprinting schemes have been proposed, some are not rigorously tested against model extraction (Cao et al. [202 Pan et al. 2022)
and others can be computationally expensive to derive (Lukas et al. 2021
In this backdrop, Dataset Inference (DI), which appeared in ICLR 2021 (Maini et al. 2021) promises to be an effective fingerprinting mechanism. Intuitively, it leverages the fact that if model owners trained their models on private data, knowledge about that data can be used to identify all stolen models. DI was shown to be effective against white-box and black-box attacks and is efficient to compute (Maini et al., 202T). It was also shown not to conflict with any other defenses (Szyller & Asokan 2022). Given its promise, the guarantees provided by DI merits closer examination.
In this work, we first show that DI suffers from false positives (FPs) it can incorrectly identify an independent model trained with non-overlapping data from the same distribution as stolen. The authors of DI provided a correctness proof for linear model. However, DI in fact suffers from high FPs, unless two assumptions hold: (1) a large noise dimension, as explained in the original paper and (2) large proportion of the victim's training data used during ownership verification, as we prove in this paper. Both of these assumptions are unrealistic in a subspace of the linear case used by DI: (i) we prove that large noise dimension can lead low accuracy in the resulting model, and (ii) revealing too much of the victim's (private) training data is detrimental to privacy. Furthermore, we prove that DI also triggers FPs in realistic, non-linear models. We then confirm empirically that DI leads to FPs, with high confidence in the black-box verification setting, "black-box DI", where the DI verifier has access only to the inference interface of a suspect model, but not its internals.
We also show that block-box DI suffers from false negatives (FNs): an adversary who has in fact stolen a victim model can avoid detection by regularising their model with adversarial training. We provide empirical evidence that an adversary who steals the victim's 's dataset itself and adversarially trains a model can evade detection by DI.
We claim the following contributions:
Following the same simplified theoretical analysis used by the original paper (Maini et al., 202T). in subspace of the linear case used by DI, we show that for linear suspect model, a) high-dimensional noise (as required in (Maini et al. 2021 leads to low model accuracy (Lemma [ Section 3.1). and 2) DI suffers from FPs unless large proportion of private data is revealed during ownership verification (Theorem J
Section 3.1
Extending the analysis to non-linear suspect models, using a PAC-Bayesian frame- work (Neyshabur et al. 2018), we show that DI suffers from FPs in non-linear models regardless of how much private data is revealed (Theorem 2 Section Se.tion3...III:
We empirically demonstrate the existence of FPs in realistic black-box DI setting (Sec- tion 3.2.2):
We show empirically that black-box DI also suffers from FNs: using adversarial training to regularise the decision boundaries of a stolen model can successfully evade detection by
DI while incurring only a modest loss in accuracy (+= 6pp) (Section
DATASET INFERENCE PRELIMINARIES
Dataset Inference (DI) aims to determine whether a suspect model JsP was obtained by an adver- sary A who has stolen a model (f.)) derived from a victim V's private data Sy, or belongs to an independent party I (fz). DI relies on the intuition that if a model is derived from Sy, this infor- mation can be identified from all models. DI measures the prediction margins of a suspect model around private and public samples: distance from the samples to the model's decision boundaries. If
2

---

Under review as a conference paper at ICLR 2023
Table 1: Summary of the notation used throughout this work.
V
the victim
model
I
an independent party
fv
a model trained on Sy
adversary
Jo
model trained on So
S
dataset
fi
model trained on S
Sy
V's private dataset
fA
A's model
So
public dataset
JsP
suspect model
S₁
I's data
model weights
D
distribution that all datasets follow sample from
gv regression model
(x, v)
noise dimension
JSP has distinguishable decision boundaries for private and public samples DI deems it to be stolen; otherwise the model is deemed independent.
In the rest of this section, we explain the theoretical framework that DI uses consisting a linear suspect model the embedding generation necessary for using DI with realistic non-linear suspect models, and the verification procedure, A summary of the notation used throughout this work appears in Table [
2.1 THEORETICAL FRAMEWORK
The original DI paper (Maini et al. 2021 used a linear suspect model theoretically prove the guarantees provided by DI. We first explain how DI works in this setting.
Setup. Consider data distribution D, such that any input-label pair (xx, y) can be described as:
MRRRR RK 22 ~ ~N(0, ER.
where (æ1, x2) E R K+D and RKK is fixed vector. The last D dimensions of represent Gaussian noise (with variance d²).
Structure of the linear model. Assuming linear model f. with weights w (W1, W2). such that f(x) = w1 æ1 + U2 22, then the final classification decision is sgn(f(x)). With the weights initialized zero, learns the weights using gradient descent with learning rate until (f(x) is maximized. Given private training dataset Sy D {xx(1) ( 1,... m), and public dataset S. ~ (both of size m), then wi mu and W2 = Dm= (x regardless the batch size.
In DI, the prediction margin p(-) is used to imply the confidence of f in its prediction. It is defined the margin (distance) of data point from the decision boundary.
p(x)
A f(x).
The authors (Maini et al. 2021) show that the difference of expected prediction margins of two datasets Sv and So is = The threshold can be set À E (0, Do²), and by estimating the difference of the prediction margins on S, and Sy on fsp, DI is able to distinguish whether that model is stolen.
Note that DI uses approximations of the prediction margins based on embeddings. The theoretical framework assumes that the approximations are accurate, and we can use them directly for the theoretical analysis (Equation[]) Tp. For the linear model, the margins can be computed analyticalpy however, in Section 2.2 we explain how the approximations of the margins are obtained.
2.2 EMBEDDING GENERATION
In order to use DI one needs to generate embeddings of the samples. V queries their model fy with samples in their private dataset Sy and public dataset Sp, and assigns the labels b 1 and b respectively. The authors propose two methods generating the embeddings: white-box approach (MinGD) and black-box one (Blind Walk). In this work, use only Blind Walk as it outperforms
3

---

Under review as a conference paper at ICLR 2023
MinGD in most experimental setups in the original work, and is more realistic, it only requires access to the API of the suspect model.
Blind Walk estimates the prediction margin of sample by measuring its robustness random noise. For a sample (x, y), to compute the margin, first choose a random direction and take & E N steps in the same direction until the misclassification f( + kō) + y. This is repeated multiple times to increase the size of the embedding. As reported (Maini et al. 2021). obtaining embeddings for 100 samples can take up to 30, 000 queries.
Having obtained the embeddings, V trains a regression model gv that predicts the confidence that a sample contains private information from Sy.
2.3 OWNERSHIP VERIFICATION
Using the scores from gV and the membership labels, V creates vectors and cv of equal size from Sy and So, respectively. Then for a null hypothesis Hg μ jV where f and j ov are mean confidence scores. The test rejects Ho and rules that the suspect model is 'stolen', or gives an inconclusive result.
To verify whether fsr is stolen or independent, V obtains the embeddings by querying the model (using Blind Walk) using samples from Sy and So- Then they use the embeddings to obtain the confidence scores from the gv, and performs hypothesis test the two distributions of scores.
3 FALSE POSITIVES IN DATASET INFERENCE
To generate the embeddings for a specific sample in the private dataset Sv, DI requires query- ing the suspect model SSP hundreds of times. To reduce the total number of queries, DI was shown to be effective with only 10 private samples with at least 95% confidence. Ad- ditionally, DI requires a large random noise dimension D such that probability of success in- creases to 1 as D + co. In this section, we prove that these two assumptions are not real- istic in the case of linear model: 1) DI susceptible to false positives (FPs) unless V re- veals a large number of samples; 2) large D will harm the utility of the model (Section 3.1 Furthermore, we find that the theoretical re- sults on linear suspect models which say that
the margins on different models are distinguish- able with some strict conditions do not hold for
0.486
k=50
a 0.437
k=1000
on
0.400
more realistic non-linear suspect models. Us-
Probability of
ing PAC- -aC-Bay margin based generaliza-
0.309
k=10000
tion bound (Neyshabur et al. 2018) we prove
that models trained the same distribution are indistinguishable, and will trigger FPs (Sec- tion 3.2.I Next, we provide empirical evidence for the existence of FPs (Section 3.2.2
0.200
k 50000
0.132
0.100
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of tested samples
3.1 LINEAR SUSPECT MODELS
Probability of an FP as the fraction of revealed private samples for ear suspect model (Equation 6 V needs to use many private samples to guarantee low false posi- tive rate.
Figure 1:
In section 2. we have a distribution ID set up for linear models. The linear model f should cor- rectly classify most of the randomly picked data from this distribution. However, in subspace of the linear case used by DI, we find that the dimension of the noise part of needs to be small, otherwise it will harm the utility of the model.
10 for a lin-
Lemma 1 (Need for Bounding Noise Dimension). Let f be a linear model trained on S ~ D. For sample (z, y) sampled from D which is independent of S, assuming that ||u S ym and o² > / then, the linear model f correctly classifies (x, y) with probability larger than 0.9 only if 10.
The details of the proof are in the Appendix A Lemma shows that if the dimension of x2, which follows a2), is large, then the noise will dominate and mislead it into making incorrect
4

---

Under review as a conference paper at ICLR 2023
predictions. For example, set D 1000 and assume that the variance of 22 is 0.25 (close to the CIFARIO dataset). Then. f can correctly classify a sample that different from f's training set with probability up to 0.69.
Theorem (Existence of False Positives with Linear Suspect Models), Let fr be linear classifier trained on the independent dataset St DD with accuracy more than 0.9. Assume that |St] m, |all2 S 1 and 0² > 1_ Let k be the number of samples estimated required for the verification. 7mm
7m Then, the probability that V mistakenly decides that Jr. is a stolen model PIV Sv; D) = 1] >
Sk) 1 1 mm)
Where V is Vss decision function (Maini et 2021
[1. if fsr ~ fA. W(ss, S;D)
10 if fsp fz,
Proof. Recall that V tries reveal only few samples during the verification. For distribution D where ||س|||
I vm and x²
mm
Following the intuition from DI Yeom et al. 2018), for satisfactory performance, DI must minimise both false positives and false negatives. Hence, the objective function defined as: 0l
minx ([v D) = P(V
where the margin of D is estimated using Sy and So. Note that are only interested in the false positives PP (fz, 11, let S = {(x(11) (() = 1,..., m), S be subset of S, consisting of l: samples.
P[v( D); F le. sest [u/I(x]]]] [[([a))]][[aaaaa Es [M/x(x))] 2 A]
P (m.) Σ
y 22 x2
E Elwyesst
Σ ( (1)
(2)
x2
2
Al
(4)
Fl
(0) ( x2
2
(
(p) 22
A].
Recall that 222 x.(0) and x2 are D-dimensional vectors sampled independently from N o²). Using central limit theorem
approximate the terms. We have Σ y( (O, mo²). Then, we can approximate T Σ E. (( a2 mn())) (f) by 4 NO, mD
k 0²) and approximate
Σ En y (0) (6) x² by 12 ~ NO, mD 04) (Maini et al. 2021 A
Thus, we get t ~ N(O, 2mD o4),
and
2mD Ply
Vkd
(T; 1| = P X|
0²
11 = PIZ 2
1- 4
k
v 2
V 2m Do
(5)
where 7 ~ NN0 1). The optimal threshold is given as x =
2
2
VID
PV:(f2,Sv:D) 1] =1 Φ
(6)
2 2m
From Equation 6 we see that the probability of false positives relies on the number of points used for the verification m and the size of D. Combining with Lemma the proof complete.
In other words, the success of DI is directly related to the number of samples used for the verification. This is similar to the analysis of failure of membership inference in the original paper when the is extremely low, e.g. only 10 samples. In the DI paper, it explained that DI succeeds because it
5

---

Under review as a conference paper at ICLR 2023
calculates the average margin for multiple verification samples; whereas membership inference fails as it relies on per-sample decision. So when the number of tested samples is smaller, the success rate of DI will be close to 0.5, just like for membership inference. In Figure I we show the probability of an FP (Fquation 66 for different values of k; even for = 10000 the probability is 0.309.
Hence, even the simple linear setup, ( has false positives with high probability; in partic- ular, when the fraction of tested samples is small.
3.2 NON-LINEAR SUSPECT MODELS
Having demonstrated the limitations of the linear model, we now focus non-linear suspect mod- els. The intuition is based on the margin-based generalization bounds. Note that the generalization bounds states that the expected error of the margin based loss function is bounded, and the bound is mostly related to the distribution (Neyshabur et al. 2018). Since DI assumes all the datasets fol- low the distribution D, our intuition to directly the generalization bounds and the triangle inequality to prove the similarity of the models trained on the same distribution.
3.2.1 THEORETICAL MOTIVATION
Let fw be real-valued classifier fw RR. ||ж|| s B with parameters (W,) For any distribution D and margin p(f, x)) = f(x) [yl - max f S % where > 0. The margin is same for the linear model with labels as:
+1, Then, we define the margin loss function
C,(F.) P(a.)w [x))[ max marr <]
Note that the PAC-Bayes framework Neyshabur et al. 2018) provides guarantees for any clas- sifier ʃ trained on data from a given distribution. We define the expected loss of a classifier on distribution 'D LD ((,,))) (=)(()) ( and the empirical loss a dataset S as Then, for a d layer feed-forward network f with parameters
is = 1. ( (m)))ss (((()s)))())))))) w ( and ReLU activation (Neyshabur et al. 2018). The empirical loss very close the expected loss. For any 0,7 > O, with probability 1 over the training set, we have:
(oo(fs) - {
(8)
where (dhhhi(dh IT [W
Wall EL 11W dm
mm [W
and h is the upper bound dimension for
This PAC-Bayes based generalization guarantee states that for model f. the distance between the empirical loss and the expected loss bounded, and the bound can be very small when the model's margin is large. Thus, distribution to be similar. This contradicts the intuition of DI.
expect that the margins of
any dataset that follows given
Moreover, since DI assumes that Sy and S, follow the same distribution D, can show that the margins for fv and fi are similar to each other.
Theorem (k-independent False Positives with Non-linear Suspect Models). For the victim private dataset Sy D and an independent dataset S₁ D. let fao be a d-layer feed-forward network with ReLU activations and parameters w = [W.]d= i=1- Assume that fy is trained on Sy and Jr. is trained on S1, fv and fi have the same structure. Then, for any B, d, h and any æ X, there exist prior P on w. s.t. with probability at least
E(p(fv. =() Se.
The details of the proof are in the Appendix A Hence, for any two models trained on the same distri- bution, the expectation of margins for any sample similar. Given that DI works by distinguishing the difference margins for two models, it will result in false positives with probability at least (Theorem

---

Under review as a conference paper at ICLR 2023
3.2.2 EMPIRICAL EVIDENCE
Having proved the existence of FPs for non-linear models, we now focus on empirically confirming it.
First, recall the original experiment setup (Maini et al. 2021): ; let us consider the following two models: 1) fy trained using Sy. and 2) fo trained using So. In the original formulation, c.g. for CIFARIO, CIFAR 10-train (50, 000 samples) is used as Sv, and CIFARIO-tes 10-test is used as So (10, 000 samples). Recall that V uses their Sy and S obtain the embeddings that are then used train the regression model gv.
DI was shown be effective against several post- proccesonn used to obtain dependent models which are expected to be flagged as stolen true positives However, the independent model fo is trained on Sq the same data that is used to train gv. This means that the same dataset So used both to train gv and subsequently, to evaluate it. This is likely to introduce bias that overestimates the efficacy of gy and DI as a whole.
To address this, and test whether DI works for a more reasonable data split, we use the following setup:
1) randomly split CIFARIO-train into two subsets (Atroin and Berain) of 25, 000 samples each;
2) assign Sy = Atrain, and train fy using it;
3) continue using CIFARIO 10-test as So (nothing changes), and train fo using it;
4) gv is trained using the embedding for So and the new Sy, obtained from the new Jv;
5) assign S₁ Btrain, independent data of a third-party I. who trains their model fz.
This way, we have an independent model fr that was trained on data from the same distribution D as Sy but data that was not seen by gv_
Recall that to determine whether the model is stolen, DI obtains the embeddings for private (Sv) and public (Sy) samples. Then it measures the confidence for each of the embeddings using the regressor gy. For model derived from V's Sy, the mean difference (Ap) between the confidence assigned to Sy and Sn should be large. If the model not derived from Sv, the difference should be small. The decision is made using the hypothesis test that compares the distributions of measures from gv.
In Figure 2 we visualise the difference in the distributions for three models. For fv we observe two separable distributions with a large (Ap), while for fo the difference small - is working as intended. However, for fz, even though Aμ is smaller than for fy it is sufficiently large reject Ho with high confidence. Therefore, fz is marked as stolen, a false positive, In Table 2 we provide A/r and the associated p-values for multiple random splits.
Table 2: Verification of an independent model trained on the same data distribution triggers FP. Also, we report the accuracy of the models on the test set. We provide the mean and standard deviation computed across five runs. Verification done using k = 10 private samples. FPs become more significant as A increases (see Appendix B
Model
Accuracy
Ay 1.62 + 0.08
p-value
fv
0.87 + 0.03
10-18 + 10-18
fz
0.87 + 0.03
1.14 4: 0.12
10-8 + 10-8
fo
0.64 + 0.02
-0.29 + 0.12
0.46 0.04
We discuss the implications of our findings in Section 5
We use the official implementation of DI, together with the architectures and training loops. Our changes limited the data splits only.
7

---

Under review as a conference paper at ICLR 2023
250
6.72
200
Sy 5-m33/00.15
200
150
150-
100
50
50
-1.5
-2.5
-1.4
-0.5
0.5
Confidence score
Confidence score
-100
-0.5
0.0
1.0
Figure 2: Left to right: Jv. JI. Jo- Comparison of distributions of the confidence scores assigned to the embeddings by gv- Au smaller for fz than for fy but large enough to trigger an FP.
Table 3: LA adversarially trained Sy results in false negative, Also, report the accuracy of the models on the test set. We provide the mean and standard deviation computed across five runs. Verification done using k: = 10 private samples.
Model
Accuracy
A/
p-value
fv
0.92 + 0.01
1.59 + 0.04
10-21 + 10
fa
0.86 0.01
0.12 1 0.06
0.15 0.07
fo 0.64 + 0.02
-0.29 + 0.12
0.46 0.04
FALSE NEGATIVES IN DATASET INFERENCE
Having demonstrated the existence of false positives, we now show that DI can suffer from false neg- atives (FNs). A can avoid detection by regularising f.. and thus changing the prediction margins. This in turn, will mislead DI into flagging fA as independent.
Recall that Blind Walk relies on finding the prediction margin by querying perturbed samples de- signed to cause a misclassification. In order to avoid detection, A needs to make the prediction margin robust such perturbations. We do using adversarial training: popular regularisation method used to provide robustness against adversarial examples. A who launches model extraction attack against fv. steals V Sv can adversarially train fAA
During adversarial training, each training sample (ær, y) is replaced with an adversarial example that misclassified [a +7) # y. There exist many techniques for crafting adversarial examples. We use projected gradient descent (Madry et al. 2018) (PGD), and we set = 10/255 (under loo).
We evaluate adversarial training as a way to avoid detec- tion in a setting where A steals V Sy and trains their own model fA- fA has the same architecture and hyper- parameters as fv. but is adversarially trained. Hence, the experiment is biased in favour
S
400
Se-mean=-0070 Sy-mean=-0.8
300
Count
In Figure 3 we visualise the difference in the distributions of scores assigned by Jy to fA embeddings derived for Sy and So. We observe that the distributions are not clearly separable and result low Ap, and hence Ho in-
cannot be rejected. Therefore, fA is marked dependent model, a false negative. In Table we provide Ap and the associated p-vuuues for multiple runs.
0.5
1.0
Confidence
Figure Confidence scores assigned to embeddings by obtained from fA- Ap is small enough to trigger FNs. of noise affects the verification in Appendix D Also, we discuss the resulting implications in Sec- tion 5
Note that adversarial training comes with an accuracy trade-off. In experiments, the accuracy of f. goes from 0.92 to 0.001. We study how the amount
We use the official implementation of DI, together with the architectures and training loops. Our changes limited adding adversarial training.
8

---

Under review as a conference paper at ICLR 2023
5 DISCUSSION
Revealing private data. We have shown that DI requires revealing significantly more than 50 samples to avoid false positives the case of linear models (Figure []] Since the core assumption of DI is that Sv is private, revealing too much of Sy during the ownership verification constitutes a privacy threat. cannot query the model sufficiently without leaking the query data to the adversary. Additionally, it was shown that using more samples gives V more information about the prediction margin than using stronger embedding methods (Maini et al., (021). Model owners that operate in sensitive domains such as healthcare or insurance industry need to comply with strict data protection laws,
neither of the settings described in Section of the original DI paper the victim
and hence need to minimise the disclosure.
One potential way to protect the privacy of the private samples used for DI ownership verification is to use oblivious inference (Liu et al. 2017. Juvekar et al.) 2018). This way V could query fA without revealing Sy. Despite recent advances in efficient oblivious inference (Samragh et al, 2021 Watson et al.] 2022] Samardzie et al. 2021 2022). it requires all parties (including A!) to update their software stacks which may not always be realistic.
Viability of ownership verification using training data. We have demonstrated that DI suffers from FPs when faced with an independent model trained on the same distribution. While is rea- sonable assume that V"s data is private, the uniqueness of the distribution is difficult to guarantee in practice. For example, two model builders may have data from the same distribution because they purchased their training data from a vendor that generates per-client synthetic data from the same distribution (e.g., regional financial data). In fact, two model builders working on the same narrow domain and independently building models that are intended to represent the same phenomenon, may very well end up using data from the same distribution.
There are other methods that attempt to detect stolen models based on the dataset used to train them (Sablayrolles et al., 22200] Pan et (022). However, they rely on flaws in the model to estab- lish the ownership (susceptibility to adversarial examples (Sablayrolles et al. (2220) or membership inference attacks (Pan et al. 2.22)) Intuitively, given a perfect membership inference attack, fingerprinting scheme should be possible. However, recent work shows that for a balanced dataset, only a fraction of records is vulnerable to confident membership inference attack (Carlini et al. 2022 Duddu et al. 202T) which in turn reduces the capabilities of membership inference-basee fingerprinting scheme. Therefore, any improvements generalisation robustness (such as ad- versarial training or purification (Nie et al. 2022)) of ML models reduce the surface for ownership verification schemes.
White-box theft Our experiments in Section 4 Jrre limited to A that trains their own model - they either steal the data or conduct a model extraction attack. If A obtains an exact copy of the model, they might lack the data to fine-tune it with adversarial training. Hence, our findings do not apply to the white-box setting. We leave the examination of other threat models out as future work.
Black-box vs. white-box verification setting. Our evaluation is focused on the black-box DI set- ting. We do not consider the white-box DI setting which uses MinGD. While white-box Ds feasible in scenario where V/ takes A (the holder of a suspect model) to court, requiring A to provide white- box access the suspect model, prosecution is an expensive undertaking. Realistically V is likely to first conduct black-box DI to decide whether the expense of prosecution is justified. Therefore, FPs in the black-box DI setting can cause substantial monetary loss to V.
CONCLUSION
We analyzed Dataset Inference (DI) (Maini et al. 2021) ) a promising fingerprinting scheme, to show theoretically and empirically that DI is prone to false positives in the case of independent models trained from distinct datasets drawn from the same distribution. This limits the applicability of DI only to settings where model builder uses dataset with definitively unique distribution. We also showed that an attacker can use adversarial training to regularise the decision boundaries of a stolen model to evade detection by DI at the cost of modest (6pp) drop in accuracy.
Nevertheless, DI is a promising ML fingerprinting scheme. Model owners can use our results to make informed decisions as whether DI is appropriate for their particular settings.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weak- ness into strength: Watermarking deep neural networks by backdooring. In 27th USENIX Secu- rity Symposium, pp. 1615-1631, 2018.
Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, and N. Asokan. Extraction of com- plex DNN models: Real threat or boogeyman?". In "Engineering Dependable and Secure Ma- chine Learning Systems", " pp. 42-57, 2020.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. IPGuard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, ASIA CCS '21, pp. 14-25, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 1781388888878 doi: 10. 10.1145/3433210.4475522
URL
00000 3437522
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Mem- bership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914, 14, 2022. doi: 10. 10.1//9// 14. 2022.999334...
Zizhuang Deng, Kai Chen, Guozhu Meng, Xiaodong Zhang. Ke Xu, and Yao Cheng. Understanding real-world threats to deep learning models in android apps. 2022. doi: 10.48550/anX1..2220009 577. URL httpttsss///xiii org/ /ttp:s////arrlooorrab//////// 0b552229.00.77
Vasisht Duddu, Sebastian Szyller, and N. Asokan. SHAPr: An efficient and versatile membership privacy risk metric for machine learning. CoRR, abs/21 120.0220,,, 2021. URL https://arxi
v. w.org/abs/211 2ttp:/2211200000
Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, and Nicolas Papernot. Increasing the cost of model extraction with calibrated proof of work. In International Conference on Learning Representations, 2022. URL httph:ttt///peeeee
ew fert/fforum? A1a
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N. Asokan. PRADA:
protecting against DNN model stealing attacks. In IEEE European Symposium Security & Privacy, pp. 1-16. IEEE, 2019.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. GAZELLE: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pp. 1651-1669, Baltimore, MD, August 2018. USENIX Association. ISBN 978-1- 939133-04-5. URL https://p//.....
www. www.uusnnnn....
/presentation jjuvkar
Ram Shankar Siva Kumar, Jeffrey Snover, David O'Brien, Kendra Albert, and Salome Viljoen. Failure modes in machine learning. hnw.ueeeeoooe---------ee-----eee
2019. Online; accessed
20 September 2022.
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against neural network model stealing attacks using deceptive perturbations. In 2019 JEEE Security and Privacy Work- shops (SPW). pp. 43 49, 2019. doi: 10.1 109/SPW.2019.00020
Jian Liu, Mika Juuti, Yao Lu, and N. Asokan. Oblivious neural network predictions via min- ionn transformations. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 17, pp. 619-631, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450349468. doi: 1,.1145/1133556.11444
URL
wtt.:////..11111111111111 1ee.11113311533 3134056
Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep neural network fingerprinting by con- ferrable adversarial examples. In International Conference on Learning Representations, 2021.
httptttss:/// httphh/ppps///ppeeeeiiieee
n forum? wwt.:/www..eeeooteddddeeem ddarruunidaVV HH1
Nils Lukas, Edward Jiang, Xinda Li. and Florian Kerschbaum. Sok:
How robust is image classifi- cation deep neural network watermarking? In 2022 IEEE Symposium on Security and Privacy
(SP), pp. 804, 2022. doi:
10

---

Under review as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
net metteoorrmidddra.......
7Ab
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolu- tion in machine learning. In International Conference on Learning Representations, 2021. URL
https://openreviee net/ Tire/forose hi
Mantas Mazeika, Bo Li, and David Forsyth. How to steer your adversary: Targeted and efficient model stealing defenses with gradient redirection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15241-15254. PMLR, 17-23 Jul 2022. URL hnttps://proceett press/v162/nen http:////.maaaeeaaaa22 html
mml..
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normmaizeed margin bounds for neural networks. In International Conference on Lear- ing Representations, 2018. / For. ott.:ttwee...eaaeeeeeeeeeeee/ere
_WbCC
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML). 2022.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In CVPR, pp. 4963, 2019.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses
against dun model stealing attacks. In International Conference Learning Representations, 2020. URL / httttp:://openneiie.... net/ htte///orruuddoooeeeeooeaa
SyevYxHtDD Syw.
Xudong Pan, Yifan Yan, Mi Zhang, and Min Yang. Metav: A meta-verifier approach task- agnostic model fingerprinting. 2022. doi: 1.485500ARXXV.2201.00
URL https:
07391
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In ACM Symposium on Information, Computer and Communications Security, pp. 506-519. ACM, 2017.
E. Quiring, D. Arp, and Rieck. Forgotten siblings Unifying attacks on machine learning and digital watermarking. In IEEE European Symposium on Security & Privacy, pp. 488-502, 201 8.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Herve Jegou. Radioactive data: tracing through training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th In- ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8326 8335. PMLR, 13-18 18 Jul 2020. URL https: sttps://proceediing .mlr....r...... malr.p. pr eess/v119/ htss//119/sabllste awwobllyyolleeeeeee html
Nikola Samardzie, Axel Feldmann, Aleksandar Krastev, Srinivas Devadas, Ronald Dreslinski, Christopher Peikert, and Daniel Sanchez. Fl: A fast and programmable accelcrator for fully homomorphic encryption. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO '21, pp. 238-252, New York, NY. USA, 2021. Association for doi: 10.1145/3466752.4488 3480077
Computing Machinery. ISBN 978 9781450385572.
URL
hhttps://oo. org. 10.
Nikola Samardzic, Axel Feldmann, Aleksandar Krastev, Nathan Manohar, Nicholas Genise, Srini- vas Devadas, Karim Eldefrawy, Chris Peikert, and Daniel Sanchez. Craterlake: A hardware accelerator for efficient unbounded computation on encrypted data. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA '22, pp. 173-187, New York, NY, USA, 2022, Association for Computing Machinery. ISBN 9781450386104, doi: 10. 011145/4477496355779999 URL https: //doi.//oiii. http:///ooooo//0..1011// wtt.://.....1/340099808.8000 3527333
11

---

Under review as a conference paper at ICLR 2023
Mohammad Samragh, Siam Hussain, Xinqiao Zhang, Ke Huang, and Farinaz Koushanfar. On the application of binary neural networks in oblivious inference. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4625-4634, 2021. doi:
N0.110//CVRR233388.222
Sebastian Szyller and N. Asokan. Conflicting interactions among protection mechanisms for machine learning models. 2022. doi: 10.48550/ARXIV.20000 220.XXXX...2...99... URL https:
// 7/arxiv.co wrw./org/ooo abs 2207 01991
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th USENIX Security Symposium, pp. 601-618, 2016.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12(4):389 434, 2012.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin' ichi Satoh. Embedding watermarks into deep neural networks. In ACM International Conference Multimedia Retrieval, pp. 269-277. ACM, 2017.
Jean-Luc Watson, Sameer Wagh, and Raluca Ada Popa. Piranha: A GPU platform for se- cure computation. In 31st USENIX Security Symposium (USENIX Security 22), pp. 827-844, Boston, MA, August 2022. USENIX Association. ISBN 978-1-939133-31-1. URL https:
www.ix.enii.......... 7/ w/wwixusenix.. http:////...aaaaaeeeeeaa/aee///a tottiiitoowonnsnoooooooo
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 IEEE 31 Computer Security Foundations Symposium (CSF), pp. 268-282, 2018. doi: 10. 10.11099FFF2218.000221
Jialong Zhang, Zhongshu Gu. Jiyong Jang, Hui Wu, Marc Ph Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In ACM Symposium on Information, Computer and Communications Security, pp. 159-172, 2018.
Huadi Zheng, Qingqing Ye, Haibo Hu, Chengfang Fang, and Jie Shi. Protecting decision bound- ary of machine learning model with differentially private perturbation. IEEE Transactions on Dependable and Secure Computing, 19(3):2007-2022, 2022. doi: 10. 10.1109/TDSC..222033333333
A EXISTENCE OF FALSE POSITIVES IN DATASET INFERENCE
Calculating the prediction margin. We assume that the model weights are initialized to zero. For each sample I in a dataset S ~ D = (+)(1) ( = 1,... m). y ~ {-1, -1]. The learning
algorithm observes all samples in S once and maximize the loss function L(x, y) y f(x). For the learning rate a = 1, the weights are updates as:
w + any ( (t)
(10)
Recall that x = (æ1, E KRD the weights of the linear model are wi mu and W2 Em 20
(6) when the training is completed.
When writing out the linear classifier explicitly, we can easily calculate the prediction margin of each sample (I, y) in S.
y. f(x) y- (wiwr 21 + w222) y. (mu yuu 00 x2) c
(0
(6) x2 22 (11)
i=1
The expectations of the prediction margin for the points in training set S =(xx (z, 1) E S) is.
Est [s+(((())= yo Es+
x2
x2i = yc +
Es+1
Σ
y 10 (1)
x2 + yz
i=1
2/ (8)
(12)
= =c 0 + 2o?.
12

---

VD ) Under review as a conference paper at ICLR 2023
Note that in Equation 12 since æ2 [(, Do then 2²
x². Ela22 (8)
Do
Consider new dataset Sp D, the expectations of the prediction margin for the points in Sg are,
s (u](()))) = yc + Fss
32 =c.
(13)
Finally, we see that the difference of prediction margin of training set and test set So is Est [v]())) Do
Es+ [v](x))
(14)
DI's decision function. From the above analysis, we know that the statistical difference between the distribution of training and test data is Do which is usually larger than 1 in numerical. DI utilizes this difference to predict if a potential adversary's model stole their knowledge.
Since we know that [s, dataset inference victim's decision function. is defined as,
and Es [s(y[(()] c Do Let ()) represent the
(f [1, if E(r.))esly - f(x) Eply f(x)) 2 X,
(15)
(, otherwise,
where À E [0, Do is some threshold that the decision function uses to maximise true positives and minimise false positives.
Proof for Lemma I For a linear model f trained on distribution D where æ = (2), I1 yu,zz ~ (0, o²) and ||u||2 S / f is expected achieve high accuracy on any sample (æ, y) sampled randomly from which is independent of the training data set of f.
Proof. Given linear model f trained on dataset ~ (x=(+) y ) 1...., m), and test sample (xe, y) sampled randomly from D which is independent of S, the probability that (æ, v) is correctly classified by can be represented as:
Plyf(x) 2 0] = PPmma 2 + > (1))=22 (8) 0)
-ly y(i) x2 2 -mu²
(16)
m
Fl y vi)a
x2 N -1]
Since I2 ~ (N000 are D-dimensional vectors, we can use central limit theorem to approximate the term. Thus, the internal term can be approximated by variable ~ (N0 mDo4). Let Z (O,0.)). 1).
(()f(()) 2 & P(VmDo22 -1]=1 D(
(17)
where & is the normal CDR For distribution where the randomness 2 mm
1ym
4
Plyf(x) 0] < 1 Φ(
(18)
4. ~ 0.10. The linear model f can correctly classify a sample with probability more where Φ(- (T than 0.9 only if D < 10.
13

---

Under review as a conference paper at ICLR 2023
We can also calculate the accuracy of the training set S similarly. For training sample (æ. y) sampled randomly from S,
m
(lyf(x)= 2 0] = P|mu 2 + Σ
y())zz
10)
22 2 20]
mn
Plmm² + 2 + Σ
y(1) 10) 22 2 0]
(19)
m
= + Σ
(i)) x22 2 -mu²
2ly2 _ y1))a x2 2 -1].
Since y²0 2 0 for any sample in S, we have
y 22 + _ 3(1)=2 2
Σ 3(1)
(20)
Then, P(ayy)s 2 P(a.))ED/S. This completes the proof.
Proof for Theorem 2 Let fw be a d-layer feed-forward model trained on distribution D with pa- rameters w (W.)--1 i and the ReLU activation function. Assuming training dataset S D, the model is given as fs = fwtus. where ug is random variable whose distribution may also depend on S.
Since the key to analyze the margin is the output of the model, we first introduce Lemma that analyzes the perturbation bound of the model trained on and D.
let Lemma 2 (Perturbation Bound (Lemma 2) in (Neyshabur et al., 2018)) For any B,d fw X - R2t be d layer neural network with ReLU activations. Then for any and x E X, and any perturbation us {U:]a= such that ||U:||2 S dllWillz. the change in the output of the network can be bounded as follow,
Lull
Ifu+us (x) - fwo S eB
Wall
(21)
WW
Since our proof is also based on Lemma 2
it is analogous to the analysis of generalization bound in (Neyshabur al. 2018) and is essentially the same for the first part.
Proof. The proof involves two parts. In the first part, we show the maximum allowed perturbation of parameters as shown (Neyshabur et al. (018). the second part, we show that the margin difference of the models trained on Sy and S₁ is also bounded by the perturbation of parameters. Let (II 1l2),, and consider network with normalized weights W, [W
8 W.. Due
to the homogeneity of the ReLU, we have fwr = fww We can also verify that (II [W]l))) sufficient to prove the Theorem only for the normalized weights w, and hence w.l.o.g assume that for any layer i, ||W/||2 = B.
I [alla and ||W]|e [[ille Therefore, ?Wil [Will
Choose the distribution P of the prior w to be N(0,021), and consider the random perturbation us (N,00 (U.)_.. Since the prior cannot depend the learned model w its norm, we set based on the approximation 3. For each value of & on a pre-determined grid, we compute the PAC-Bayes bound, establishing the generalization guarantee for all w for which |3
s 13,
and ensuring that each relevant value of B is covered by some 3 on the grid. We then take union bound over all 3 on the grid. For now, we consider a fixed 3 and the w for which | - 31 < 13, and hence g < fd-1 egdd1
14

---

Under review as a conference paper at ICLR 2023
Since us 12), we get the following bound for the spectral norm of U, Tropp. 2012):
Pus:Wo..=:/(lllilll > f 2he /2/2h0"
(22)
Taking union bound over the layers, we get that with probability at least /2 the spectral norm of perturbation of U in each layer is bounded by / 221n(22hh). Plugging this spectral norm bound into Lemma 2 we have that with probability at least 1 the maximum allowed perturbation bound
v2
is:
||Uella 3
max fouaex/lw-us(e (x) - (x)) SeBB" Σ
S dBdd /2hnn((2d))) S 4
(23)
where a 12dBi4-Lo 22222n(22h) Then we can compute the difference of expectation margins for
fo which is trained on Sy and which is trained Sy. Firstly, compute the difference margins for any model Is trained on S DD and the target model fpp For any verified dataset $ E DD
F[r(Js m)) F(y(fD,
= =|E(sm ms (x) jw (x) ((Sw(a)) marj (a)
=I(E(sw+ur (x)(k() E(sw (x) - tt (2) Elmaz (x)
SmoTwEN (Jwtus (x) [y] lw (y]) + maTwEx (mt [ mar Sw (x)
22ma (z) fw (x) s 2
(24)
So, for fv trained on Sy and fi trained on S₁, we have with probability at least that the predictions margins are bounded by e:
|E(p(fv, E(p( fol.fz.10 x))] <|E(p(fv,x)) E(p(fr x)) E(p(fr,))) E(p(fo).
(25)
Sc.
B IMPACT OF REVEALING MORE PRIVATE SAMPLES ON FALSE POSITIVES
7.5
0.2
0.0
0.0
60
20
40
60
80
100
Figure 4: Left: Comparison the verification confidence of fy and fr. FP becomes stronger (lower p-value) more samples revealed. Right: same comparison, however, we include fo to show the desirable behaviour an independent model.
In Figure 4 we show the results for verification, using Blind Walk, with more data (up to k: 100 private samples). As we increase the number of revealed private samples, the confidence of DI increases both for fy (true positive) and fz (false positive).
C RELATED WORK
Model extraction detection and prevention. Detection methods rely on the fact that many ex- traction attacks have querying patterns that are distinguishable from the benign (Juuti et al.
15

---

Under review as a conference paper at ICLR 2023
2019 Atli et al. 2020 Zheng al. 2022 Quiring et al. 2018). All of these can be circumvented by the adversary who has access to natural data from the same domain as the victim model (Atli et all, 2020). Prevention techniques aim to slow down the attack by injecting the noise into the prediction, designed to corrupt the training of the stolen model (Orekondy et al. 2020 Lee et al.) 2019 Mazeika et al. 2022), by making all clients participate in consensus-based cryptographic protocols (Dziedzic et al. 20022.. Even though they increase the cost of the attack, they do not stop a determined attacker from stealing the model.
Ownership verification. There exist many watermarking schemes for neural networks (e.g. ZZhngg ct al. 2018: [Chida al., 2017 et al. (2188) that have the same goal as DI does. However they were shown to be brittle (Lukas et al. 2022p. It was shown that adversarial examples (Lukas et al.. 2021 can be used to fingerprint a model or to watermark the dataset (Sablayrolles et al. 2020).
However, adversarial training can be used to weaken both schemes (Lukas et al., 2021: Szyller & Asokan 2022). On the other hand, if a model is sufficiently vulnerable to membership inference attacks, it can be used to fingerprint it (Pan al. 2022
D VERIFICATION WITH MORE NOISE
Table 4: Impact of the amount of noise (maximum number of perturbation steps) added during the verification on the success of DI (baseline 50 steps). Using more noise does not prevent FNs against fA. However, it increases the standard deviation across all experiments, and has negative effect on the verification of fo- We provide the mean and standard deviation computed over five runs. Verification done using k 10 private samples. ENs highlighted in red.
Model
Accuracy
Steps
A
p-value
fv
0.92 0.01
50
1.59 = 0.04
10 10-21 + 10
25
0.09
0.04
0.09 0.07
50
0.12 + 0.06
0.15
SA
0.86 + 0.01
100
0.10 + 0.05
0.08 0.09
200
0.14 1: 0.08
0.16
50
0.29 0.12
0.46 :: 0.04
fu
0.64 + 0.02
100
0.19 0.16
0.37 0.12
V who suspects that A might be using ad adversarial training to avoid the detection, can carry out the verification with more noise order to escape the guarantees provided by adversarial training to A.
In the experiments presented in Section 4 the average noise added during Blind Walk is 0.12 + 0.05 (under ),o), and adversarial training is done with y 10/255(= 0.039). In this experiment, we vary the number of maximum steps taken by V. and hence the maximum amount of noise added during the verification. We consider (25, 50, 100, 200} steps (baseline 50 steps) which corresponds to {0.10 + 0.03, 0.12 + 0.05, 0.33 + 15, 0.38 + 23) noise added (under fxx) during the verification.
Since V does not know which fsp indeed stolen, addition to fA, we also conduct this experi- ment for fo (for (50, 100} steps).
In Table 4 we provide the results for the experiments with different amounts of noise. Using more steps does not improve the result against fA compared to the baseline: 1) the standard deviation of the p-value increases; 2) we do not observe any linear relationship between the noise and A or the associated p-value.
On the other hand, the confidence of the verification of fo decreases. The standard deviations of Áμ and its associated p-value increase. Nevertheless, the p-value remains sufficiently high.
In conclusion, increasing the amount of noise during Blind Walk does not allow V to circumvent A's adversarial training. Hence, DI remains susceptible to false negatives induced by adversarial training.
16


Reviews:
[{"review_id": "bDx8R2ROLd", "reviewer": "Reviewer_TtqN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the dataset inference mechanism from an offensive perspective. The paper starts with the theoretical analysis of the failure scenarios, such as the existence of false positives. The paper analyzes linear cases and then extends it to non-linear cases. The paper empirically demonstrates that there are indeed false positives when the victim uses a smaller number of queries. The paper also shows some potential cases of false negatives when the attacker employs adversarial training. The paper concludes with the possible limitations of the dataset inference.\n", "summary_of_the_review": "The paper studies the weaknesses of the dataset inference mechanism. I agree that exposing a protection mechanism's weaknesses is essential to understand its possible risks. However, the weaknesses this paper shows seem not to be the actual weaknesses of the dataset inference. I can safely imagine that those weaknesses are not any concern in practical scenarios. Thus, I am leaning a bit more toward rejecting this paper.", "strength_and_weaknesses": "Strengths:\n\n1. This paper suggests a false sense of security that the dataset inference mechanism may make.\n2. The paper empirically shows some downsides of the dataset inference mechanisms.\n\nWeaknesses:\n\n1. It's a bit unclear why reducing the number of queries is important.\n2. It's also unclear why the existence of false positives is the weakness.\n3. It's further unclear why the attacker performs adversarial training to construct a model.\n4. The paper has large room for improving the quality of the writing and the presentation.\n\nDetailed comments:\n\nI like the research question that this paper asks: \"what would be the failure modes of the dataset inference?\" which is the highlighted work in the last year's ICLR. The paper approaches this question theoretically and empirically, and it shows in the evaluation that there are such cases.\n\nHowever, the major problem of this paper is that the failure modes that this paper exposes do not seem to be the actual weakness of the dataset inference. Moreover, some scenarios, such as an adversary training a model with adversarial training (for evasion), are misleading. Thus, I believe the paper is not ready to appear in ICLR 2023.\n\nHere, I provide my detailed comments on the weaknesses.\n\n\n[False Positives with Smaller Number of Queries]\n\nIf I understood the threat model correctly, the attacker trains a model with the stolen dataset, and the victim wants to identify whether the attacker uses the stolen data. In this case, the model trained by the adversary is already open to the victim; thus, I am a bit confused why we assume a smaller number of queries bounds the victim. Of course, reducing the number of required queries is good for efficacy. However, as long as the victim can query the model sufficiently, the dataset inference seems to work.\n\nMore generally, I imagine a scenario where two parties are at the court, and one party has to show that the other party's model uses the data extracted from the first party's model. In this case, unless the verification process takes days or years, we could allow the victim to do as many queries as possible. Running 50k queries on CIFAR10 only takes a few minutes (at most).\n\n\n[Existence of False Positives]\n\nI am a bit skeptical about the problem of false positives. Just having some false positives is insufficient to claim that the dataset inference is unsafe. I would expect some more strong evidence or claims, such as \"even if the victim uses 50k queries, the false positive is still around 50%.\" It is, then, indeed a problem.\n\nI like the results showing that if the number of private samples is large, the false positives become higher. But on second thought, it's somewhat trivial as the victim has to make predictions on many private samples for the ownership claim. Moreover, there is no reason why the victim wants to use a larger number of private samples. I would imagine if the victim uses a smaller number of private samples, then correctly estimating the membership of a subset of the entirety could be sufficient to claim the ownership.\n\n\n[Use of Adversarial Training as a Defense]\n\nThe paper shows that an adversary can increase false negatives by training their model using adversarial training. This is confusing because I could not imagine a scenario where the adversary harms the accuracy of their model by running adversarial training. One of the purposes of stealing private data is to achieve state-of-the-art accuracy; then, running adversarial training contradicts the motivation.\n\nI also assume that many trivial things could achieve the same as running adversarial training. One example would be running with DP-SGD. As the dataset inference is the membership identification, training a model with DP can reduce the inference success. BUT, the private model trained by the adversary cannot perform well; even for CIFAR10, it achieves ~75% accuracy with the epsilon 7-8.\n\n\n[Minors]\n\n1. The citation is not compatible with the ICLR; \"Deng et al. (2022)\" is better \"[Deng et al., 2022].\"\n2. Many notations appear over reading; better to summarize by creating the \"Notations\" section.\n3. Sec 2 largely depends on the setup proposed by Maini et al. and omits many details; for example, \"what are the embeddings?\" or \"what is the white box (MinGD)?\" It would be nice to re-write the entire section so that readers without prior knowledge of the dataset inference can understand the details clearly.\n4. It is a bit difficult to understand the paragraph starting with \"The authors ...\"; it should be re-written.\n5. The introduction contains sufficient related work, so I wonder if the paper needs more in Sec 6.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ky0omJ1YkE", "reviewer": "Reviewer_PqLk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work does a deep dive into dataset inference, an up-and-coming ownership verification technique for machine learning models. The authors show how DI suffers from a nontrivial FPR in both theoretical and practical settings and FNRs in practical settings with simple adversarial modifications. These analyses highlight how DI can falsely accuse trainers of stealing data, while at the same time failing to detect actual stealing adversaries.", "summary_of_the_review": "The paper makes some very good arguments about the validity and utility of dataset inference, with both theoretical analyses and empirical experiments that bring to light the existence of false positive and false negative cases, which can be harmful in practical scenarios. Apart from a few minor comments and some hiccups in the proofs (which I think might be a mix of some typos and unexplained intermediate steps, so nothing that can't be fixed), I think it would be a good addition to ICLR.", "strength_and_weaknesses": "## Strengths\n\n- Discussion on False Positives starting with the theoretical analyses drives the point home, showing that the flaw is critical to the approach itself, and not just an empirical loophole explored by the adversary. All the proofs are detailed, thorough, and easy to follow.\n\n- Analyses are thorough, with proper visuals and numbers (with error margins, to highlight variance and stability of results), as well as realistically-sized models and datasets. All the figures are very helpful in understanding the points being made with their aid.\n\n- The inclusion of False Negatives is very useful and much needed; any adversary that is malicious enough to steal data would not stop from going a step further and evading such detection techniques. It definitely opens up the sub-field to the same kind of cat-and-mouse game that adversarial machine learning is subject to, which isn't necessarily a bad thing (and can be good in fact).\n\n## Weaknesses\n\n- I'm not convinced about the applicability of Theorem 2. It states that False Positives occur with P at least 0.5, but this is true when _one_ sample is used for DI, right? Assuming $n$ samples are used, wouldn't the expected FPR bound decay with $2^{-n}$?\n\n- Section 4: to account for potentially perturbed inputs, can the party executing Blind Walk not simply adjust to a higher step size (perturbation)? At least some form of adaptive defense should be evaluated for a complete picture.\n\n- I have some concerns with some of the reductions, especially in Appendix B. They can probably be fixed by showing more intermediate steps, but would like some clarification from the authors:\n\n  - In (11), $c=m\\cdot u\\cdot y^2$ ; since this has a $y$ term, how can it be treated as a constant?\n  - In (12), $c$ from (11) becomes $y\\cdot y$ (which makes me wonder if (11) has a typo), and the $y$ term outside the summation (present in (11)) seems to be missing here.\n  - \"Note that in Equation 12...\" here assumes that each $y_i=1$, which may not be true- should the term here not be $D\\sigma^2\\sum_{i} y_i$ instead?\n  - In (13), the same $yc$ term is used, but would it really be the same? \n  - Assuming $\\Phi$ in (18) refers to the Normal CDF: 0.9 here seems like an arbitrary choice. Also, is $\\Psi(-\\frac{1}{\\sqrt{10}})=0.38$? Not sure where the 10 or 0.9 appears from this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "H6FxF6-Vql", "reviewer": "Reviewer_ZZmg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes the reactive defense against model stealing called dataset inference that was proposed at ICLR 2021. It is shown in this submission that dataset inference suffers from false positives (FP) and false negatives (FN). For FPs - it is presented that dataset inference can incorrectly resolve the model ownership when the independent model is trained with non-overlapping data from the same distribution as in the victim model. For FNs - it is claimed that dataset inference can be fooled when the stolen model is trained using adversarial training. ", "summary_of_the_review": "The paper requires more experiments, for example, to consider the whole spectrum of the number of data points used for dataset inference in the case of neural networks. The assumptions in the theoretical part are different than in the original work. Additionally, no suggestions are made that could improve the dataset inference defense. ", "strength_and_weaknesses": "Strengths:\n\n1. The authors provide a check of the defense technique proposed at ICLR 2021.\n2. The paper provides both theoretical and experimental analysis.\n\nWeaknesses:\n\n1. Dataset inference is a statistical and not a deterministic method so the existence of FP (False Positives) and FN (False Negatives) is not surprising. The only important aspect is under which conditions such false alarms occur. Note that the ownership resolution is done through \"statistical hypothesis testing, which takes the false positive rate $\\alpha$ as a hyper-parameter and produces either conclusive positive results with an error of at most $\\alpha$, or an 'inconclusive' result.\"\n1. Regarding False Negatives - Adversarial training is more expensive thus stealing with this approach is not done. Additionally, adversarial training usually lowers the performance of a model (even in the experiments in this paper the authors report the drop in accuracy by 6pp) - which is why an adversary would not use it - the stolen service would be worse than the service exposed by competition - the victim. In Table 2, the accuracy values should be reported, as mentioned at the end of Section 4.\n2. The assumption: \"(2) a large proportion of the victim’s training data is used during ownership resolution/verification\" can be fulfilled also by assuming a third-trusted party - an arbitrator - which is a realistic assumption for an ownership resolution (when considered as a court case). Note that revealing any data to an adversary might enable the adversary to retrain their model on the revealed data or simply claim ownership of the data. The approach with private or oblivious inference is also feasible, however, as pointed out - it is computationally intensive (hardware-based solutions might alleviate this overhead).\n4. It is not shown how to obtain a false negative when a model is stolen from a private API by other means than decreasing the number of samples used for the ownership resolution. With respect to Section 5: \"False negatives vs white-box theft.\": What changes does an adversary have to make to fool dataset inference in the white-box setting when the victim model is simply copied by an attacker? For example, how should the adversary change the outputs from their stolen model to avoid detection by the dataset inference? \n7. False negative is based only on the case where an adversary steals the private dataset but not when the adversary steals a model behind a publicly exposed API.\n8. In Section 3.2.2 it should be also shown how dataset inference performs depending on the number of samples used for tests. The authors used the number of samples $k=10$, which is \"extremely low\", as stated at the end of Section 3.1 in this submission. A similar Figure to Figure 1 should be provided in Section 3.2.2. The authors showed a single/corner case instead of thoroughly analyzing the spectrum of possible cases for dataset inference. It is mentioned at the end of the caption for Table 1 that FPs become more significant as $k$ increases. This should be shown experimentally, at least in the appendix. The probability of an FP should be shown as the fraction (from 0 to 1) of revealed private samples (similarly to Figure 1).\n9. Blind Walk and MinGD are two ways to estimate the prediction margin introduced in the dataset inference paper. For example, Tables 1 and 2 should be run for many more samples than $k=10$ and also with MinGD (not only with Blind Walk).\n10. The assumptions for the theoretical results in Section 3 are different than those in the original paper. These should be stated explicitly. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject