On the Robustness of Dataset Inference
Authors: Sebastian Szyller
Rui Zhang
Jian Liu
N Asokan
Venue: Accepted by TMLR ()

Abstract: Machine learning (ML) models are costly to train as they can require a significant amount of data, computational resources and technical expertise. Thus, they constitute valuable intellectual property that needs protection from adversaries wanting to steal them. Ownership verification techniques allow the victims of model stealing attacks to demonstrate that a suspect model was in fact stolen from theirs. Although a number of ownership verification techniques based on watermarking or fingerprinting have been proposed, most of them fall short either in terms of security guarantees (well-equipped adversaries can evade verification)  or computational cost. A fingerprinting technique, Dataset Inference (DI) has been shown to offer better robustness and efficiency than prior methods. The authors of DI provided a correctness proof for linear (suspect) models. However, in a subspace of the same setting, we prove that DI suffers from high false positives (FPs) -- it can incorrectly identify an independent model trained with non-overlapping data from the same distribution as stolen. We further prove that DI also triggers FPs in realistic, non-linear suspect models. We then confirm empirically that DI in the black-box setting leads to FPs, with high confidence. Second, we show that DI also suffers from false negatives (FNs) -- an adversary can fool DI by regularising a stolen model's decision boundaries using adversarial training, thereby leading to an FN. To this end, we demonstrate that black-box DI fails to identify a model adversarially trained from a stolen dataset -- the setting where DI is the hardest to evade. Finally, we discuss the implications of our findings, the viability of fingerprinting-based ownership verification in general, and suggest directions for future work.

Published in Transactions on Machine Learning Research (06/2023)
On the Robustness of Dataset Inference
Scbastian Szyller
contacl@sccozyller.cco
Aolto University
Rui Zhang Zhejiang University
(anngrui98@@jj..duu.co
Jian Liu Zhefiang University
1iujian 1ww.:/241112/uucuuccu
N. Asokan University of Waterloo & Aalto University
asokan @acm.org
Reviewed on OpenReview: ht tp ei ew. net d
Abstract
Machine learning (ML) models are costly to train as they can require significant amount of data, computational resources and lechnical expertise. Thus, they constitule valuable intellectual property that needs protection from adversaries wanting to steal them. Owner- ship verification techniques allow the victims of model stcaling attacks to demonstrate that suspect model was in fact stolen from theirs.
Although number of ownership verification techniques based on watermarking finger- printing have been proposed, most of them fall short either in terms of security guarantees (well-equipped adversaries can evade verification) or computational cost. A fingerprinting technique, Dataset Inference (DI), has been shown to offer better robustness and efficiency than prior methods.
The authors of DI provided correctness proof for linear (suspect) models. However, in subspace of the same setting, prove that DI suffers from high false positives (FPs) it can incorrectly identify
independent model trained with non-overlapping data from the same distribution stolen. We further prove that DI also triggers FPs in realistic, non-linear suspect models. We then confirm empirically that DI in the black- box setting leads to FPs, with high confidence.
Second, we show that DI also sulfers from false negatives (FNs) an adversary can fool DI (at the cost of incurring some accuracy loss) by regularising a stolen model's decision boundaries using adversarial training, thereby leading to FN. To this end, demonstrate that black-box DI fails to identify model adversarially trained from stolen dataset the setting where DI is the hardest to evade.
Finally, we discuss the implications of our findings, the viability of fingerprinting-best based ownership verification general, and suggest directions for future work.
1 Introduction
Machine learning (ML) models are being developed and deployed at an increasingly faster rate and in several application domains. For many companics, they are not just part of the technological stack that offers an odge over the competitors but core business offering. Hence, ML models constitute valuable intellectual property that needs to be protected.

---

Published in Transactions on Machine Learning Research (06/2023)
Model stcaling is considered one of the most serious attack vectors against ML models (Kumar et al. 2019 The goal of a model stealing attack is to obtain a functionally equivalent copy of victim model that can be used, for example, to offer a competing service, or avoid having to pay for the use of the model.
In the white-boz attack, the adversary obtains the exact copy of the victim model, for example by reverse engineering an application containing an embedded model (Deng et al. 2022). In contrast, in black-bor attacks (known as model extraction attacks) (Papernot et al. [2017] [Orekondy et al.] 2019 [Tramrr et al. 2016) the adversary gleans information about the victim model via its predictive interface.
Two possible 2020 Zheng
approaches to defend against model extraction are 1) detection (Juuti et al.. 2019 Atli et al.
ct al. 2022) and 2) prevention (Orekondy et al., 2020 Mazcika et al. 2022 Dziedzic et al. 2022 However, a powerful, yet realistic attacker can circumvent these defenses (Atli et al. 2020).
An alternative defense applicable to both white-box and black-box model theft is based on deterrence. It concodes that the model will eventually get stolen. Therefore, an ownership verification technique that can identify and demonstrale suspect model as having been stolen can serve a deterrent against model theft. Early research in this field focused on watermarking based embedding triggers or backdoors (Zhang et al., 2018 Uchida et al.) 2017 Adi et al. 2018) into the weights of the model. Unfortunately, all watermarking schemes were shown to be brittle [Lukas et al. 2022) in that an attacker can successfully remove the watermark from a protected stolen model without incurring a substantial loss in model utility.
An alternative approach to ownership verification is fingerprinting. Instead of embedding a trigger or back- door in the model, one can extract a fingerprint that matches only the victim model, and models derived from it. Fingerprinting works both against. white-box and black-box attacks, and does not affect the performance of the model. Although several fingerprinting schemes have been proposed, some not rigorously tested against model extraction (Cao et al. 2021 Pan et al. 2022 and others can be compulationally expensive to derive (Lukas et al., 20213.
In this backdrop, Dataset Inference (DI) Maini et al. 2021 promises to be an effective fingerprinting mechanism. Inluitively, it leverages the fact that if model owners trained their models on privale dala, knowledge about that data can be used to identify all stolen models. DI was shown to be effective against whito-box and black-box attacks and is efficient to compute (Maini et al. 2021). It was also shown not to conflict with any other defenses (Szyller& & Asokan 2022). Given its promise, the guarantees provided by DI merits closer examination.
In this work, we first show that DI suffers from false positives (FPs) it can incorrectly identify an independent model trained with non-overlapping data from the same distribution as stolen. The authors of DI provided correctness proof for a linear model. However, DI in fact suffers from high FPs, unless two assumptions hold: (1) large noise dimension, as explained in the original paper and (2) large proportion of the victim's training data is used during ownership verification, as we prove in this paper. Both of these assumptions are unrealistic in subspace of the linear case used by DI: (i) we prove that a large noise dimension can lead to low accuracy in the resulting model, and (ii) revealing too much of the victim's (private) training data is detrimental to privacy. Furthermore, we prove that DI also triggers FPs in realistic, liner models. We then confirm empirically that DI leads to FPs, with high confidence in the black- box verification setting, "black-boz DI", where the DI verifier has access only to the inference interface of a suspect model, but not its internals.
We also show that black-box DI suffers from false negatives (FNs): an adversary who has in fact stolen victim model can avoid detection by regularising their model with adversarial training. We provide empirical evidence that an adversary who steals the victim's 's dataset itself and adversarially trains a model can evade detection by DI by trading off accuracy of the stolen model.
We claim the following contributions:
Following the same simplified theoretical analysis used by the original paper (Maini el al. 2021), in a
subspace of the linear case used by DI, we show that for a linear suspect model, 1) high digensimnsio noise (as required in (Maini et al. 2021 leads to low model accuracy (Lemma 3.3 Section 3.1)
and 2) DI suffers from FPs unless large proportion of private data is revealed during ownership verification (Theorem 3.2 Section 3.1
2

---

Published in Transactions on Machine Learning Research (06/2023)
Extending the analysis to non-linear suspect models, using a PAC-Bayesian framework (Neyshabur et. al. 2018), we show that DI suffers from FPs in non-linear models regardless of how much private data is revealed (Theorem 3.4 Section 3.2.1
We empirically demonstrate the existence of FPs in a realistic black-box DI setting (Section 3.2.2
We show empirically that black-box DI also suffers from FNs: using adversarial training to regu- larisc the dccision boundarics of a stolen model can successfully evade detection by DI at the cost of some accuracy (+s 6-13pp) (Section
2 Dataset Inference Preliminaries
Dataset Inference (DI) aims to determine whether a suspect model fsp was obtained by an adversary A who has stolen a model (fA) derived from a victim V's private data Sy; or belongs to an independent, party I (fz). DI relics on the intuition that if model is derived from Sv, this information can be identified from all models. DI measures the prediction maryins of suspect model around private and public samples: distance from the samples to the model's decision boundaries. If fsP has distinguishable decision boundaries for private and public samples DI deems it to be stolen; otherwise the model is deemed independent.
In the rest of this section, we explain the theoretical framework that DI uses consisting of a linear suspect model - the embedding generation necessary for using DI with realistic non-linear suspect models, and the verification procedure. A summary of the notation used throughout this work appears in Table [
2.1 Theoretical Framework
The original DI paper (Maini et al. 2021) used a linear suspect model to theoretically prove the guarantees provided by DI. We first explain how DI works in this sctting.
Setup. Consider a data distribution D, such that any input-label pair (ar can be described as:
y (11,,11,, = y uERR 22 ~ o((0
where x =x11 x2) E KDD and R is fixed vector. The last D dimensions of æ represent Gaussian noise (with variance )²).
Structure of the linear model. Assuming a linear model f, with weights w = (w1, W2), such that f(x) W1 I1 + W2 22, then the final classification decision is sgn(f(x)). With the weights initialized to zero, / learus the weights using gradient descent with learning rate until yf(x) is maximized. Given a private training dataset Sy D {(x(t), yl))ji I,..., m), and public dataset Sn D (both of size m),
then W1 mu and W2 = EE (0) regardless of the batch size.
In DI, the prediction margin p(-) is used to imply the confidence of f in its prediction. is defined as the margin (distance) of a data point from the decision boundary.
p(x) 2 y. f(x).
(1)
The authors (Maini al. 2021) show that the difference of expected prediction margins of two datasets Sy and So is = The threshold can be set X E (0. Do²), and by estimating the difference of the prediction margins on S, and Sy on fsp, DI is able to distinguish whether that model is stolen.
Note that DI uses approximations the prediction margins based on embeddings. The theoretical framework assumes that the approximations are accurate, and we can use them directly for the theoretical analysis (Equation D For the linear model, the margins can be computed analytically; however, in Section 2.2 we explain how the approximations of the margins obtained.
3

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Summary of the notation used throughout this work.
V
the victim
& model
I
an independent party
fv
a model trained on Sy
A
an adversary
fo
a model trained on So
S
dataset
fI
model trained on S A's model
Sv
V's private dataset.
fA
So
public dataset I's data
fSP
suspect model
S₁
w
model weights
D
distribution that all datasets follow sample from DD
gv
regression model
(z. y)
D
noise dimension
2.2 Embedding Generation
In order to use DI one needs to generate embeddings of the samples. V queries their model fv with samples in their private dataset Sy and public dataset So, and assigns the labels b = 1 and b b= 0 respectively. The authors propose two methods of generating the embeddings: a white-box approach (MinGD) and a black-box only Blind Walk it oulperforms MinGD in most experimental
one (Blind Walk). In this work, setups in the original work, and more realistic, as it only requires access the API of the suspect model.
Blind Walk estimates the prediction margin of sample by measuring its robustness to random noise. Por sample (x, y), to compute the margin, first choose a random direction 8, and take k E N steps in the same direction until the misclassification f(x + kō) + y. This is repeated multiple times to increase the size of the embedding. As reported in (Maini et al. 2021 obtaining embeddings for 100 samples can take up to 30, 000 queries.
Having obtained the embeddings, V trains regression model gv that predicts the confidence that sample contains private information from Sy.
2.3 Ownership Verification
Using the scores from gy and the membership labels, V creates vectors e and cy of equal size from Sy and So, respectively. Then for null hypothesis Ho μ pV where μ and p cv are mean confidence scores. The test rejects Hu and rules that the suspect model is 'stolen', gives inconclusive result.
To verify whether fsr is stolen or independent, V obtains the embeddings by querying the model (us- ing Blind Walk) using samples from Sy and Sp. Then they use the embeddings to obtain the confi-
0,486
k=50
0.437
k=1000
Probability of an
0.400
dence scores from the gy, and perform hypothesis
test on the two distributions of scores.
0.309
k=10000
3 False Positives in Dataset Inference
0.200
k=50000
To generate the embeddings for specific sample the private dataset Sv, DI requires querying the
0.132
0.100
0.0
0.2 0.4 0.6 0.8
1.0
suspect model fsp hundreds of times. To reduce the total number of queries, DI was shown to be effective with only 10 private samples with at least 95% confidence. Additionally, DI requires a large random noise dimension D such that probability of success increases to as - oo. In this section, we prove that these two assumptions are not realistic in a subspace of the linear case used by DI: 1) DI is susceptible false positives (FPs) unless V reveals large number of samples; 2) large D will harm the utility of the model (Section 3.1)
Fraction of tested samples
Probability an FP as the fraction of re- vealed private samples for model (Equation 6p. V needs to use many private sam- plcs to guarantee low false positive rate.
Figure 1:
for a linear suspect
4

---

Published in Transactions on Machine Learning Research (06/2023)
Furthermore, find that the theoretical results linear suspect models which say that the margins different models are distinguishable with some strict conditions do not hold for more realistic non-linear suspect models. Using a PAC-Bayessan margin based generalization bound (Neyshabur et al. 2018) we prove that models trained on the same distribution are indistinguishable, and will trigger FPs (Section 3.2.1 Next, we provide empirical evidence for the existence of FPs (Section 3.2.2p.
3.1 Linear Suspect Models
In Section 2 we have a set up for linear models. DI shows their success theoretically on the linear set up where u E R.K and a TR.
Theorem 3.1 (Success of DI Maain et al., 2021)). Choose b +- {0. 13 uniiormly random. Given an adversary's 's linear classifier f trained on S D, s. = Sv if 0, and on Sy otherwise. The proba- bility V correctly decides if on adversary stole its knowledge P[ = bj = 1 11-4(-- 2 2-2222) Pimp=sP(l(f..y;]]) b] =.
vD Moreover,
However, realistic datasets often contain more noise and lack the correct signal. We consider subspace where the signal ||||| is upper-bounded and the noise is larger. We construct the subspace by assuming that |ulla I 1_ and o² > 10.m Note that the linear model should correctly classify most of the randomly picked data from this subspace, use Lemma 3.2 to analyze the accuracy of the linear model in this subspace. The results show that only the subspace where D is bounded can guarantee high performance for the model.
vm
Lemma 3.2 (Need for Bounding Noise Dimension). Let f be a linear model trained on S ~ D. For sample (x. y) sompled from DD which is independent of S, f correctly classifies the sample with a probability
(lyf() 0] = -1 Φb
Voz). Assuming that ||ua12 I Vm and 0² movm the accuracy of f is JD Therefore, the model f trained on the D can achieve high accuracy only if D
mu²
Plyf(x) 2 0] S 1 D
10
is bounded.
The details of the proof are in the Appendix g Lemma 3.2 shows that if the dimension of æ2, which follows W(0,o²), is large, then the noise will dominate f and mislcad it into making incorrect predictions. For example, for a datasel with more than 500 samples, the std of 22 is 0.25 (close to the CIFAR1O dataset) where x² = 0.0625 > 10ym = 10/500 ~ 0.0044. If = 1000, f can correctly classify a sample that is different from f's training set with a probability up to +6 achieve an accuracy of 0.9992.
V1000 < 0.6241. However, if = 10, f can
Since V limits the number of samples used for the verification, Theorem 3.3 show that the false positives (FPs) rate dircctly related to the samples usod for the verification our subspace.
Theorem 3.3 (Existence of False Positives with Linear Suspect Models). Let fi be a linear classifier trained on the independent dataset S₁ D. Let k be the number of samples estimated required for the verification. Then, the probability that V mistakenly decides that fr is stolen model is PP Dy:
1] > Φ( VKD
22/mm
Where & is V's decision function MMaini et al.
2021
1, if fsp fA.
V(SsP,
(2)
0, if fsr ~ fz.
Proof. Recall that V trics to reveal only few samples during the verification. For distribution D where |wl| < vm and o² vm
Following the intuition from DI YYom et al. 2018 for satisfactory performance, DI must minimise both false positives and false negatives. Hence, the objective function is defined as:
P(V(fz,V; 1] P(V(fv, D) =
minA
(3)
2
5

---

Published in Transactions on Machine Learning Research (06/2023)
where the margin of D is estimated using Sy and Sp. Note that only interested in the false positives Sv; D) = 1), let S {(x!0) = S be a subset of S, consisting of l samples.
P| Sv; D) =
PIE (s [Vfr(=)] - E (a (ffz(())
2 >
(ma)e
(0) 22
(
22
(s.
2
x2
x2 x2 2
11)
A
(4)
1 Pl k
m
22 22
(4) (3)
m ΣΣ P
k
(p) 22
2
A/.
k
Recall that 22 ( and 22 (Pp)
are D-dimensional vectors sampled independently from (N0, o²). Using the
central limit theorem we can approximate the terms. We have Σ. y0 (0) NO mo?) Then, we can approximate 1 E m (0 x2 22
(8) (i) by t1 NoO0, & "Poo") and approximate
1 Ex Σ. y 22 22
(5) (p)
by
N(0, mD )²) &
MMaini et al. 2021 Thus, we get N(O, 2mD oa), and
k
P[ Sv: = = ^
2mD
Ply
0² P N
h
2m
(5)
KA
1 +1
12mDo22
where 2 ~~0(0,11 1). The optimal threshold is given as
Do² 2
ViD
Sv: 1]= (
=1
(6)
2 2m
From Equation G we see that the probability of false positives relies on the number of points used for the verification k and the size of D.
In our subspace where D bounded for the model with high accuracy, the success of DI is directly related the number of samples used for the verification. This is similar to the analysis of failure of membership inference in the original paper when the k is extremely low, e.g. only 10 samples. In the DI paper, it was explained that DI succeeds because it calculates the average margin for multiple verification samples; whereas membership inference fails as it relies on per-sample decision. So when the number of tested samples is smaller, the success rate of DI will be close to 0.5, just like for membership inference. In Figure I we show the probability of an FP (Equation 6) for different values of k; even for k = 10000 the probability is 0.309
Hence, even the simple linear setup, D) has false positives with high probability; in particular, when the fraction of tested samples small.
3.2 Non-linear Suspect Models
Having demonstrated the limitations of the lincar model, we now focus on non-lincar suspect models. The intuition is based on the margin-based generalization bounds. Note that the generalization bounds states that the expected error of the margin based loss function is bounded, and the bound is mostly related to the distribution (Neyshabur et al. (018). Since DI assumes all the datasets follow the distribution D, our intuition is to directly use the generalization bounds and the triangle inequality to prove the similarity of the models trained on the same distribution.

---

Published in Transactions on Machine Learning Research (06/2023)
3.2.1 Theoretical Motivation
Let fw be a real-valued classifier fw : X + R², |||||| - B with parameters w = (Wi) For any distribution D and margin p(ʃ. .) = [x)]yy my [z) S 7; where y > 0. The margin is the same as for the linear model with labels y E (-1, +1}. Then, we define the margin loss function as:
(), = ((()))) maz <]
(7)
Note that the PAC-Bayes framework (Neyshabur et al. 2018 provides guarantees for any classifier f trained on data from a given distribution. We define the expected loss of a classifier f on distribution DD as Lp == ((e.))>((((( (z), y)] and the empirical loss on a dataset S as Ls = m [ Then, for layer feed-forward network f with parameters w {W.) and ReLU activation (Neyshabur et al. 2018p. The empirical loss is very close to the expected loss. For any o, y 0, with probability 1 a over the training set, have:
(DD(fs) Ls(ss) s O((),
(8)
B'hhin(dh)) IT,,, where 6
IB Σ +In
11
dm
1
and h is the upper bound dimension for (Wi
7.m
This PAC-Bayes based generalization guarantee states that for model f, the distance between the empirical loss and the expected loss is bounded, and the bound can be very small when the model's margin is large, Thus, we can expect that the margins of f on any dataset that follows given distribution to be bounded by (c). This contradicts the intuition of DI. DI's intuition is that the difference of margins of two datasets Sy and Sj on the model f is large enough such that DI able to distinguish whether the model f is stolen.
Moreover, since DI assumes that Sy and S, follow the same distribution D, we can show that the margins for fy and fi are similar each other.
Theorem 3.4 (k-independent False Positives with Non-linear Suspect Models). For the victim private dataset Sy D and an independent dataset S₁ D, let fw be a d-layer feed-forward network with ReLU activalions and parameters w (Wi) Assume that. fy is trained on Sy and fi is trained on St, Jv and fr have the same structure. Then, for any B, h, and any X, there erist prior P on w, s.t. with probability at least 2
[F(((Vv,)) p(fz,x))) Sc.
The details of the proof are in the Appendix 9 Hence, for any two models trained on the same distribution, the expectation of margins for any sample similar. Given that DI works by distinguishing the difference of margins for two models, it will result in false positives with probability at least 1/ (Theorem 3.4
3.2.2 Empirical Evidence
Having proved the existence of FPs for non-lineat models, we now focus on empirically confirming it.
First, recall the original experiment setup (Maini et al. 2021 ; let us consider the following two models: 1) fv trained using Sy, and 2) fo trained using Sg. In the original formulation, e-g. for CIFAR10, CIFAR10-train (50, 000 samples) is used as Sv, and CIFAR10-test used as So (10, 000 samples). Recall that uses their Sy and Sp to obtain the embeddings that are then used to train the regression model gv.
DI was shown to be effective against several post-processing used to obtain dependent models which are expected to be flagged as stolen true positives However, the independent model fo is trained on Sy the same data that is used to train gv. This means that the same dataset. So is used both to train gv and subsequently, to evaluate it. This is likely to introduce a bias that overestimates the efficacy of gy and DI as whole.
To address this, and test whether DI works for more reasonable data split, we the following setup:

---

Published in Transactions on Machine Learning Research (06/2023)
Sp
400
300
Sy
250
6.72
Sp-mean=0055
300
Sy Sy-man--O.55
200
Spmman--0.3
200
Count
1.50
200
150
100
100
50
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Confidence score
Confidence score
Confidence score
Figure Left to right: Svs SI. fo. Comparison of distributions of the confidence scores assigned to the embeddings by gv. Ap smaller for fi than for fy but large enough trigger an FP.
5
0.8
7.5
0.6
fy
5.0
0.4
2.5
0.2
0.0
20
60
80
100
40 60
80
100
m
Figure 3: Left: Comparison of the verification confidence of fv and fz. FP becomes stronger (lower value) as more samples are revealed. Right: same comparison, however, we include fo to show the desirable behaviour of an independent model.
1) randomly split CIFAR10-train into two subsets (Atrain and Birain) of 25, 000 samples cach;
2) assign Sy Atraini and train fy using it;
3) continue using CIFAR10-tesI So (nothing changes), and train So using it;
4) @v is trained using the embedding for So and the new Sv, obtained from the new fyy
5) assign S₁ = Btrain, independent data of a third-party I, who trains their model fz-
This way, we have an independent model fr that was trained on data from the same distribution D as Sy but data that was not seen by 5h We use an analogous split for CIFAR100.
Recall that to determine whether the model is stolen, DI obtains the embeddings for private (Sv) and public (Sg) samples. Then it measures the confidence for each of the embeddings using the regressor gy. For a model derived from V's Sv, the mean difference (Aμ) between the confidence assigned to Sy and So should be large. If the model is not derived from Sy, the difference should be small. The decision is made using the hesis test that compares the distributions of measures from gy-
In Figure 2 we visualise the difference in the distributions for three models. For fy we observe two separable distributions with a large (Аμ), while for fo the difference is small
is working as intended. However, for fz, even though Ap is smaller than for fy is sufficiently large to reject Ho with high confidence. Therefore, fi marked as stolen, a false positive, In Table 2 we provide Ap and the associated p-values for multiple random splits. In Figure 3 we show the results for verification, using Blind Walk, with more data (up to & 100 private samples). As increase the number of revealed private samples, the confidence of DI increases both for fy (truc positive) and fz (false positive).
We discuss the implications of our findings Section 6
We the official implementation of DI, together with the architectures and training loops. Our changes limited the data splits only.

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: Verification of an independent model trained on the same data distribution triggers an FP. Also, we report the accuracy of the models on the test set. We provide the mean and standard deviation computed across five runs. Verification done using l 10 private samples. FPs become more significant & increases. FPs are highlighted.
CIFAR10
CTFAR100
Model
Accuracy
Aμ
p-value
Accuracy
Ap
p-value 30 1 10-30
fv
0.87 0.03
1.62 + 0.08
10 10 10
0.65 0.00
1.80 + 0.04
fz
0.87 = 0.03
1.14 0.12
10 = 10
0.66 + 0.01
0.32 = 0.03
10 - 10 10
0.64 0.02
-0.29 1 0.12
0.46 0.04
0.51 0.01
-0.37 + 0.11
0.67 + 0.15
4 False Negatives in Dataset Inference
Having demonstrated the existence of false positives, we now show that DI can suffer from false negatives (FNs). A can avoid detection by regularising fA, and thus changing the prediction margins. This turn, will mislead DI into flagging fA as independent.
Recall that Blind Walk relies on finding the predic- tion margin by querying perturbed samples designed to cause misclassification. In order to avoid detec- tion, A needs to make the prediction margin robust to such perturbations. We do so using adversarial training: & popular regularisation method used to provide robustness against adversarial examples.
Sq
400
Sp-mean= Sy-mean=-0.88
300
Count
The original paper considered three adversary mod-
200
els: 1) Ap who has access to V's Sy; 2) Ag who
has query (i.e. black-box) access to fv, and corre- sponds to a model extraction attack; 3) AM who has white-box access to fv-
100
0
-1.0
-0.5
0.0
0.5
1.0
Ap: we evaluate adversarial training as a way to avoid detection in the Ap setting Ap steals V's Sy and trains their own model f.A. Also, fa has the same architecture and hyperparameters as Jv. but is adversarially trained
Confidence
Figure Confidence scores assigned embeddings by Oy obtained from fa- Ap is small enough to trigger FNs.
During adversarial training, each training sample (x, y) is replaced with an adversarial example that misclassified fA(x + 7) * y- There exist many techniques for crafting adversarial examples. We use projected gradient descent Madry et al. 2018 (PGD), and we set 7 = 10/255 (under l.0).
In Figure we visualise the difference in the distributions of scores assigned by gv to fA embeddings derived for Sy and Sg. We observe that the distributions Ho cannot. be rejected. Therefore, SA is marked as an independent model, false negative. In Table 3 we provide Aμ and the associated p-values for multiple runs.
not clearly separable and result in low Aμ, and hence
Note that adversarial training comes with an accuracy tradc-off. In our experiments, the accuracy of fa goes from 0.92 0.01 0.86 0.01 (CIFARIO), and 0.73 + 0.03 to 0.60 0.01 (CIFAR100). However, recent work shows that one can adversarially train models without incurring as much accuracy loss {Cui et al. 2021 Debenedetti al. 2022 Wang al. 2023). We study how the amount of noise affects the verification Also, discuss the resulting implications in Section
in Section 5.2
We the official implementation of DI, together with the architectures and training loops. Our changes limited adding adversarial training.

---

Published in Transactions on Machine Learning Research (06/2023)
Table 3: fA adversarially trained on Sy results in a false negative. Also, we report the accuracy of the models on the test set. We provide the mean and standard deviation computed across five runs. Verification done using k 10 private samples. FNs are are highlighted.
CIFARIO
CIFAR100
Model
Accuracy
Aμ
p-value
Accuracy
Ap
p-value
fv
0.92 0.01
1.59 + 0.04
21 +: 10-16
0.73 0.03
1.58 0.10
10-31 + 1001
fA
0.86 + 0.01
0.12 + 0.06
0.15 + 0.07
0.60 + 0.01
0.11 +11+000 0.02
0.43 + 0.18
fo 0.64 0.02 -0.99 + 0.12
0.46 + 0.04
0.51 0.01
10.77=0.11
0.67 + 0.15
Table 4: Verification of independent models trained on the same data distribution as fv: a) using more data to train gv; b) using bigger, four-layer gv trained with dropout. We provide the mean and standard deviation computed across five runs. Verification done using k = 10 private samples. FPss are highlighted.
Model
Accuracy
Ap
p-value
Model
Accuracy
Aμ
p-value
fv
0.87 + 0.03
1.34 + 0.08
10 31 + 10-30
fv
0.87 + 0.03
1.30 + 0.17
10-17 + 10-18 16
fz 0.87 + 0.03
0.83 + 0.13
10 11 + 10-10
fr
0.87 + 0.03
0.84 0.16
10- + 10-6
fo 0.64 + 0.02
0.08 + 0.01
0.15 + 0.08
fu 0.64 + 0.02
0.05 + 0.01
0.37 + 0.30
(a) FPs remain when is trained with more data.
(b) FPs remain when using a bigger gv-
We consider Ad in detail because triggering FNs against DI via adversarial training harder for Ap the models have the same architecture, and are trained using exactly the same data.
Ao: the same approach also applies in the Ag setting - Ae conducts model extraction attack with their attack dataset, and can train fA with adversarial training (without needing access to Sy).
Am: while this approach is not directly applicable to the AM adversary, an adversary with AM capabilities and sufficient data (not the same as Sv) can indeed mount the same attack as Ae by distilling the model themselves.
5 Towards Countermeasures
In Sections
3 and we showed that DI suffers from false positives and false negatives in certain settings. In this section, we explore whether potential countermeasures exist. First,, we analyse possible tweaks to gv that could help alleviate the FPs (Section 5.1p. Next, test if increasing the amount of noise used during the embedding generation avoids FNs despite adversarial training (Section 5.2
5.1 Avoiding FPs via Modified Distinguisher
Recall that DI does not suffer from any FPs in the original data split where both the fo and gy are trained on Sg. One possible explanation for the FPs that gy overfits to So, and does not generalise to Sy. To verify this hypothesis, we augment Sp with a portion of S₁ (of size (Sol), which then used train gv- For fz, the p-value still remains low and hence triggers an FP (Table 1a..
We then consider alternative hypothesis gv underfits and fails to predict the margins correctly. To examine this, we increase the size of gv. The original gy is two-layer network with tanh activation. We increase the number of layers to four. Although this does not eliminate the FPs, it does increase the average p-value (Table 4bb
In conclusion, neither of these approaches alleviates the FPs. The margins of different models trained on the same distribution (estimated by gv) are difficult to distinguish. Finding a robust approach avoid FPs in DI remains open problem.
10

---

Published in Transactions on Machine Learning Research (06/2023)
5.2 Avoiding FNs via Verification with More Noise
Table 5: Impact of added noise (maximum number of perturbation steps) during verification on DI success (baseline 50 steps). Using more noise does not. prevent FNs against SA but increases the standard deviation across all experiments, thus negatively impacting verification of fo. We provide the mean and standard deviation computed over five runs. Verification done using k = 10 private samples. FNs are highlighted.
CIFARIO
CIFAR100
Model
Accuracy
Steps
Au
p-value
Accuracy
Steps
Au
p-value
fv
0.92 1 0.01
50
1.59 1 0.04
10-21 1 10 16
0.73 + 0.03
50
1.58 1 0.10
10-31 1 1
0.09 + 0.04
0.09 0.07
25
0.12 0.02
0.39 0.22
50
0.12 + 0.06
0.15 + 0.07
50
0.11 : 0.02
0.43 + 0.18
fA
0.86 + 0.01
0.60 0.01
100
0.10 + 0.05
0.08 0.09
100
0.13 0.01
0.50 0.16
200
0.14 + 0.08
0.16 + 0.11
200
0.12 0.01
0.47 + 0.24
50
0.29
0.12
0.46 0.04
50
0.37 + 0.11
0.67 0.15
fo
0.64 + 0.02
0.51 0.51
100
-0119 + 0.16
0.37 + 0.12
100
0.43 + 0.03
0.72 + 0.09
V suspects that A might be using adversarial training to avoid detection, they can carry out the verification with more noise in the attempt to circumvent the effect of adversarially training S.A
In the experiments presented in Section 4 the average noise added during Blind Walk is 0.12 0.05 (under fee), and adversarial training is done with 7 = 10/255(= 0.039). In this experiment, we vary the number of maximum steps taken by V. and hence the maximum amount of added noise. We consider (25, (25,50, 200} steps (baseline 50 steps) which corresponds to {0.10 = 0.03, 0.12 + 0.05, 0.33 + 15, 0.38 + 23} noise added (under (x)) during the verification. needs to use more noise for the verification of any model, hence we also conduct the experiment for fo (for (50, 100) steps). The goal is to ensure that the increased amount of noise does not have any negative impact results. Using more steps does not improve the result against SA compared to the baseline: 1) for CIFARIO, the standard deviation of the p- p-value increases; 2) we do not observe any linear relationship between the noise and Ap or the associated p-value (neither for CIFARIO or CIFAR100).
the independent models. Table 5 summarizes our experiment
On the other hand, for CIFARIO, the confidence of the verification of So decreases. The standard deviations of Âμ and its associated p-value increase. Although the p-value remains sufficiently high, using more noise has a negative impact on the verification of fo- On the contrary, for CIFAR100, the confidence of the verification of fo increases: Ap is lower, p-value is higher, and standard deviations decrease.
In conclusion, increasing the amount of noise during Blind Walk does not allow V to circumvent A's adver- sarial training. Hence, DI remains susceptible to false negatives induced by adversarial training.
6 Discussion
Revealing private data. We have shown that DI requires revealing significantly more than 50 samples to avoid false positives the case of linear models (Figure D. Since the core assumption of DI is that Sy is private, revealing too much of Sy during the ownership verification constitutes a privacy threat. In neither of the settings described in Section 5 of the original DI paper the vietim cannol query the model sufficienlly without leaking the query data to the adversary. Additionally, it was shown that using more samples gives V more information about the prediction margin than using stronger embedding methods (Maini et al. 2021] Model owners that operate in sensitive domains such as healthcare or insurance industry need to comply with strict data protection laws, and hence need to minimise the disclosure.
One potential way to protect the privacy of the private samples used for DI ownership verification to use oblivious inference (Liu et al. 2017 Juvekar et al. 2018). This way V could query fA without revealing Sy. Despite recent advances in efficient oblivious inference (Samragh et al. 2021 Watson et al. 2022 Samardzic
11

---

Published in Transactions on Machine Learning Research (06/2023)
et al. 2021 2022), it requires all parties (including A!) to update their software stacks which may not always be real lsticc..
Viability of ownership verification using training data. We have demonstrated that DI suffers from FPs when faced with an independent model trained on the same distribution. While it is reasonable to assume that V's data private, the uniqueness of the distribution For example, two model builders may have data from the same distribution because they purchased their training data from a vendor that generates per-client synthetic data from the same distribution (e.g., regional financial data). In fact, two model builders working the same narrow domain and independently building models that are intended to represent the same phenomenon, may very well end up using data from the same distribution.
difficult to guarantee in practice.
There are other methods that attempt to detect stolen models based on the dataset used to train them (Sablayrolles et al., 2020 Pan et al.) (022). However, they rely on flaws in the model to establish the ownership (susceptibility to adversarial examples (Sablayrolles et al.] 2020j or membership inference attacks (Pan et al. 2022) Intuitively, given a perfect membership inference attack, fingerprinting scheme should be possible. However, recent work shows that for balanced dataset, only fraction of records is vulnerable to confident membership inference attack (Carlini et al.) 2022 Duddu et al. 2021) which in turn reduces the capabilities of a membership inference-based fingerprinting scheme. Therefore, any im- provements to generalisation or robustness (such as adversarial training or purification (Nie et al. 2022 of ML models reduce the surface for ownership verification schemes.
Nevertheless, leveraging advancements in membership inference might make DI more robust, e.g. regularising DI with many shadow models to avoid FPs, using MIAs that were shown to be more effective against adversarially trained models that overfit (Song et al. 2019a b Hayes 2020). However, it is worth noting that the orignal DI paper (Maini et al., 2021) differentiates DI from membership inference. DI relies on weak signals over many records unlike MIA that needs strong signal for individual records. Hence, relying on ideas from MIAs introduces additional considerations, e.g. MIA evasion.
False positives in other schemes. Our work raises the question whether other watermark- ing/ htg/fingeprrnnting schemes also vulnerable. Existing schemes for model ownership verification typically focus robustness against adversaries who want to evade detection. Systematically examining such schemes for false positives can be an interesting line of future work. An interesting question is whether a malicious model owner can successfully accuse independent models as stolen by systematically inducing false positives.
White-box theft. Our experiments in Section !| are limited to A that trains their own model - they either steal the data or conduct a model extraction attack. Tf A obtains an exact copy of the model, they might lack the data to fine-tune with adversarial training. Hence, our findings do not apply to the white-box setting. We leave the examination of other threat models out future work.
Black-box vs. white-box verification setting. Our evaluation is focused on the black-box DI setting, We do not consider the white-box DI setting which uses MinGD. While white-box DI is feasible in a scenario where V takes A (the holder of a suspect model) to court, requiring A to provide white-box access to the suspect model, prosecution is an expensive undertaking, Realistically is likely to first conduct black-bor DI docide whether the expense of prosocution is justified. Therefore, FPs in the black-box DI sctting can cause substantial monetary loss to V.
7 Conclusion
We analyzed Dataset Inference (DI) Maini et al. (2021) 1, a promising fingerprinting scheme, to show theo- retically and empirically that DI is prone to falsc positives in the case of independent models trained from distinct datasets drawn from the same distribution. This limits the applicability of DI only to settings where a model builder uses a dataset with a definitively unique distribution. Alleviating FPs in other settings may not be possible. Furthermore, DI might not be applicable in settings where privacy of the records must be respected, c.g. healthcare. We also showed that an attacker can use adversarial training to regularise the decision boundaries of stolen model to evade detection by DI at the cost of drop in accuracy (6 13pp).
12

---

Published in Transactions on Machine Learning Research (06/2023)
Such decrease is acceptable if the model can be further fine-tuned with additional data and recover the performance.
Nevertheless, DI is promising ML fingerprinting scheme. Model owners can use our results to make informed decisions as to whether DI is appropriate for their particular settings.
Acknowledgement
We would like to thank the authors of Datasct Inference (Maini et al. 2021) for their feedback work presented in this paper. This work was supported in part by Intel (in the context of the Private-Al Institute), National Natural Science Foundation of China (Grant No. 62002319, U20A20222), Hangzhou Leading Innovation and Entrepreneurship Team (TD2020003), and Zhejiang Key R&D Plan (Grant No. 2021C01116).
References
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into strength: Watermarking deep neural networks by backdooring. In 27th USENIX Security Symposium, pp. 1615 1631, 2018.
Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, and N. Asokan. Extraction of complex DNN models: Real threat or boogeyman?". In "Engineering Dependable and Secure Machine Learning Systems', pp. 42 57, 2020.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. IPGuard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In Proceedinggs of the 2021 ACM Asia Conference on Computer and Communications Security, ASIA CCS '21, pp. 14 44, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450382878. doi: 10.1145/ 10.1145/3433210022 3331333333110..33526 URL httre://oa..og//11111/4442222223
115.1.11111433311.33333333 3437526
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914, 2022. doi:
Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia. Learnable boundary guided adversarial training. In 2021.
Proceedings of the IEEE/CVF international conference computer vision, pp.
Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers. arXiv preprint XrX:::2209 07399, 2022.
Zizhuang Deng, Kai Chen, Guozhu Meng, Xiaodong Zhang, Ke Xu, and Yao Cheng. Understanding real- world threats to deep learning models in android apps. 2022. doi: 10.48550 00.48550/arXiv.2220 www.Xxv.2209099...
URL
https://arxiv.orrt www.axivv..../aaaoo org/abs////////oeooooor... httpa/aab/2220000 09577v1
Vasisht Duddu, Sebastian Sxyller, and N. Asokan. SHAPr: An efficient and versatile membership privacy risk metric for machine learning. CoRR, abs/2112.00223) 2021. URL htttss//aaxiit http//aaxvvv.../.aeea/ 02230)
Adam Dziodzic, Muhammad Ahmad Kalcem, Yu Shen Lu, and Nicolas Papernot. Increasing the cost of
model extraction with calibrated proof of work. In International Conference on Learning Representations, 2022. URL https://openrevier wtwp:rrwwwwiieeeeeooronoeuee aett/EorrmndddFAA//CCggg
Jamie Hayes. Provable trade-offs between private & robust machine learning.
arXiv preprint
arXiv:2006. w.Xiv:200604422,, 2020.
Hengrui Jia, Christopher A. Choquetto-Choo. Varun Chandrasekaran, and Nicolas Papernot. Entangled watermarks as defense against model extraction. 2021a.
13

---

Published in Transactions on Machine Learning Research (06/2023)
Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Chon Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Prooffof-learning: Definitions and practice. In 2021 IEFF Sym- posium on Security and Privacy (SP), pp. 1039- 1056. IEEE, 2021b.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N. Asokan. PRADA: protecting against DNN model stealing attacks. In IEEE European Symposium on Security & Privacy, pp. 1-16. IEEE, 2019.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. GAZELLE: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pp. 1651 1669, Baltimore, MD, August 2018. USENIX Association. ISBN 978-1-939133-04-5. URL https:
juvekar
Ram Shankar Siva Kumar, Jeffrey Snover, David O'Brien, Kendra Albert, and Salome Viljoen. Failure modes in machine learning. https://learn microsoft. cow/ http///eerrrtty/gineeerttee
masss-in-machnn--eariii 2019. Online; accessed 20 September 2022.
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against neural network model stealing attacks using deceptive perturbations. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 43 49, 2019. doi: 10.1109/SPMV 2019.00020.
Jian Liu, Mika Juuti, Yao Lu, and N. Asokan. Oblivious neural network predictions via minionn trans- formations. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Securily, CCS '17, pp. 619 631, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450349468. doi: 10.1145/ 10.1145/3133956 31133956
URL httt::://doi.orgg/// wtt..////......1111131111 33141333333333333366633 3134056
Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep neural network fingerprinting by conferrable adversarial examples. In International Conference on Learning Representations, 2021. URL https:
//openreview.netto wttp://oorormmiiiig////.....
Nils Lukas, Edward Jiang, Xinda Li, and Florian Kerschbaum. Sok: How robust is image classification deep neural network watermarking? In 2022 IEEE Symposium on Security and Privacy (SP), pp. 787 804, 2022. doi: /1/.////1111..2.9888833333
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial allacks. In International Conference on Learning Represenlations, 2018. URL htttss:/ wewe:/formmmddrrraaaee
Pralyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In International Conference on Learning Representations, 2021. URL https://opens:
view. wwt.:neee///nmnioeoeeeeoeuuu hdd=hdkkV2yy777
Mantas Mazeika, Bo Li, and David Forsyth. How to steer your adversary: Targeted and efficient model stealing defenses with gradient redirection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15241-15254 PMTR, 17-23 Jul 2022. URL
mlr, pwtps//v1//maaeeeeeeeeeeeeeeeeee html
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. PAC-bayesian approach to spectrally- normalized margin bounds for neural networks. In International Conference on Learning Representations, 2018. URL htttp:://ppenrevier.. aaaeeenrrviieenen/ffrrrrmmt net/forum?id-Skk. WfbC
Weili Nic, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML), 2022.
Tribhuvancsh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stcaling functionality of black-box models. In CVPR, pp. 4954 4963, 2019.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stealing attacks. In International Conference on Learning Representations, 2020. URL https: hwtpppprwrriv..tt..foorrlraaaaae
14

---

Published in Transactions on Machine Learning Research (06/2023)
Xudong Pan, Yifan Yan, Mi Zhang, and Min Yang. Metav: A meta-verifier approach to task-agnostic model fingerprinting. 2022. doi: 10.48550/ R00550/ARRXT ARXIV 20..2200.000 .07391. URL
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In ACM Symposium on Information, Computer and Communécations Security, pp. 506-519. ACM, 2017.
E. Quiring, D. Arp, and K. Rieck. Forgotten siblings: Unifying attacks on machine learning and digital watermarking. In IEFE European Symposium on Security & Privacy, pp. 502, 2018.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Herve Jegon. Radioactive data: tracing through training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8326-8335. PMLR, 13 18 Jul 2020. URL https::// htttss://prooceeinng..oo
www.ang.smm.....m... mlr wtt.ppeee/v////vepipbe pttpeeese/1111//aaaaallllee20000
Nikola Samardzic, Axel Feldmann, Aleksandar Krastev, Srinivas Devadas, Ronald Dreslinski, Christopher Peikert, and Daniel Sanchez. Fl: A fast and programmable accelerator for fully homomorphic encryption. In MICRO-54: 54th Annual IEEE/ACK International Symposium on Microarchütecture, MICRO '21, pp. 188-252, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385572. doi: 0411445/34466722448007777 URL.
11411146661///46672 3480077
Nikola Samardzic, Axel Feldmann, Aleksandar Krastev, Nathan Manohar, Nicholas Genise, Srinivas Devadas, Karim Eldefrawy, Chris Peikert, and Daniel Sanchez. Craterlake; A hardware accelerator for efficient unbounded computation on encrypted data. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA '22, pp. 173- 187, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450386104. doi:
URL https: http:::///oo..o/11...1111
496. 3527393
Mohammad Samragh, Siam Hussain, Xinqiao Zhang, Ke Huang, and Farinaz Koushanfar. On the application of binary neural networks in oblivious inference. In 2021 LEEE/CVE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4625-4634, 2021. doi: D01109/CCPRW53308220022 20000000...21.002222
Liwei Song, Reza Shokri, and Pratock Mittal. Membership inference attacks against adversarially robust deep learning models. In 2019 IEEE Securily and Pricacy Workshops (SPW), pp. 50 56, 2019a. doi: 10.1109/SPP.2219.0000 201..0021...
Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communi- cations Security, pp. 241-257, 2019b.
Sebastian Szyller and N. Asokan. Conflicting interactions among protection mechanisms for machine learning models. 2022. doi: 10.48550/ ARR.:///RRVVV.... ARX.V.22070019900000.22 22007019911 URI. htttps://arxiv.org//////// htt.bb//22770099999 01991
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N. Asokan. DAWN: Dynamic Adversarial Water- marking of Neural Networks, pp. 4417 4425. Association for Computing Machinery, New York, NY, USA, 2021. ISBN 9781450386517. URL https://doi. 1015/...100....11.....1..1... 1111111///444.....1 3475591
Florian Tramèr, Fan Zhang, Ani Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th USENIX Securily Symposium, pp. 601-618, 2016.
Joel A Tropp. Usee-fri tail bounds for sums of random matrices. Foundations of computational math ematics, 12(4):389 434, 2012.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. In ACM International Conference on Multimedia Retrieval, pp. 269-277. ACM, 2017.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. Xix preprint or Xrv::2300 046688 2023.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Jean-Luc Watson, Sameer Wagh, and Raluca Ada Popa. Piranha: A GPU platform for secure computation. In 31st USENIX Securily Symposium (USENIX Security 22), pp. 827 844, Boston, MA, August 2022. USENIX Association. ISBN 978-1-989133-31-1 URL https://www usenix
curtty2//preemmaaiiioo//tttttt
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyz- ing the connection overfitting. In 2018 IEEF s1st Computer Security Foundations Symposium ( (CSF). pp. 268 282, 2018. doi: 10.1109/CSIP /00///CF..0..00222
Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Marc Ph Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In ACM Symposium Information, Computer and Communications Security, pp. 159-172, 2018.
Rui Zhang, Jian Liu, Yuan Ding, Zhibo Wang, Qingbiao Wu, and Kui Ren. "adversarial examples" for proof-of-lerning. In 2022 TEEE Symposium on Security and Privacy (SP), pp. 1408-14 14222. IEFE, 2022.
Huadi Zheng, Qingqing Ye, Haibo Hu, Chengfang Fang, and Jie Shi. Protecting decision boundary of machine learning model with differentially privale perturbation. IEEE Transuctions on Dependable and Secure Computing, 19(3):2007-2000 2022. doi: 10.100//TDSC.2200003333
8 Related Work
Model extraction detection and prevention. Detection methods rely on the fact that many extraction attacks have querying patterns that are distinguishable from the benign ones (Juuti et al. 2019 Atli et al. 2020 Zheng el al. 2022 Quiring et al. 2018). All of these can be circumvented by the adversary who has access to natural data from the same domain as the victim model (Atli et al. 2020 Prevention techniques aim to slow down the attack by injecting the noise into the prediction, designed to corrupt the training of the stolen model (Orekondy al., 2020 Lee et al.) 2019 Mazeika al.) 2002), or by making all clients participate in consensus-based cryptographic protocols (Dziedzic al. 2022). Even though they increase the cost of the attack, they do not stop determined attacker from stcaling the model.
Ownership verification. There exist many watermarking schemes for neural networks (e.g. (Zhang et al. 2018 Uchida et al.) 2017 Add et al. 2018 that have the same goal as DI does. However, they mostly rely on embedding a backdoor into a model that serves as a watermark. Unfortunately, they do not protect against model extraction atlacks, and hence, several schemes were proposed that specifically larget model extraction (Szyller et al. 2021 Jia et al.) 2021a).
It was also shown that adversarial examples (Lukas et al. 2021 be used to fingerprint model by finding examples that transfer to models only derived from the original. One could also use adversarial examples to watermark datasets by including some in the training set (Sablayrolles et al., 2020). However, adversarial
training can be used to weaken both schemes (Lukas et al. 2021 Szyller & Asokan (022). On the other hand, if a model is sufficiently vulnerable to membership inference attacks, it can be used to fingerprint it Pan et al. 2022
Alternatively, one could prove ownership of the model using proof of the integrity of training (Proof-of- Learning) (Jia et al. 2021b). Proof-of-earring relies the fact that successive training checkpoints serve as proof of expended compute, and (probabilistic) correctness of the training procedure.
We believe that the following work is the closest to ours. Lukas et al. (2022) showed that all watermarking schemes are brittle. They designed various attacks that successfully remove watermarks, or prevent their embedding. However, they do not evaluate any fingerprinting schemes. ZZhang et al. (2022) showed that A can construct a spoof against Proof-of-Learning. iteratively creates the spoof by finding adversarial examples that match the successive training steps, hence resulting in false positive.
16

---

Published in Transactions on Machine Learning Research (06/2023)
9 Existence of False Positives in Dataset Inference
Calculating the prediction margin. We assuine that the model weights are initialized to zero. For each
sample in dataset S ~ D = {xx(1) y 1,... l,...,,.. y ~ { {-1, +13. The learning algorithm observes
all samples in S once and maximizes the loss function L y) = y f()). For the learning rate a = 1, the weights are updates as:
== oz 20
(10)
Recall that a (21, 22) RK+ the weights of the linear model are w1 mu and w2 Σn y0 (6)
x2
when the training is completed.
When writing out the linear classifier explicitly, we can easily calculate the prediction margin of each sample (z, y) in S,
(0) The expectations of the prediction margin for the points in training set S+ = {(2r, 1)((a, 1) E S) is.
y (xx) y (W111 + (2222) = (mu yu +
Σ y 20 x2) c Σ y 20 02
(11)
Es- [x)]) = yc + Est
22
x2l = yc + Estl Σ
y x0 (1)
x2 + yo?
22pl (4)
(12)
=c+ Do
E( (8) Do Note that in Equation 12 since 22 ~ N(0, Do?), then x2 ~
x2,
Consider a new datasct So ~ D, the expectations of the prediction margin for the points in Spt are,
m
(=f(u((=))] = yc + Est
x2
(4) x22 =c
(13)
Finally, we see that the difference of prediction margin training set S and test set Sg is
Es+ (s+lys - Est [u]( = Do".
(14)
DI's decision function. From the above analysis, we know that the statistical difference between the distribution of training and test data is D which is usually larger than 1 in numerical. DI utilizes this difference to predict if potential adversary's model stole their knowledge.
Since we know that Es. [x)) = e and Es Es(yf(=)) = Do². Let D) represent the dataset inference victim's 's decision function. It defined as,
1, if E(aa)esly f(x)) Eply f(x) 2
(f = 0, otherwise,
(15)
where X E [0, Do2) is some threshold that the decision function uses to maximise true positives and minimise false positives.
Proof for Lemma 3.2 For a linear model f trained on distribution D where x (I1, Ø2), σ1 = yu. ~
Ni0, o²) and ||u||| S vm is expected to achieve high accuracy on any sample (ar, y) sampled randomly from D which is independent of the training data set of f.
Proof. Given a linear model ff trained on dataset S D (1)(1) (a(f),yf (0)0)) 1, m), and a test sample (a, y) sampled randomly from D which is independent of S, the probability that (æ,, v) is correctly classified by
17

---

Published in Transactions on Machine Learning Research (06/2023)
can be represented as:
m
Plyf(a) 2 0] Plmma 2 + y Σ v1) (6)
(0)))/ 22 0]
(16)
Ply y (1))zz
(1) x2 2 -mu²)
Since 22 o2) are D-dimensional vectors, can use the central limit theorem to approximate the
term. Thus, the internal term can be approximated by variable ~ NO0, (DD"). Let Z ~ (O 1),
VmD mu² 7 2 -1] =1 $ mu²
Pluf(x) 2 0J S P
(17)
Vo
where the normal CDF.
We can also calculate the accuracy of the training set similarly. For training sample (a, v) sampled randomly from S,
Plyf(x) 2 0J Pmmu 2 + Σ v1)azz x2 2 0]
m-
Plmme + + + Σ
y(1) x2 2
(18)
m
Ply Σ y(i) x2 2 -mum
m
Ply _
v() x2 2 1].
Since y² 2 0 for any sample in S, have
m
2 + Σ 0 (1)a
22 2 y1
> y())ar) 22.
(19)
Then, ((a,))s 2 ((..))DD/S. This completes the proof.
Proof for Theorem 3.4 Let fw be d-layer feed-forward model trained on distribution D with parameters (Wik and the ReLU activation function. Assuming training dataset S ~ D, the model is given Js futug where us is random variable whose distribution may also depend on S.
Since the key to analyze the margin is the output of the model, we first introduce Lemma 9.1 that analyzes the perturbation bound the model trained on and D.
Lemma 9.1 ureaturbation Bound (Lemıma 2) in Neyshabur et al. (0188) For any B, d 0, let fw X R a d layee neural network with ReLU activations. Then for any w, and x E X, and any perturbation us = {U,) such that ||U,|/2 S [/Wal,,, the change in the output of the network can be bounded as follow,
1uwtus (x) fw) S eB
(20)
W.
Since our proof is also based on Lemma 9.1 is analogous the analysis of generalization bound in Neyshabur al. (2018) and essentially the same for the first part.
Proof. The proof involves two parts. In the first part, we show the maximum allowed perturbation of parameters as shown in Neyshabur al. (2018). Tn the second part, show that the margin difference of
18

---

Published in Transactions on Machine Learning Research (06/2023)
the models trained on Sy and S₁ is also bounded by the perturbation of parameters. Let B (TT [| W1ll)) 8
and consider network with normalized weights W; [[all W,. Due to the homogeneity of the ReLU, we have fa = fw. We can also verify that (I [[a]le - II ||all2 and ||.llr l| TW || Therefore, it is sufficient to prove the Theorem only for the normalized weights w, and hence w.l.o.g we assume that for any layer |W:ll2 = B.
Choose the distribution P of the prior of w to be N(00,21), and consider the random perturbation us V(00,21) = {U.) Since the prior cannot depend on the learned model w or its norm, we set based on the approximation B. For each value of f on a pre-determined grid, we compute the PAC-Bayes bound, establishing the generalization guarantee for all w for which | - 3 s J and ensuring that each relevant value of B is covered by some 3 on the grid. We then take union bound over all B on the grid. For now, we consider a fixed 3 and the w for which | - 31 s 3, and hence gd < Bdd I egd-1.
Since us ~ N(0,021), we get the following bound for the spectral norm of U Tropp 2012
P > t] S 2he 1/2h0?
(21)
Taking a union bound over the layers, we get that with probability at least /2' the spectral norm of perturbation of U; in each layer is bounded by oy 222hmm(2h)). Plugging this spectral norm bound into the maximum allowed perturbatin bation bound is:
Lemma 9.1 we have that with probability at least v2
||Ulll B
max fw+us( (x) [w)( <eB3"
Σ
e² ddB / <
(22)
where a 42dBB4 22hd2((2h Then we can compute the difference of expectation margins for fv which is trained Sy and fI which is trained on Sy. Firstly, compute the difference margins for any model is trained S ~ D and the target model fD. For any verified dataset S E D,
F(((Js, m)) - F(p(Jo,2)))
= =|E(s (x) - max mat (1: - mma
=I(E(fwtus (x)( (=f - (E(mar,+y fwtus (x)(s1) ((maa (x)[1)))
(23)
Smar wu (x) fw (x)())) + mar (maz) (x))] madj+yfu (7) f/ (xx) s
2marnex
So, for fy trained on Sy and fz trained on Sn, we have with probability at least 1 that the predictions margins are bounded by e:
[(p(fv, x)) E(p(fz, x))]
<|E(p(fv.x)) E(p(fr, x)) + E(p(fz,))) E(p(fo). x)
(24)
<e.
19


Reviews:
[{"review_id": "5RA3kmaoS1", "reviewer": "Reviewer_esN8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper tackles an important subject, namely data ownership / verification\n- The paper does a critical analysis of a recent technique, Dataset Inference\n\nWeaknesses\n- The proposed model is not really applicable (to the best of my knowledge no one uses GD on yf(x), and it is an important aspect because that loss tends to push margins even for correctly classified samples which makes verification easier), but this is more a problem of the original DI technique than of this paper.\n- Some parts of the paper tend to exaggerate the consequences of theoretical results, like \"Thus, we can expect that the margins of f on any dataset that follows a given distribution to be similar. This contradicts the intuition of DI.\" -> I am not sure this is quite true, given that epsilon is still relatively high. Also, the results from Neyshabur are about the difference between the model on its private set and the model on its test set, not between the model on its private set and *another model* on the same private set\n- Lemma 3.1 seems to be the crux of the argument of this paper against Dataset Inference, and it is very particular. It requires that the norm of u is lower than 1/sqrt(m) and sigma^2 is higher than 1/sqrt(m). These assumptions feel very specific. I dug a bit in the proof, and it boils down to this: the \"accuracy\" signal-to-noise ratio is mu^2 / (sqrt(mD) sigma^2). That ratio needs to be high enough to have good accuracy. If u<1/sqrt(m) and sigma^2>1/sqrt(m), the ratio becomes bounded by 1/sqrt(D) which is where the \"D<10\" comes from. Now the authors state vaguely that we \"need high D\", so I did the maths for that one too (same equations but different signal): to detect that a sample was seen at training time, the \"signal\" is Dsigma^2 and the noise variance is still mDsigma^4. So for detection the signal-to-noise is sqrt(D/m) for each tested sample, which would amount to sqrt(kD/m) is k samples are tested. I found this to be a much clearer picture than what is proposed by the authors, in particular because we can see that sigma plays not role in the detection but plays one for the accuracy of the model. Thus we can very well choose sigma to be low enough to ensure that the model is accurate, and tune D for detection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyses the process of Dataset Inference (DI), a technique proposed to evaluate whether a model was trained on a given dataset. The theoretical framework is the following: each data sample is the sum of a \"signal\" vector u and random Gaussian noise with per-coordinate std sigma and D dimensions. The authors show that DI is subject to false positives when the norm of u is small and sigma is high for a linear model, and propose to generalize to non-linear models. The paper also shows that the DI has a high number of  false negatives if trainers use adversarial learning to tweak the margin of the decision boundary. Experiments are shown on CIFAR-10 showing the practical aspects of the paper's points. Finally the authors consider some counter-measures, a modified distinguisher to avoid FP and using more noise to avoid FN. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I think the authors should properly lay down the hypothesis that come from the dataset inference paper and the ones that they are making. As explained above, I think a first principled approach shows that their hypothesis are purely adversarial (i.e. made for dataset inference to fail) but do not correspond to a realistic use case (where sigma and D can be tuned).", "broader_impact_concerns": "This paper is helping the ownership verification problem, which is an important ethical aspect of modern ML.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R5eBuWH2Gz", "reviewer": "Reviewer_vDV7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe results are intuitive and investigate a variety of real difficulties in running dataset inference.\n\nI find invoking generalization bounds for dataset inference to be interesting.\n\nThe empirical results all use the training code provided by the original dataset inference paper.\n\n\nWeaknesses:\n\nThe theoretical results are based on PAC-Bayes bounds, which generally are not very meaningful for modern networks. I don't think this is a big disadvantage of the work, as this is probably the best way to prove such a statement, and the intuition behind the theorem is anyways interesting. \n\nThe discussion of false negatives follows a rather strange threat model. They assume that the training data for the model is stolen rather than the model itself, so that adversarial training can be used to defend against dataset inference. Model extraction would not provide this threat model, so it's a bit unclear how an adversary would go about doing this. I think it would be interesting to consider if any defenses would be possible in the model stealing setting. This is also addressed a bit in section 5.2, but membership inference attacks have been shown on adversarially trained models [R1], I wonder if similar ideas can be used here. I don't think this is a huge issue, since in some sense dataset inference should work on any model trained from a specific dataset.\n\nThe only empirical results are provided on CIFAR-10, given that the original dataset inference paper used many more datasets. I'd like to see a more thorough evaluation.\n\nOverall, I believe this paper warrants acceptance to TMLR.\n\n[R1] - https://www.princeton.edu/~pmittal/publications/liwei-dls19.pdf\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\nThe paper considers failure models of dataset inference, an approach to ownership verification. They consider both false positives and false negatives. They show both theoretical results for false positives (relying on PAC Bayes theory), and empirically find both false negatives and false positives are possible. Finally, they discuss approaches to circumvent these raised issues.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am happy with accepting the paper as is. I'd suggest some more discussion to address the weaknesses I've mentioned in my review. \n\nExperiments on more datasets.\n\nConsidering \"adaptive\" dataset inference attacks specifically on adversarially trained models, perhaps taking advantage of some ideas from [R1].\n\nI would also suggest, perhaps as future work, for the authors to consider whether new ideas in membership inference (e.g. [R2] style techniques) could improve dataset inference as well.\n\n[R2] - https://arxiv.org/abs/2112.03570", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dpqh6Lg1eC", "reviewer": "Reviewer_dh3Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Although the paragraphs are separated weirdly sometimes, I find the writing to be smooth and explains things wells with good logical flows.\n2. The findings in this paper could be of interest to people who research in this area or people who are currently using DI.\n\nWeaknesses:\n1. Might be minor, but the way that authors cites Data Inference looks a little bit odd to me, instead of normal reference format such as Maini et al. (2021), the authors emphasize ICLR'21 or ICLR 2021 multiple times, which is rare to see in a paper.\n2. The related work sections seems under-worked. Or in this case, since the paper is mostly solely based on DI, I am not sure if the current related work section serves any purpose.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts more performance analysis on Dataset Inference (DI), a fingerprinting technique proposed to efficiently protect model ownership against both while-box and black-box attacks. The main contribution of the paper is to show that DI suffers from high false positives as well as false negatives. More specifically, the authors proves theoretically that the original assumption required by DI (high-dimensional noise) leads to low (linear) model accuracy and DI suffers from false-positives unless lots of private data is reveals during ownership verification. Extending the above results, the paper shows theoretically that such high false-positives also exist in non-linear models, and empirically shows both false-positives and false-negatives in a realistic black-box setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Some writings could use some revision, as noted in the weaknesses section above.\n2. Despite the authors mention in the abstract that they point out future works based on their analysis, I don't find much on this later in their discussion and conclusion. It would be good that the authors could propose some directions that could mitigate the problems in this analysis in a high-level (as I know detailed implementation is out of scope of this paper).", "broader_impact_concerns": "I don't have concerns on the ethical implications of the work that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers failure models of dataset inference (DI) proposed by Maini et al., 2021 in terms of false positives and false negatives. They show both theoretical results for false positives (relying on PAC Bayes theory), and empirically find both false negatives and false positives are possible.   From the feedback from the reviewers, two reviewers are satisfied with the responses from the authors and recommended acceptance.  

Considering one reviewer's initial concerns regarding the limited applicability of the setup and the specific hypothesis required to break DI, it is encouraging to note that the authors have addressed these issues in the revised version. The modifications made to lemma 3.1 and theorem 3.2 also appear to be satisfactory.

Moreover, the paper explores an important topic, shedding light on the challenges associated with dataset inference, and the investigation of generalization bounds for dataset inference is intriguing. Taking into account the explanations provided by the authors and the majority recommendation of the reviewers, I concur with their assessment and recommend accepting the paper.
Decision: Accept (poster)