Dataset Inference: Ownership Resolution in Machine Learning
Authors: Pratyush Maini
Mohammad Yaghini
Nicolas Papernot
Venue: ICLR 2021 Spotlight (2021)
Keywords: model ownership
model extraction
MLaaS

Abstract: With increasingly more data and computation involved in their training,  machine learning models constitute valuable intellectual property. This has spurred interest in model stealing, which is made more practical by advances in learning with partial, little, or no supervision. Existing defenses focus on inserting unique watermarks in a model's decision surface, but this is insufficient:  the watermarks are not sampled from the training distribution and thus are not always preserved during model stealing. In this paper, we make the key observation that knowledge contained in the stolen model's training set is what is common to all stolen copies. The adversary's goal, irrespective of the attack employed, is always to extract this knowledge or its by-products. This gives the original model's owner a strong advantage over the adversary: model owners have access to the original training data. We thus introduce $\textit{dataset inference}$, the process of identifying whether a suspected model copy has private knowledge from the original model's dataset, as a defense against model stealing. We develop an approach for dataset inference that combines statistical testing with the ability to estimate the distance of multiple data points to the decision boundary. Our experiments on CIFAR10, SVHN, CIFAR100 and ImageNet show that model owners can claim with confidence greater than 99% that their model (or dataset as a matter of fact) was stolen, despite only exposing 50 of the stolen model's training points. Dataset inference defends against state-of-the-art attacks even when the adversary is adaptive. Unlike prior work, it does not require retraining or overfitting the defended model.

Published as a conference paper at ICLR 2021
DATASET INFERENCE:
OWNERSHIP RESOLUTION IN MACHINE LEARNING
Pratyush Maini IIT Delhi
Mohammad Yaghini, Nicolas Papernot University of Toronto and Vector Institute
pratyush. monyushhmmiii@gmml..commm
(mohammad.yaghin,.
ABSTRACT
With increasingly more data and computation involved in their training, machine learning models constitute valuable intellectual property. This has spurred inter- est in model stealing, which is made more practical by advances learning with partial, little, or no supervision. Existing defenses focus on inserting unique wa- termarks in a model's decision surface, but this insufficient: the watermarks are not sampled from the training distribution and thus are not always preserved during model stealing. In this paper, we make the key observation that knowl- edge contained in the stolen model's training set is what is common to all stolen copies. The adversary's 's goal, irrespective of the attack employed, is always to extract this knowledge its by-products. This gives the original model's owner a strong advantage over the adversary: model owners have access to the origi- nal training data. We thus introduce dataset inference, the process of identifying whether a suspected model copy has private knowledge from the original model's 's dataset, as a defense against model stcaling. We develop an approach for dataset inference that combines statistical testing with the ability to estimate the distance of multiple data points to the decision boundary. Our experiments on CIFAR1O 10, SVHN, CIFAR 100 and ImageNet show that model owners can claim with confi- dence greater than 99% that their model (or dataset as a matter of fact) was stolen, despite only exposing 50 of the stolen model's 'ss training points. Dataset inference defends against state-of -tate-of-the-art attacks even when the adversary is adaptive. Un- like prior work, it does not require retraining overfitting the defended model
INTRODUCTION
Machine learning models have increasingly many parameters (Brown et al.
2020 Kolesnikov et al,,
2019), requiring larger datasets and significant investment of resources. For example, OpenAI's
development of GPT-3 is estimated to have cost over USD 4 million LLi 2020). Yet, models are often exposed to the public provide services such machine translation
2016) image
recognition (Wu et al. 2019). This gives adversaries an incentive to steal models via the exposed interfaces using model extraction. This threat raises question of ownership resolution: how can an owner prove that another suspect model stole their intellectual property? Specifically, we aim to determine whether a potentially stolen model was derived from an owner's 's model or dataset.
An adversary may derive and steal intellectual property from a victim in many ways. A prominent way is (1) model extraction (Tramèr et al. 2016). where the adversary exploits access to a model's (1.a) prediction vectors (e.g., through an API) reproduce a copy of the model at lower cost than what incurred in developing it. Perhaps less directly, (1.b) the adversary could also use the victim model labeling oracle to train their model on an initially unlabeled dataset obtained either from a public source or collected by the adversary. In a more extreme threat model, (2) the adversary could also get access to the dataset itself which was used train the victim model and train their own model by either (2.a) distilling the victim model, or (2.b) training from scratch altogether. Finally, adversaries may gain (3) complete access to the victim model, but not the dataset. This may happen when a victim wishes to open-source their work for academic purposes but disallows its
Work done while an intern at the University of Toronto and Vector Institute
Code and models for reproducing our work he found at htggghhh.....doetcraaaaaaaaaaeee

---

Published as a conference paper at ICLR 2021
commercialization, simply via insider-access. The adversary may (3.a) fine-tune over the victim model, or (3.b) use the victim for data-free distillation (Fang et al. 2019).
Preventing all forms of model stealing is impossible without decreasing model accuracy for legit- imate users: model extraction adversaries can obfuscate malicious queries as legitimate ones from the expected distribution. Most prior efforts thus focus watermarking models before deployment. Rather than preventing model stealing, they aim to detect theft by allowing the victim to claim own- ership by verifying that a suspect model responds with the expected outputs on watermarked inputs. This strategy not only requires re-training and decreases model accuracy, it can also be vulnerable to adaptive attacks that lessen the impact of watermarks on the decision surface during extraction. Thus, recent work that has managed to prevail (Yang et al. (2019) despite distillation (Hinton et al.) 2013) or extraction (Jia et al., 2020), has suffered a trade-off in model performance.
In our work, we make the key observation that all stolen models necessarily contain direct or indirect information from the victim model's training set. This holds regardless of how the adversary gained access to the stolen model. This leads us to propose a fundamentally different defense strategy: we identify stolen models because they possess knowledge contained in the private training set of the victim. Indeed, successful model extraction attack will distill the victim's 's knowledge of its training data into the stolen copy. Hence, we propose to identify stolen copies by showing that they were trained (at least partially and indirectly) on the same dataset the victim.
We call this process dataset inference (DI) In particular, we find that stolen models are more con- fident about points in the vietim model's training set than a random point drawn from the task distribution. The more an adversary interacts with the victim model to steal it, the easier it will be to claim ownership by distinguishing the stolen model's behavior on the victim model's training set. We distinguish a model's behavior on its training data from other subsets of data by measuring the 'prediction certainty' of any data point: the margin of a given data point to neighbouring classes.
At its core, DI builds on the premise of input memorization, albeit weak. One might think that DI succeeds only for models trained on small datasets when overfitting is likely. Surprisingly, in prac- tice, we find that even models trained on ImageNet end up memorizing training data in some form.
Among related work discussed in 12, distinguishing a classifier's behavior on examples from its train and test sets is closest to membership inference (Shokri et al.) 2017). Membership inference (MI) is an attack predicting whether individual examples were used to train a model or not. Dataser inference flips this situation and exploits information asymmetry: the potential victim of model theft now the one testing for membership and naturally has access to the training data. Whereas MI typically requires large train-test gap because such setting allows greater distinction between individual points in and out the training set (Yeom et al. 2018 Choo et al. (2220), dataset inference succeeds even when the defender has slightly better than random chance of guessing membership correctly; because the victim aggregates the result of DI over multiple points from the training set.
In summary, contributions are:
We introduce dataset inference as a general framework for ownership resolution in machine learning. Our key observation is that knowledge of the training set leads to information asymmetry which advantages legitimate model owners when resolving ownership.
We theoretically show on linear model that the success of MI decreases with the size of the training set (as overfitting decreases), whereas DI is independent of the same. Despite the failure of MI on a binary classification task, DI still succeeds with high probability.
We propose two different methods to characterize training vs. test behavior: targeted adver- sarial attacks in the white-box setting, and novel 'Blind Walk' method for the black-box label-only setting. We then create a concise embedding of each data point that fed to a confidence regressor to distinguish between points inside and outside a model's training set. Hypothesis testing then returns the final ownership claim.
Unlike prior efforts, our method not only helps defend ML services against model extrac- tion attacks, but also in extreme scenarios such as complete theft of the victim's model or training data. In $ 7 we also introduce and evaluate our approach against adaptive attacks.
We evaluate our method on the CIFARIO. SVHN, CIFAR100 and ImageNet datasets and obtain greater than 99% confidence in detecting model or data theft via the threat models studied in this work, by exposing as low as 50 random samples from our private dataset.
2

---

Published as a conference paper at ICLR 2021
We remark that dataset inference applies beyond intellectual property issues. For example, [Song & Shmatikov (2019) showed that models trained for gender classification also learn features predictive of ethnicity. This raises ethical concerns, and dataset inference could assess whether sensitive dataset was used by model developer for different purposes than stated at data collection time.
2 RELATED WORK
Model Extraction. Model extraction (Tramèr et al. 2016 Jagielski et al. 2020) Truong et al, 2021) is the process where adversary tries steal copy of machine learning model, that may have been remotely deployed (such as over a prediction API). Depending on the level of access pro- vided by the prediction APIs, model extraction may be performed by only using the labels (Chan-
drasekaran et al., 2019 Correia-Silva et al. 2018) or the entire prediction logits of the deployed service (Orekondy al. 2018). Model extraction has seen cycle of attacks and defenses. Once an adversary has knowledge of the defense strategy adopted by the victim, they adaptively modify the attack cireumvent that defense (see watermarking). Model extraction can also be recon- naissance step used to prepare for further attacks, e.g., finding adversarial examples (Papernot et al.) 2017 Shumailov et al. 2020).
Watermarking, Since Uchida et al. (017) embedded watermarks into neural networks and Adi et al. (2018) used them as signatures to claim possession, watermarks have been widely adopted as way to resolve ownership claims. The underlying idea is to manipulate the model learn information other than that from the true data distribution, and use this knowledge for verification afterwards. This strategy not only requires new training procedures and decreases the model's accu- racy (Jia et al., 2020), but is also vulnerable to adaptive attacks that lessen the impact of watermarks on the model's decision surface during extraction (Liu al. 2018 Chen al. 2019 Wang al., 2019 Shafieinejad et al. 2019).
Membership Inference. Shokri al. (2017) train number of shadow classifiers on confidence scores produced by the target model with labels indicating whether samples came from the training or testing set. MI attacks are shown work in white- (Leino Fredrikson 2020 Sablayrolles al., 2019) as well as black-box scenarios against various target models including generative models (Hayes et al., 2019). Yeom al. (2018) explore overfitting as the root cause of MI vulnerability. Choo et al. (2020) show that MI can succeed even in scenarios when the victim only provides labels.
Out of Distribution Detection. Liang et al. (2017) and Lee et al. (2018) measure model perfor- mance modifying an input to find if a sample is in or out-of-distribution The premise is that in-distribution samples are easier to manipulate, whereas out-of-distribution samples require more work. In contrast, work solves much more challenging problem: the dataset distribution may be the same, but can we still identify which the datasets was used for training?
3 THREAT MODEL AND DEFINITION OF DATASET INFERENCE
Consider a victim V who trains a model fy on their private data Sy S Ky, where Ky represents the private knowledge of V. While Ky is abstract concept that can not be concretely defined, the private dataset Sy represents a definite part of the victim's 's knowledge that can be formalized. An adversary A. may gain access to a subset of Ky and use it to train its model fA.- V suspects theft, and would like to prove that f.. is indeed a copy of fy. Hence, V employs dataset inference on SA. determine if subset of their private knowledge K.C. Ky was used to train JA.. We formally define the victim and their dataset inference experiment below.
be victim with private access to Sy C Ky,, where F represents the set of all classifiers trained on samples from a data distribution 'D. Given classifier f. V can reveal at most m samples from Sy to either conclusively prove that a subset of their private knowledge K. C Ky has been used in the training of f with Type-I error (FPR) < a, or return inconclusive result 0.
Definition 1 (Dataset Inferring Vietim V(J, a. m)) Let : Fx
Definition 2 (Dataset Inference Experiment EI m. a. Sy. D)) Let F. be in Definition I and assume Fy to be the set of all classifiers trained on the victim's 's private dataset Sy ~ D, and natural number. The dataset inference experiment follows:
3

---

Published as a conference paper at ICLR 2021
1. Choose b {0, 1} uniformly at random.
2. fA. =f if else fA. = -F
3. a, Sv, 11 if am) m)
and
(
otherwise
THEORETICAL MOTIVATION
Dataset Inference (DI) aims to leverage the disparity in the response of an ML model to inputs that it saw during training time, versus those that did not. We call this response 'prediction margin'. we introduce our theoretical framework. In $ [.11 we quantify the difference in the expected response of a model to any point in the training and test set. Finally,
In & 4
4.2 we describe how DI succeeds with high probability in this setting, while membership inference (MI) fails.
Setup. Consider a data distribution D, such that any input-label pair (x, y) can be described as:
y ~ [1,,
- y--yuu u R.K 1e~N(0,o01)) R ER
where x (x1.X2) RK+D and u E RK is a fixed vector. Observe that the last D dimensions of x represent Gaussian noise (with var. σo) having no correlation to the correct label. However, the first K dimensions are sufficient to separate inputs from classes {-1, +1} (Nagarajan & Kolter 2019).
S ~ s.t. [S] = m represents the private training set of a model with m distinct training examples.
Architecture. We consider the scenario of classifying the input distribution using linear classifier, f, with weights w = -W11, W2), such that for any input x: fx) - W1 X1 + W2 X2. And the final classification decision is sgn()(x)). While we only discuss the case of a linear network in this analysis, the success of DI only increases with the number of parameters in machine learning model. as is the case for MI (Yeom et al., 2018). This, in effect makes the following analysis stronger result to prove. Prior works have also argued how over-parametrized deep learning networks memorize training points (Zhang et al., [016 Feldmann 2019). At its core, builds on the premise of (weak) input memorization. Results on DNNs are discussed in $
4.1 PREDICTION MARGIN
In our work, we use 'prediction margin' to imply the confidence of machine learning model of its prediction. In other words, we try to capture the robustness of a model's prediction under uncertainty. which is equivalent viewing the local landscape of machine learning model. For the purpose of the theoretical analysis, it is convenient to define it as the margin of a data point from the decision boundary (y fx)). As we scale our method to deep networks in the empirical evaluation, we will describe alternative methods of measuring the 'prediction margin' in multi-class settings.
Theorem (Train-Test Margin) Given a linear classifier f trained to classify inputs (x, E S (training set), the difference in the expected prediction margin for samples in S and 'D is given by E(x,)) ly f(x) (x by f(x)] = Do² where o² is the Gaussian noise variance as in d7).
first calculates the weights of the learned classifier f by assuming that is trained using gradient descent with a fixed learning rate, and viewing all training points exactly once. We then analyze the expected margin for data points included in training or not.
The proof (Appendix A.2
4.2 DATASET INFERENCE v/s MEMBERSHIP INFERENCE
We now show how MI fails to distinguish between train and test samples in the same setting. This happens because an adversary has make decision about the presence of a given data point in the training set by querying single point. However, DI succeeds with high probability in the same setting because it aggregates signal over multiple data points. We note that the statistical differences between the 'prediction margin' of training and test data points in 4.1 are only known when we calculate an expectation over multiple samples.
4

---

Published as a conference paper at ICLR 2021
Failure of Membership Inference. Consider membership in- ferring adversary M that has no knowledge of the victim's ' train- ing data S. but has domain knowledge such the publicly avail- able data distribution D. Define M(x, f) as the adversary's 's de- cision function predict whether belongs to S. Let R. repre- sent distribution that uniformly at random samples from either S6 1) (66 0). Then, M makes membership decision about (x, b) ~ R. & denotes the Gaussian CDF.
dc
dg
dp
(a) If in training
Theorem (Failure of MI) Given a linear classifier f trained on S, the probability that an adversary M correctly predicts the mem- bership of inputs randomly belonging to the training or test set.
de
dg
Px-R [M(x, f) = b] 11-10
and decreases with
|S| m. Moreover, 1immm+ooo PX-R M = b] 0.5.
(b) If x is not in training set
The theorem suggests that the success of MI when querying a single data point is extremely low. Ass increases, the adversary do no better than coin flip. This means that the success is directly proportional overfitting (we present the proof in Appendix (A.3)
Figure The effect of includ- ing (x, 'A') in the train set. If x is in the train set, the classi- fier will learn to maximize the decision boundary's s distance to \ {'A'). If x is in the test set, it has no direct impact on the learned landscape.
Success of Dataset Inference. Take V to be a dataset inferring victim (Definition D Let (hv(S, S; be V's decision function for ownership resolution. In the next theorem, we show that the success of DI in practice is high and independent of the training set size. (Proof in Appendix A.4)
Theorem (Success of DI) Choose b 1) uniformly random. Given adversary's 's linear classifier f trained on S' ~ DD s.t. |S" [S] if b = O, and on S otherwise. The probability correctly decides if an adversary stole its knowledge IP [ S;D) = b] =
.D
2v
Moreover, P (1)((,
11
Example. Assume a dataset of training size 50K and input dimensions K = 100, D 900 (i.e., 100 strongly correlated features which roughly similar the MNIST dataset) We have ((x.g)>s [w S; = - 10-20 1.0 while ((x,))= [M(x, f) = 6] 0.526. Therefore, problem setting where membership inference succeeds only by slightly above random chance, dataset inference succeeds nearly every time.
DATASET INFERENCE
Dataset Inference is the process of determining whether a victim's private knowledge has been directly or indirectly incorporated in model trained by an adversary. Our key intuition is that classifiers generally try to maximize the distance training examples from the model's decision boundaries. This means that any model which has stolen the victim 's private knowledge should also position data similar to victim's private training data far from its own decision boundaries. (See Figure M When victim suspects knowledge stolen from their model, they may measure how the adversary's s model responds to their own training data to substantiate their ownership claim.
5.1 EMBEDDING GENERATION
For model f and data point x, we aim to extract feature embedding for x that characterizes its "prediction margin' (or distance from the decision boundaries) w.r.t. f. The victim V extracts these embeddings for points (x.3) D and labels them inside (b 1) outside (b 0) of their private dataset Sy We introduce two methods for generating embeddings based on the level of access the victim may have the adversary's model.
RReaall that for our discussion linear networks in $ used simple metric to compute the 'prediction margin' of given data point as (y f(x)). However, the same does apply deep networks.
5

---

Published as a conference paper at ICLR 2021
White-Box Setting: MinGD White-box bedding generation is used when V and A, re- solve the claim for ownership in the presence of a neutral arbitrator, such as a court. Indeed, Ku- mar al. (2020) highlight that such attacks po- lentially fall under Computer Fraud and Abuse Act in the USA and are prosecutable for 're- verse engineering' the model's 'source code'. Both parties provide access to their models, and then the 'prediction margin' is measured for suspected adversary's 's model on the victim's train and test data points. For any data point (x, y) we evaluate its minimum distance A to the neighbouring target classes by performing gradient descent optimization of the following objective (Szegedy et al. 2013): min, x+ 5) s.t. f(x + d) = t. The distance metric A(x(), x())) refers to the lp distance between points (0 and x(5) for E {1, 2, j. and the target label. The distance Á each tar- get class is a feature in the embedding vector analyzed by the ownership tester from (5.2
fa.
Embedding
Private Samples
Generator
Sv
Public Samples
(ggus
Opuo
b
Ownership Tester
S (Public)
Stolen Inconclusive
Figure 2: Training (dotted) the confidence regres- sor with embeddings of public and private data, and victim's 's model fy; Dataset Inference (solid) using m private samples and adversary model fA.
Black-Boy Box Setting: Blind Walk. V may want to perform DI on a publicly deployed model f that only allows label query access. This makes them incapable of computing gradients required for MinGD. Moreover, querying f would be costly for V. Therefore, we introduce a new method called Blind Walk which estimates the 'prediction margin' any given data point through its robustness to random noise rather than gradient search. We sample a random initial direction 5. Starting from an input (x. y), we take l E N steps in the same direction until f(x + kō) = t; t * y. Then, proxy for the 'prediction margin' of the model. Thus, the approach only
Axx, x + k6) is used requires label access to f. We repeat the search over multiple random initial directions to increase the information about the point's robustness, and use each of these distance values as features in the generated embedding. In practice, we find Blind Walk to perform better than MinGI) with the We discuss further details justifying these observations in Appendix C
ownership tester from 5.2
5.2 OWNERSHIP TESTER
is important for the victim to resolve ownership claims in as few queries as possible, since each query involves the victim revealing part of their private dataset Sy. Since claiming ownership would likely lead to legal action, is paramount that the victim minimizes their false positive rate. We thus test ownership in two phases: regression model first infers whether the potentially stolen model's s predictions on individual examples contain the victim's 's private knowledge, this is then followed by hypothesis test which aggregates these results to decide dataset inference. This is another key difference with membership inference efforts: rather than always predicting that point is from the 'train' 'test' data, we claim ownership of model only when we have sufficient confidence. This is done through statistical hypothesis testing, which takes the false positive rate as a pyreee-aramettr, and produces either conclusive positive results with an error of at most a, or an 'inconclusive' result.
Confidence Regressor. As defined in 15.1 we extract distance embeddings w.r.t fy for data points in both V's private data Sy and unseen publicly available data. Using the embeddings and the ground truth membership labels, we train a regression model gv. The goal of gy is to predict (proxy) measure of confidence that sample contains fv's private information. For our hypothesis testing, we require that gv produce smaller values for samples from Sy. Complete access to the dataset Sy allows to train gy accurately, as illustrated via dotted arrows Figure 2
Hypothesis Testing, This is the step where dataset inference claims are made (solid lines in Fig- ure Using the confidence scores produced by gv and the membership labels, we create equal- sized sample vectors and cy from private training and public data, respectively. We test the null hypothesis Hn : < jy where μ = and byy = Cy are mean confidence scores. The test would either reject Ho and conclusively rule that SA. is 'stolen", or give an inconclusive result.

---

Published as a conference paper at ICLR 2021
6 EXPERIMENTAL SETUP AND IMPLEMENTATION OF DATASET INFERENCE
Unlike prior work on membership inference, which evaluates over victim models trained to overfit on small subsets of the original dataset, we train all of our victim models on large common benchmarks.
Datasets. We perform our experiments on the CIFARIO, CIFARI00, SVHN and ImageNet datasets. These remain popular image classification benchmarks, further description about which can be found in Appendix E.T All details about experiments on SVHN and ImageNet are in Appendix E.2
Model Architecture. The victim model is a WideResNet (Zagoruyko & Komodakis 2016) with depth 28 and widening factor of 10 (WRN-28-- (-----881)) 10) for both CIFARTO and CIFAR- and is trained with a dropout rate of 0.3 (Srivastava et al. 2014). For the model stealing attacks described in $ 6.1
we use smaller architectures such as WRN-16-1 on CIFARIO and WRN-16- 10 on CIFARI00.
6.1 MODEL STEALING ATTACKS
We consider the strongest model stealing attacks in the literature, and introduce new attacks targeting dataset inference to perform an adaptive evaluation of our defense. The adversary A. can gain different levels of access to v private knowledge:
(1) Aq has query access to fv. We consider model extraction (Tramrr et al. 2016) based adversaries which may (1.a) have access to the model's ' prediction vectors (via an API). Aq queries fy on a non- task specific dataset, and minimizes the KL divergence with its predictions. (1.b) Alternately. to further distance its predictions from the victim, the adversary may only use the most confident label from these queries (as pseuls-labels) to train. (2) AM has access to the victim's 's model fv. This may happen when V wishes open-source their work for academic purposes but does not allow its commercialization, or via insider-ace access.. (2.a) Aas may fine-tune over fv. or (2.b) use fy for data- free distillation (Fang et al. 2019) in a zero-shot learning framework that only utilizes synthetic and non-semantic queries (3) An has access to the complete private dataset, Sy of the victim. They may train their own model either (3.a) by distilling fv (over query access), or (3.b) training from scratch using different learning schemes or architectures. (For further details see Appendix B).
Finally, we also perform DI against an independent and honest machine learning model I that was trained on its own private dataset. This model is used as a control, to ensure that we do not claim ownership of models that were not trained by stealing knowledge from our victim model.
Training the threat models. For model extraction and fine-tuning attacks on CIFARIO and CI- FARTO0, we use subset of 500,000 unlaheled Tinylmages that are closest to CIFAR1O, 10, as created by CCarmon et al.] (2019). For SVHN, we use the "extra" training data released by the authors. We train the student model for 20 epochs for model extraction methods and 5 epochs for fine-tuning. For zero-shot learning, we use data-free adversarial distillation method (Fang et al. 2019) and train the student model for 200 epochs. In case of distillation and modified architecture, we have access to the original training data of the victim. We train both models for 100 epochs on the full training set.
In all the training methods, we use a fixed learning rate strategy with SGD optimizer and decay the learning rate by factor of 0.2 at the end of the 0.3x, 0,6x, and 0.8x the total number of epochs.
6.2 IMPLEMENTATION DETAILS FOR DATASET INFERENCE
Embedding generation. For the white-box method (MinGD), we perform the attack against each target class while optimizing the ε f2, too norms. Hence, we obtain an embedding of size 30 (classes distance measures). In the case of CIFAR 100, we only attack the 10 most confident target classes, as indicated by the prediction vector f(x). For the black-box method (Blind Walk), we sample 10 times from uniform, Gaussian, and laplace distributions to perturb the input. Once again, we obtain an embedding vector of size 30. More details are deferred to Appendix C
Training the confidence regressor. We train two-layer linear network (with tanh activation) gy for the task of providing confidence about a given data point's membership in 'private' and 'public' data. The regressor's loss function is L(x, 3) -b- gy where the label for a point in the (public) training set of the respective model, and -1 if it came from victim's 's private set.
"This is the first work consider data-free distillation as stealing attack.
7

---

Published as a conference paper at ICLR 2021
Hypothesis Tests. We query models with equal number of samples from public and private datasets, create embeddings and calculate confidence score vectors с and cy. respectively. We form two sample T-test on the distribution of e and Cy and calculate the p-value for the one-sided hypothesis Ho 14 < jyy against Halt (v(x) when Sy, and maximizes it otherwise. Therefore, vector that contains samples from Sy produces lower confidence scores, and decreases the test's p-value. If the p-value is below a predefined significance level a, Ho rejected, and the model under test is marked 'stolen'.
jay. From L(x, y). it follows that gv learns to minimize
In the following results, we repeat all experimental statistical tests for 100 times with randomly sampled data with replacement. To control for multiple testing, and account for the unknown depen- dence of the p-values thus generated, we aggregate these values using the harmonic mean (Wilson. 2018). To produce bootstrap 99-pereentile confidence intervals, we repeat the experiment 40 times.
7 RESULTS
Table [ shows p-values and the effect size, Aμ = py, which captures the average confidence of our hypothesis test in claiming that a model was stolen. We test our approach against 6 different attackers and in two different settings (Black- and White-box). In addition, Table [ also reports 'Source' where the victim's complete model fy has been stolen, and 'Independent" the control model trained on a separate dataset. Understandably, we typically observe the largest and smallest effect sizes for these two baselines, which serve as bounds to interpret our evaluation of attacks.
Our evaluation shows that DI is robust to both the strongest model stealing techniques, but also an adaptive attack we propose based on zero-shot learning. DI can claim a model was stolen with at least 95% confidence for most threat models with only 10 samples. Hence, the defense exploits an inherent property of model training. Among the six attacks we considered, we observe that our model consistently flags
models as stolen. This departs from prior defenses against model extraction: c.g., watermarks often lack robustness to fine-tuning. Here, DI is unaffected because fine- tuning does not remove knowledge from all private data used train the stolen model. The label- query and zero-shot attacks challenge DI the most. This is expected because zero-shot learning uses only synthetic data points for querying; and in case of logit-query, Aq is merely using V to label their dataset, which leaks much less private knowledge than distillation-basee model extraction. In practice, their higher query complexity makes both these attacks the most (financially) expensive to mount. We present concurring results on SVHN and ImageNet in Appendix E.2
DI requires few private points. In Figure 3 we show the number of private points the victim has to reveal (from its training set) to achieve a particular p-value when claiming model ownership is low: 40, and often as few as 20, samples to achieve a false positive rate (FPR) a of at most 1%.
Query efficiency. For the black-box scenario where the victim wants to assess the ownership of model served through an API, DI is a query efficient approach that comes at low cost for the victim.
CIFARIO
CIFAR100
Model
MinGD
Blind Walk
MinGD
Blind Walk
Stealing Allack
Av
p-value
Ap
p-value
p-valuc
Aμ
p-valuc
V Source
0.838
10
1.823
10 10-42
1.219
10 16
1.967
10 44
Distillation
0,586
10-4
0.778
10-5
0.362
10-2
1.098
10-5
Ap
Diff. Architecture
0.645
10-4
1.400
10-10
1.016
10-6
1.471
10-14
Zero-Shot Learning
0.371
10-2
0.406
10-2
0.466
10-2
0.405
10-2
Au
Fine-tuning
0.832
10
1.839
10-27
1.047
10-7
1,423
10-10 10
Aq
Label-query 0.475
10-3
1.006
10-1
0.270
10-2
0.107
10-1
Logit-query
0.563
10
1.048
10-4
0.385
10-2
0.184
10-1
I
Independent 0.103
-0.397
0.675
-0.242
0.545
-1.793
Table 1: Ownership Tester's ' effect size (higher is better) and p-value (lower is better) using = 10 samples on multiple threat models (see $ 6.11). The highest and lowest effect sizes among the model stealing attacks (AD, AM; are marked red and blue respectively.
8

---

Published as a conference paper at ICLR 2021
p-value CIFARIO- White Bex
CIFARIO - Black Bex
CIFAR (00- White Box
CIFARI00 Black Box
0.05
n1
Distillation
Label-Query
11.15
Logit Query
Zero Shot Learning
0.10
10
Fine: Tuning
Diff. Architecture
0,05
0.00
20
Figure 3: p-value against number of revealed samples (m). Significance levels (FPR) a = 0.01 and 0.05 (dotted lines) have been drawn. Under most attack scenarios, the victim V can dispute the adversary's s ownership of fA. (with FPR at most 1%) by revealing fewer than 50 private samples.
For 100 data points, DI can be performed less than 30,000 queries to the API. More efficient embedding generation optimizations can significantly improve this further. (See Appendix D
White-box access not essential to DI. While access to gradient information can help in partic- ular scenarios (such as, logit-query in CIFAR CIFAR100) which reduces from 40 to 20 samples, for fine-tuned adversary models, or those that are trained against a different architecture to evade detec- tion, our proposed black-box solution (Blind Walk) performs surprisingly better than its White-box counterpart. We conjecture that the Blind Walk's s advantage stems from a combination of factors: (a) gradient-based approaches sensitive numerical instabilities, (b) the approach is stochastic and aims to find the expected prediction margin rather than the worst-case (it searches for any incorrect neighboring class randomly chosen direction rather than focusing on the distance possible target classes). Hence, our proposed Blind Walk inference procedure is highly efficient.
DI does not require overfitting or retraining. Unlike past defenses (watermarks) and attacks (MI) which we discussed previously, DI uniquely applies as a post-hoc solution to any publicly deployed model. irrespective of whether it 'overfit' on its training set. This means that model owners in the real-world can perform DI immediately, to protect models that they have already deployed.
DISCUSSION AND CONCLUSION
While adversarial MI often consists of cycle of attacks and defenses, turn this game its head. Dataset inference leverages knowledge a defender has of their training set to identify models that an adversary created by either directly accessing this training set without authorization or indirectly distilling knowledge from one of the models released by the defender. With dataset inference, model developers resolve model ownership conflicts without making changes their existing models.
Interestingly. the ability to claim ownership through dataset inference gracefully degrades as the adversary spends increasingly more resources to train the stolen model. For instance, extracts a copy and later fine-tunes it with a different dataset to conceal the model, it will make the model more different and dataset inference will be less likely succeed. But this expected and desired: this means the adversary faced a higher cost to obfuscate this stolen copy. In itself it is not an easy task, because of accuracy degradation and catastrophic forgetting.
adversary
Finally, it remains promising direction for future work to study the confluence of DI with privacy- preserving models trained using --differential privacy (DP). [eino Fredrikson (020)) have shown that while DP can help against membership inference (MI) attacks, comes steep cost in accu- racy. We hypothesize that since DI amplifies the membership signal using multiple private samples, it follows that the values required to make DI ineffective would be even lower than it is for MI. Therefore, values that can make the model private, likely do not interfere with dataset inference.
Acknowledgments. We thank the reviewers for their feedback. We would also like to thank members of CleverHans Lab, especially Ilia Shumailov and Varun Chandrasekaran. This work was supported by Canada CIFAR Al Chair, NSERC, Microsoft, and sponsors of the Vector Institute.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weak- ness into a strength: Watermarking deep neural networks by backdooring. In 27th {USENIX) Security Symposium ( ({USENIX) Security 18), pp. 1615-1631, 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder. Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John Duchi. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan. Ex- ploring connections between active learning and model extraction, 2019.
Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, and Dawn Song. Refit: unified watermark removal framework for deep learning systems with limited data. arXiv preprint arXiv:1911.07205. 2019.
Christopher A. Choquette Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks, 2020.
Jacson Rodrigues Corveia-Silva, Rodrigo F. Berriel, Claudine Badue, Alberto F. de Souza, and Thi- ago Oliveira-Santee Santos. Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data. 2018 International Joint Conference on Neural Networks (IJCNN), Jul 2018. doi: 10. 109/ijenn. 2018 8488999.. http: /xx doi. www.o.oooggo.... 1100..1111991 1109/IJCNN IJCNN. 2018. 8489592
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Chen, and Mingli Song. Data-free adversarial distillation. arXiv preprint 1rXiv: 2019.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail, 2019.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership infer- ence attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019:
133-152, 01 2019. doi: 10.2478/popets- 20eetss-201.-0 2019-0008.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks, 2020.
Hengrui Jia, Christopher Choquette-Choo, and Nicolas Papernot. Entangled watermarks as defense against model extraction, 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2012.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks, 2014.
10

---

Published as a conference paper at ICLR 2021
Ram Shankar Siva Kumar, Jonathon Penney, Bruce Schneier, and Kendra Albert. Legal risks of adversarial machine learning research. arXiv preprint 121i::2000616779, 2020.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Process- ing Systems, pp. 70.710 7167-7177, 2018.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In USENIX Security Symposium, 2020.
Chuan Li. OpenAl's GPT-3 Language Model: A Technical Overview, 2020. URL https://
lambdalaba com/ hbtp//llogg...
Shiyu Liang, Yixuan Li, and R Srikant. Principled detection of out-of-distribution examples in neural networks. arXiv preprint TTKG::/00000669000 2017.
Kang Liu, Brendan Dovan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against back- dooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273 294. Springer, 2018.
Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching, 2019.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain gener- alization in deep learning. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http::/ufldd. stanford.ed.. hoosenumberss/miss2000 htw.i/pppoothhhuuuuuuubbrrrrr housenumbers. pdf
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: https://distill.pub//2
1..23915ddistll..
Chris Olah, Arvind Satyanarayan. Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and
Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915/ distill.00010. https://distill.uub20000 8/building-blockk blocks.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models, 2018.
Nicolas Papemot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning, 2017.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Herve Jegou. White- box vs Black-box: Bayes Optimal Strategies for Memhership Inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of Machine Learning Research, volume 97, pp- 5558-5567. PMLR, 2019.
Masoumch Shafieinejad, Jiaqi Wang, Nils Lukas, and Florian Kerschbaum. On the robustness of the backdoor-bassed watermarking in deep neural networks. arXiv preprint arXiv:1906.07745, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEF, 2017.
Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. Sponge examples: Energy-latency attacks on neural networks, arXiv preprint arXiv:2006.03466 2020.
Congzheng Song and Vitaly Shmatikov. Overlcaming reveals sensitive attributes. CoRR, abs/1905.1 1742, 2019. htt.hhttp:///aav.ooo/////// 9000.. 11742] 11742

---

Published as a conference paper at ICLR 2021
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, January 2014. ISSN 1532-4435.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprini arXiv:1312.6199 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision, 2015.
Florian Tramèr, Fan Zhang, Ari Jucls, Michael K. Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th USENIX Security Symposium (USENIX Security /6), pp. 601-618, Austin, TX, August 2016. USENIX Association. ISBN
Bean-Baptistt Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papernot, Data-frec model extraction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin' Satoh. Embedding watermarks into deep neural networks. CoRR, abs/1701 7040022 2017. URL. http:://exxiv.org////0/
17C1. 04082
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Daniel J. Wilson. The harmonic mean p-value for combining dependent tests. bioRxiv, 2018. doi: 10.1101/171751 htttt///wwww............oo//oota
771751
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144 2016. URL http::///aaa wwtp:///ggga/a/////ob/ 08144
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2.
2019.
Ziqi Yang, Hung Dang, and Ee-Chien Chang. Effectiveness of distillation attack and countermeasure neural network watermarking, 2019.
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF). pp. 268 088-282,11 2018.
Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2016.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A THEORETICAL MOTIVATION
In this section, we provide the formal proofs of Theorems 22 the preliminaries including the binary classification task and the machine learning model used to train the same in Appendix A.
2 as stated in 8 First, we describe
A. PRELIMINARIES
We repeat the preliminaries described in to discuss the proof's in the following sections.
Setup. Consider a data distribution D, such that any input-label pair (x, y) can be described as:
y
(-1,+1};
X1 yu E ERR X2~N(00,0²) E
where = X2) E RK+D and E RK is some fixed vector. This suggests that the last D dimensions of the input Gaussian noise which has no correlation with the correct label. However, the first K input dimensions are sufficient to perfectly separate data points from classes (-1, +1). The setup is adapted from Nagarajan & Kolter (2019). We use S and D represent the subset of the training set S and the distribution 2 with label = 1.
Architecture. We consider the scenario of classifying the input distribution using a linear classifier, ʃ. with weights (W1, W2), such that for any input:
f(x) W1 X1 + W2 X2
(3)
While we only discuss the case of a linear network in this analysis, the success dataset inference (like membership inference) only increases with the number of parameters in a machine learning model (Yeom et al. 2018), which in effect makes the following analysis a stronger result to prove. Prior works have also argued how wer-parameerizzd deep learning networks memorize training points (Zhang et al. 2016 Feldman 2019).
A.2 TRAIN-TEST PREDICTION MARGIN (THEOREM I
Training Algorithm. We assume that the learning algorithm initializes the weights of the classifier to zero. Sample training set S ~ D {(x(),)))) (y() =i....mi .... The learning algorithm
maximizes the loss C(x. y) = y f(x) and visits every training point once, with gradient update step of learning rate =
W1 W1 +ay (x
(4)
W2 - W2 + az 0xx2
From the optimization steps described above, one may note that the learned weights for the classifier are given by W1 and W2 Σ, (()) irrespective of the training batch size.
Inference. For any data point (x(1) )( caleulate its 'prediction margin' as the distance from the linear boundary, which is proportional to its label times the classifier's output y fxx). For any point,
(2) D, the 'prediction margin' is therefore given by:
E5 X2
y. f(x) (W1 X1 + W2 x2) y. (mu) (yu) y.
(5)
c+ y. Σ
1 ( X2
13

---

Published as a conference paper at ICLR 2021
Now, calculate the expected value of the margin for point randomly sampled from the training set. Consider any point in the training set (x, y) ~ S = (xx)) for some index j. Then, we have:
]
Ex fx = c+ 1x,00
Σ (() X2
X2 (3)
x
+ Ex (x,00000000) (100 5))
= =c+ D
(6)
Note that in (6). we utilize the fact that the square of standard normal variable follows the x distribution; and that the expected value of product of independent random variables is same the product their expectations, followed by the linearity of expectation.
Similarly, consider data point (x. 1) D*
(x (xx yfc Exx ( ((00000)
{
(
[E
x22 x2
Once again, in ( we utilize the fact that the expected value of product of independent random variables is same as the product of their expectations, followed by the linearity of expectation. At an aggregate multiple data points, can hence show that (xxx)))+ [xx) E(x,))~++ [(x) Do2. This concludes the proof for Theorem
A.3 FAILURE OF MEMBERSHIP INFERENCE (THEOREM
2
In this section, we take formal view of the conditions that lead to the failure and success of mem- bership inference. Before we begin with our formal analysis, we would like to point out that the statistical difference between the distribution of training and test data points in Theorem I is only observed when we aggregate an expectation over multiple samples. Now, we show that the variance of this difference is so large, that is very difficult to make any claims from single input data point.
Consider an adversary that does not have knowledge of the private data used to train machine learning model. However, it contains domain knowledge of the task that the model is trying to solve. This may include the range and dimension of possible inputs to the model. our case, the adversary has knowledge of the data distribution 2, but not of the training set S.
For single data point x (x1, X2), s.t. (x, y) ~ D, the adversary aims to predict whether was used to train the machine learning model, f. The prediction margin for (x, y) D is given by:
y f(x) y. (W1 X1 + W2 -2-x2) x2) cc
(v
Σ
(1) (1)
X2
From the analysis in Theorem the adversary knows that E(x,)) (y [xx) = c + Do and E(x,y)=D ly- fx)] = c. Let M(xff) represents the membership decision of the adversary for given data point and classifier f. The strongest adversary will use the following decision rule:
1, if (y f(x) c) 2
M(xif)
(0 o.w.
where E [0, 2o is some threshold that the adversary can tune in order to achieve maximum true positive rate and minimum false positives.
Similar to Yeom et al. (2018), we consider the scenario where the input data is randomly (with equal probability via coin flip b) sampled from either S (if b or D (if 0). Let such distribution be specified as (x, b) ~ R. The adversary M must maximize the single objective P( [Mxx)f) = 6). summary,
PO (M(x/S) = b) Pix.y>s [M(xff) = 1] + IP (( [M(x)f) =
(10)
2
14

---

Published as a conference paper at ICLR 2021
We simplify analysis by considering the data point (x, y) ~ D (has true label, y 1). How- ever, the analysis generally applies to any (x, y) D.
Case 1: (x, y) ~ D Assume meta-variable 22 = (E: y X2 (0). Therefore, 22 ~ (0, 1) while X2 ~ (0, o²)) Recall that X2, 22 E RDD Assuming D to be large, we can conveniently apply the central limit theorem approximate the distribution of the internal term. Let the individual dimensions of 22 be denoted by Z2 (x). Then, we have that:
{
m E 19
((x))=D+ [M 0] P ((x.))>/++
<
P ((x,)>>
[[z2 X2) <t]
(11)
1
Σ Dzz(1) x2(1)
<t
D
Let represents the distribution followed by z2 (1). From CLT, we have that the combined distribution behaves like normal distribution, with = Max and o² D
Po = - = m D2.
(12)
We use the fact that Var[XY] = Y and Varle- X] = 2² Var[X] for computing aa Therefore, let (O, mDo*):
P(x,)) [M(x))) = 0] Po= [r f|
(13)
It can be observed that P [rr < f] increases with the threshold value t. For t = 0, P [[z x2) <0
0.5. Whereas, for Do?, the probability decreases with the value of m (this can be intuitively understood as since the size of training set increases, overfitting decreases, making MI more
difficult). Even for as low as m 100 points in the training set, [ X2) < Do
0.6.
For any value of t E [0, o²], the maximum probability for size of training data m = 100 is 0.6. Further, the size of the training set increases, the probability tends to 0.5.
Case 2: (x, 1) ~ S+ Once again, as in the proof for Theorem consider any point in the training set (x, y) ~ S = (x(1), 1) for some index j. We will now calculate the probability of success of the adversary that follows the decision rule described above:
Σ
((xx)~++ [M(x = 1]= P(x
y
X2
X2
> 1
(14)
iij
y
x2 (1) + () x2 0)
Now, following the discussion in the first case, we know that the first term can be approximated by a variable (0, (m 1) Similarly, using CLT over the of multiple random variables sampled from a xi distribution, we can approximate the second term in the above equation with a variable 3 ~ NDDo2 Do Finally, using the property for sum of independent gaussians, we can approximate the entire 'prediction margin' to be represented by a sample u ~ ADDo mDo)). Then, have that:
P(x,))++ [M(x)f) = 1] Pu- [u =t]
(15)
Hence, show that the adversary do no better than coin flip. This concludes the proof for Theorem The interested reader may further analyze the assertion that the optimal value lies in 00 Do
To resolve the optimal threshold t for membership inference, we restructure the arguments as fol- lows. Recall from (1)) that the adversary aims ensure both true positive rates and true negative
15

---

Published as a conference paper at ICLR 2021
rates are high. We know:
IP (xx,))D [Mx[f) 0] P (r,N((, (r <
P (x.,))=++ M(x 1] WDo (DD"" (u t]
(16)
((),)) M(x = =
0)
(u r) ~ WDoo 2mDo"). This simplifies our discussion to a single normal distribution with mean Hy Do and variance, o 2mlDo.. We can now calculate the CDF at I evaluate the maximum probability of success of membership inference (decision taken by the optimal adversary).
We know that both u, are sampled from normal distributions. Therefore, define 7
Let Z (0 1). can hence be shown that:
14 07 +b
My 04y
Ply > 0] = P (o, M4) =
Z>
=
1
(17)
= 1 -
-
2m
Clearly, as m + 00, Ply > 0] + 0.5. This concludes the proof for Theorem
A.4 SUCCESS OF DATASET INFERENCE (THEOREE
In Theorem 2 we showed that adversary querying a single data point say no better than coin flip about the presence or absence of a given data point in a model's training set. In this section, we show that when reverse this adversarial game, the victim utilize the information asymmetry to predict with high confidence potential adversary's 's model stole their knowledge in any form.
First, recall that the victim has access to its own private training set of size For the purposes of this proof, we call it SV. As the victim has complete information the data distribution, it can randomly sample another dataset So D.
The victim considers that the potential adversary's 's model was stolen the mean 'prediction margin' for the points in Sy is greater than Sp by some threshold parameter A. Let v)v(J, S; be V's decision function to resolve ownership claims.
Recall that in Theorem we had calculated the expected value of the difference in the prediction margin for the points in the training set versus those in the test set. In the proof of this theorem, we calculate the probability of the mean of the difference being greater than some value A.
Now, let us calculate the probability of this margin for data point randomly sampled from the training Let ly represent the mean of the 'prediction margin' of all points in Sy for classifier f. Similarly, let to represent the mean of the 'prediction margin' of all points in So for the classifier f. We will use Ug denote the last D dimensions of points Sp. Then,
4v = -
(0) x2() + +x2 00)
i
X2 (1))
10 x2
(0) x2 x2 (1)
(18)
to Σ
0x2
u2 (1)
Pluv(f,;DD) = = = P(tv to) A 
Recognize the similarity of the above formulation with that discussed in the proof for Theorem in Appendix A.3 Let = ty tn- Then the random variable t represents the a sample from the distribution of means for 7 defined in Appendix A.3 We directly the Central Limit
16

---

Published as a conference paper at ICLR 2021
Theorem for this proof. Therefore,
14 = 142 Do² of o² m
(19)
=
Hence, (Do2 2Do*). It important note that this distribution independent of the number of training points. Hence, unlike membership inference, the success of DI is not curtailed by the lack of overfitting.
Similarly, for an honest adversary, the distribution of 'prediction margin' for points in Sy is the same as that for the points in So. directly follows that:
TP ( = 0]
= P X]
(20)
where, f ~ (0, 2Do Once again, like the proof of Theorem 2 by symmetry of two normal distributions with the same variance, and shifted means, can find that the optimal value of the parameter À that maximized true positives, and minimizes false positives, X 4
Let Z ~ N(0 1). Then it can hence be shown that:
PP > =| P o, + 2e
Ht
=
6
Pi
1
Mg
2
20:
20,
(21)
VD
=1 +
2/22
Clearly, D xc, Pli A] - 1.0. This concludes the proof for Theorem 3
B MODEL STEALING TECHNIQUES
In this section, provide more details about the various threat models that consider in this work. We also provide specific use-cases and motivation for the respective threat models, and introduce a new adaptive adversary targeted specifically against DI.
V: : Victim. The victim V wishes to release its machine learning model to the community, either as service, or by open-sourcing it for non-commercial academic use. V wants to ensure that the deployed model is not being misused under the terms of license provided.
An Data Access. The adversary Ap is able gain complete access the victim's private training data, and aims to deploy its own MLaaS by training the same. We note that labeled private training data one of the most expensive commodities in the deployment cycle of modern machine learning systems.
1. Model Distillation: Traditionally, model distillation (Hinton et al. 2015 was used method compress larger models by training smaller students using the logits of teacher model. We use this as threat model that the adversary may employ to distance its predic- tions from a model that was trained using hard labels from the dataset itself. The adversary requires both query access, and access to the victim's 's private training data for this attack.
2. Modified Architecture: Multiple works have attempted at identifying unique properties (or 'fingerprints') ) of a model by analyzing specific activations and representative features of internal model layers (Olah et al.] 2017] 2018 Yin et al., 2019)). We study the threat model where the adversary attempts training an alternate architecture on the victim's 's private dataset D Priv to valid the robustness of our method to changes in model structure.
Ait Model Access. The use-case of such adversary is two fold: (1) the victim open-sources their own machine learning model under a license that does not allow other individuals to monetize the same; and (2) the adversary gains insider access to the trained model of victim. In both the cases, the adversary aims to monetize its own MLaaS and deploys their own model on the web, by modifying the original victim model reduce the dependence on K.
17

---

Published as a conference paper at ICLR 2021
1. Fine-tuning: The adversary has full access to the victim's machine learning model, but not to its training data. While fine-tuning is employed used to transfer the knowledge of large pre-trained models given task (Devlin et al. 2018), we use it as stealing attack, where the adversary uses the predictions of the victim model on unlabeled public data in order to modify its decision boundaries. We consider the setting where the adversary can fine-tune all layers.
2. Zero-Shot Learning:
This is the strongest adversary that we introduce specifically targeted are the first to consider such
to evade dataset inference. To the best of our knowledge, threat model. The adversary uses no 'direct' knowledge of the actual training data to avoid any features that it may learn result of the training on the victim's 's private data set. The adversary has complete access to the victim model, and uses data-free knowledge transfer (Micaelli & Storkey 2019 Fang et al. 2019) train student model.
Aq: Query Access. Model extraction Tramer et al., 2016) is the most popular form of model stealing attack against deployed machine leaning models on the web. We discuss the related work on model extraction attacks in more detail in $2 Depending on the access provided by the machine learning service, an adversary may aim to extract the model using the logits or the labels alone.
1. Model Extraction Using Labels: The victim model is used to provide pseudo-labels for a public dataset. The adversary trains their model this pseudo-dataset. The key difference is that the input data points may be semantically irrelevant with respect to the task labels that the adversary's model is being trained on.
2. Model Extraction Using Logits:
The performance of model extraction attacks can be im- proved when the victim provides confidence values for different output classes, rather than the correct labels itself. The adversary's 'ss model is trained to minimize the K divergence with the outputs of the victim on a public (or non-task specific) dataset.
I: Independent Model. Finally, we also study the results of dataset inference on an independent and honest machine learning model that is trained on its own private dataset. This is used as a control to verify that the dataset inference procedure does not always predict that the potential adversary stole the victim's knowledge?
C EMBEDDING GENERATION
Embedding Generation Hyperparameters.
For the case of MinGD attack, we perform adversarial attacks defined by the optimization equation
min Axx, + 6) s.t. f(x + 8) = t; x x [0, 1J"h
(22)
The distance metric ( x x() refers to the Ep distance between points x(9 and x(i) for p E
{1, 2, oo}, and is the targeted label. To perform the optimization, we perform gradient descent with steps of size C'p. We take maximum of 500 steps of gradient optimization, but pre-terminate at the earliest misclassification. The step sizes for the individual perturbation types are given by {ak,,, 022 a1} {0.001, 0.01, 0.1).
For the case Blind Walk, We sample a random initial direction Starting from an input (x, y). take A E N steps in the same direction until f(x + 30)=,, t; & + y. Then, A + k:)) is used as a proxy for the ppediction margin' of the model. We repeat the search over multiple random initial directions to increase the information about a training data point's 's robustness, and use each of these distance values as features in the generated embedding.
As implementation detail, sample between uniform, laplace and gaussian noise to generate embedding features. To measure the final perturbation distance from the initial starting point, we use different b, norms for each of the noise sampling methods. For uniform noise, we compute the loo
'Note that since we consider the difference in the distribution of outputs of the auxiliary classifier on em- beddings from the test and training set (rather than hard labels from the auxiliary classifier), even in the absence of this control, we can un-deniably verify the confidence of dataset inference. This is only included to contrast the difference and make the effects of the method clearer the reader.
18

---

Published as a conference paper at ICLR 2021
distance; for gaussian noise, the &g distance; and for laplacian noise, the & distance of the nearest misclassification. While we take k steps of Blind Walk up till misclassification, however, we do not exceed more than 50 steps and prematurely terminate without misclassification in the event that the prediction label does not change.
Performance of White Box Approach. We find in our evaluations that the white-box MinGD method generally underperforms the Blind Walk method. This happens despite its ability of being able compute the nearest distance to any target class more accurately. While the onset, this may seem to be a counter-intuitive result, since generally with more access, the performance of mapping the neighbours should only increase.
However, we note an important distinction. The end goal of the query generation process is not to calculate the minimum distance target classes accurately, but rather understand the 'prediction margin' or the local landscape of given data point. Readers may recall from adversarial examples literature (Szegedy et al. (203)) that adversarial examples can easily be constructed on the dataset that a given machine learning model was trained on. This observation hurts the idea of Figure [b] Despite pushing the neighbouring class boundaries away, the existence adversarial examples elu- cidates the existence of small pits within the landscape of the model.
We hypothesize that the gradient-based optimization objective aims to capture this minimum (ad- versarial) distance, and fails to capture 'prediction margin' that is more representative of the clas- sifier's prediction confidence or general landscape. On the contrary, Blind Walk is able to perform spectacular job the same end goal. Since we minimum distance to the neighbouring classes, multiple Blind Walk runs effectively map the 'aver- age case' prediction margin, which argue is more useful than the 'worst case' prediction margin as obtained by MinGD.
longer adversarially trying to optimize the
D EFFECT OF EMBEDDING SIZE
For all models, richer embeddings reduce the need for more revealed samples. (See Figure 4 We note that the main body of this work, we had used a fixed size of embedding vector, with 30 input features. However, recall that in the black-box setting, the victim incurs additional cost for querying the potential adversary. Therefore, in this section we aim to understand the marginal utility of extra embedding features added. In general, we find that for most of the threat models studied, using only 10 features for the embedding space is sufficient to achieve the required threshold p-value of 0.01. This suggests that we can slash the number of queries made to the potential adversary by one-thirds, without loss in confidence of prediction.
Interestingly. we also note that even in scenarios where the victim reveals only 15 samples, addi- tional embedding fcatures have insignificant advantage as opposed to querying fresh samples. This suggests that the amount of entropy gained by revealing that by extracting more features (beyond 10) for the same data point. We also note that the effect is not-consistent in the zero-shot learning threat model.
data point is significantly more than
E EXPERIMENTS
E. DATASET DESCRIPTION
CIFARIO. CIFAR1O (Krizhvskyy 2012) contains 60,000 coloured images with 10,000 reserved for testing. There are 10 target classes with 5000 training images per class.
SVHN. SVHN (Ntterr et al. 2011) is a dataset obtained from house numbers in Google Street View images. The underlying task is of digit classification from 32x32 coloured images.
CIFARI00. CIFAR100 (Krzhevsky 2012) also contains 60,000 coloured images with 10,000 reserved for testing. There 100 target classes with 500 training images per class.
ImageNet. The ImageNet dataset (Deng et al. 2009) is large-scale benchmark, consisting various challenges including that of image recognition for machine learning systems.
19

---

Published as a conference paper at ICLR 2021
Threal Model Dotilation
Threat Model Label Query
Theatt Model Logit
0.22
p-vabe
0.15
0.30-
0.00
13
30
Model Leassing
Theast Model Fine-Turiny
Threst Model Archioeiue
02
0.225
p-Vabee
0.15
0.10
0.02
5
10
20
25
30
10
15
20
30
10
10
20
30
Loatine-lmmbeddingg Stee
Distonc--nnbokiig Size
Distnce-EEmbeddie Size
Figure 4: p-value vs. distance-embedding size
Dataset Inference on SVHN (Blind Walk Attack)
Model Stealing Attack
Ap
p-value
Source
0,950
10-8
Distillation
0.537
10-a
Ap
Diff. Architecture
0.450
10-2
Zero-Shot Learning
0.512
10-3
AM
Fine-tuning
0.581
10-1
Label-query
0.513
10-03
Aq Random-query
Logit-query
0.515
10 10-02
0.475
10 10-02
I
Independent
-0.322
10-01
Table Ownership Tester's effect size in a small-data regime (using only m 10 samples) on the SVHN dataset using Blind Walk attack. 2nd highest and lowest effect size is marked in red & blue.
E.2 ADDITIONAL DATASETS
To further validate our claims about the success of dataset inference, we provide evidence on two additional datasets. In this section, we present results on the SVHN and ImageNet datasets.
The results of DI via the 'Blind Walk' attack on the various threat models discussed in Ap- pendix B are presented in Table To perform model extraction and fine-tuning attacks, we utilized the set of 'extra' images available with the SVHN dataset. We use the first 50,000 images in this set to stage such attacks that require a surrogate dataset. For training on the original dataset with a different architecture, we once again utilize the Pre-activation version of ResNet- 18 as for CIFARI0 and CIFARI00 100 in the main paper. Notably, we also introduce another threat model: Random-query which describes a scenario where the victim is queried with completely random inputs. While the zero-shor learning framework also queries the victim with synthetic images, the queried images synthesized to maximize the disparity between predictions of the student and teacher. On the contrary, in case of Random- queey, we query the victim by sampling from a normal distribution, (0, 1). DI is resilient to completely random queries well.
SVHN.
20

---

Published as a conference paper at ICLR 2021
Threat Model
ImageNet Architecture
Aμ
p-value
Source
Wide ResNet-50-2
1.868
10
Diff. Architecture
AlexNet
0.790
10
Ap
Diff. Architecture
Inception V3
1.085
10-3
Table 3: Ownership Tester's effect size in a small-dats regime (using only m 10 samples) on the ImageNet dataset using Blind Walk attack.
Fraction Overlap
Aμ
p-value
0.0
-0.172
0.308
0.3
0,499
7.93 10"3
0.5
0,514
5.78
10-3
0.7
0.576
2.52 x 10
1.0
0.566
3.45 x 10-3
Table 4: Ownership Tester's effect size in small-data regime (using only m 10 samples).
Note that we do not include Random-query as threat model in case of CIFARIO and CIFAR100 datasets because random querying is insufficient to achieve model extraction accuracies greater than the majority class baseline in more complicated tasks such as these. However, in case of SVHN, we were able to train an extracted model with 90.2% test set accuracy using random queries alone. Similar observations have been shared in other model extraction literature well (Truong et al. 2021]. We found that our conclusions hold for this additional dataset and we can claim ownership with few 10 examples.
ImageNet. We remark that prior work in model extraction has not successfully demonstrated the efficacy of model stealing large-scale benchmarks like ImageNet. These methods require many queries to steal model, and are hence not practical yet. As a proof of validity of our approach, we demonstrate dataset inference (DI) in the threat model that assumes complete data theft: the adver- sary directly steals the dataset used by the victim to train a model rather than querying the victim model. We use 3 pre-trained models on ImageNet using different architectures, and treat the one with a Wide ResNet-50-2 (Zagoruyko & Komodakis 2016) backhone as the victim. We then observe if DI is able to correctly identify that the two other pre-trained models (AlexNet (Krizhevsky) 2014) and Inception V3 (Szegedy al.) (0155) were also trained on the same dataset (i.e.. ImageNet). We confirm that DI is able to claim ownership (i.e., that the suspect models were indeed trained using knowledge of the victim's training set) with p-value of 10-3 on revealing only 10 samples.
For performing DI, the victim trains the confidence regressor with the helps of embeddings generated by querying the Wide ResNet-50-2 architecture over the training and validation sets separately. We first note that the confidence-regress test set, despite the fact that the ImageNet dataset is orders of magnitude larger than the previous benchmarks experimented on. With only 10 examples, we attain p-values less than 10 10-30, 30 Finally, test if this generalization holds for other architectures that underwent disjoint training procedure, we experiment over AlexNet and Inception V3. We find that given 10 examples from the ImageNet training dataset, DI can confidently say that the suspect models utilize knowledge of the victim's training set with p-vuuues less than 10 Note that these models are trained on large datasets where works MI attacks train victims on small subsets of training datasets to enable overfitting Yeom et al. 2018). We helieve this demonstrates that DI scales to complex tasks.
generalizes well to other points in the victim's train and
F EXTENT OF OVERLAP
In this section we elaborate upon the effect of overlap between private datasets of two parties and how does dataset inference respond to such scenarios. More specifically, we study the amount of overlap required for DI to be able to claim theft of common knowledge in the following scenario: We consider a competitor (or adversary) who owns their own private training dataset S The adversary gains access the victim's training dataset Sy. The adversary now trains their MI. model
21

---

Published as a conference paper at ICLR 2021
0.05
Source
0.5
0.04
0.0
0.4
0.3
0.03
0.5
p-value
0.3
0.7
0.02
0.2
1.0
0.0L.
0.1
0.0
0,00
10
20
30
10
20
30
Figure 5: Ownership Tester's p- p-value depicted as a function number of training samples revealed (m). In the figure to the left, for honest adversary whose dataset has no overlap with the victim's 's dataset, the p-value increases as number of revealed samples increases, indicating decrease in confidence of claim for knowledge theft. For all adversaries with fractional data overlap (to the right), DI able to achieve a p-value of less than 0.01 in under 10 samples.
on S SA where Sy^ C Sy and |Sy^| = AlSyl. That is the new dataset has fraction of the training points private to V.
Since the training time the adversary optimizes the 'prediction margin' over all points in S, the prediction margin for points in Sy also gets affected. At test time when the victim queries on these points, DI is expected succeed.
We validate this claim on the SVHN dataset which provides a set of "extra" images apart from 'train" and 'test' sets. We train the adversary on the union of 'extra' and varying fractions of the 'train' set, where the 'train' set is supposed to be private to the victim. At the time of dataset inference, the victim queries 50 samples from private 'train' and 'test' set to the adversary's 's model. Dataset Inference succeeds with p-value 10-3 for fractions of overlap 0.3, 0.5, 0.7, 1.0 as tested. More importantly, as the overlap goes to 0, DI once again does not claim knowledge theft. We present our results for DI on revealing 10 samples in Table 4 and present graphical illustration of these results with varying numbers of samples revealed in Figure 5 From Table 4 it may also be noted how the effect size increases with the amount of overlap of the private training indicating that the DI is becoming increasingly confident of knowledge theft.
22


Reviews:
[{"review_id": "nVrDK7MbBfu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles a timely problem of detecting model stealing attacks. The proposed identifies the stolen model by investigating whether the model is trained on the same dataset as the victim model. \n\nPros:\n-\tThe proposed method of dataset inference for model stealing detection is novel.\n-\tThe experiments provided by the paper are comprehensive, including different assumptions of data access, model access, and query access, etc. Adaptive attacks are considered in the paper as well.\n-\tThe paper is well-written and easy to follow.\n\nCons:\n-\tZero-shot learning ([9] in the paper) cannot be directly used as a model stealing attack, because the proposed approach in [9] requires more information of the victim model than the input and output pair. It would be better if the paper could investigate other data-free model stealing approaches (e.g., [1][2]). Also, it is not clear to me how the proposed dataset inference approach is applied to data-free model stealing attacks, since the training dataset used by the attacker is different from the victim’s dataset. If the attackers use some synthetic data to steal the model, will the proposed approach work?\n-\tWhat are the white-box and black-box settings in Section 4.1?\n\n\n[1] Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 512–527. IEEE, 2019.\n\n[2] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506–519, 2017.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "C_lQJCdMuxi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes to defend against model stealing attacks by dataset inference. It is argued that other watermarking techniques are not robust enough to be transferred during model stealing.\n\nOwnership resolution is performed by kind of a membership inference attack that the model owner runs on the potential copy. As only the owner knows the true training data, the \"attack\" can be amplified by a few samples. This leads to a rather strong signal in order to show model provenance. Final decision is formulated as a hypothesis test in order to get a confidence. The argumentation why for larger models this is not dependent on some sort of over fitting is not clear.\n\nWhile the idea is interesting, it is somewhat at odds with others goals in machine learning. In general, it is not a desirable property if the trained model contains \"artifacts\" related to the training data. This on the one hand could be overfitting artifacts that hamper generalization or more prominently could mean that the model does not preserve privacy. If such overfitting is not a major issues and privacy is not a concern, the approach seems viable. However, there should be an extended discussion under which situations/models the method is feasible. (e.g. privacy preserving learning?)\n\nI somewhat disagree with the practice of moving the related work to the appendix. this is a vital part of a paper and not an appendix. this looks like breaking the page limitation. overall, having a 19 page submission is not quite following the submission guideline (although the additional material is useful). this should not become practice - in particular as some parts in the main submission could be more concise. In particular, as the actual algorithm is quite simple (which I don't want to see as a disadvantage as it's well motivated)\n\nOverall, the paper is insightful -- also adding to the understanding of the connections to membership inference. Also the attention to important details like query efficiency is taken care of.\n\nIt is bit disappointing that the method is not evaluated on more complex classification problems. In particular, as membership inference (which this method is based on) can be even weaker on those larger and well trained models. The claim about other watermarking techniques no being effective - although probably true - was not substantiated (unless I've missed). at least there is no comparison to other attribution/provenance techniques.\n\nWhile this submission is a very interesting and enjoyable read, in the end it fails to fully convince how large the impact will be. Questions are open w.r.t. models with privacy guarantees and larger models  ... and also in the end, this technique is still effected by fine tuning. It is unclear to what extend this is an issue.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3X0Urw4ywNk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update** : Since nearly all of my issues have been addressed, I have changed my rating from 5 to 7. Best of luck :)\n\nSummary:\n\nThe authors study a subset of model stealing by introducing dataset inference, a process to identify whether a suspected adversary's model has private knowledge from the original model's dataset. Through their experiments on two related tasks in the vision domain (CIFAR-10,100), the authors demonstrate their approach for dataset inference that combines statistical tests and decision boundary-based checks to make claims of model theft. They base their algorithms on theoretical results in toy settings (proofs shifted to additional materials).\n \n##########################################################################\n\nReasons for score: \n\nOverall, I feel the final observations are quite useful and pave new methods for asserting model theft in real-world settings. Although the proposed techniques are not novel (in fact, a lot of existing literature is not cited, giving the impression that a lot of proposed techniques are contributions of the paper itself), the end results and observations can be useful in practical settings. I feel the authors need to make a lot of changes to the current draft before it can be ready for acceptance. \n \n##########################################################################\n\nPros: \n  \n- Mentioning points such as how issues like these can actually get to courts in real life and are covered by intellectual theft laws is great to see. Since the proposed solution tackles a real-world (potentially) problem, it makes sense to incorporate real-world implications as well.\n\n-  Using theoretical analyses to justify the proposed methods is a great way to approach a problem like this. Testing them out on simpler problems to see how and most importantly, why they work, is important.\n\n- Evaluation over multiple model-stealing attacks is a great way to both show the robustness of the proposed attack method, as well as identify strong (fine-tuned models) as well as weak (label-query attacks) points of the algorithm.\n\n- The query efficiency of the proposed method is quite remarkable. A method that can use these little samples in a real-world setting to assert model stealing (or check for it) with high confidence can be very helpful.\n\n- Page 8: 'gradient-based approaches are sensitive to numerical instabilities' is a great and very valid point. An adversary could in fact even deliberately resort to gradient masking to avoid such an attack.\n \n##########################################################################\n\nCons: \n\n- By biggest and foremost problem is with the threat model. In all of these settings, it is assumed that the party accusing the adversary of stealing dataset access provides actual data from its own training dataset. However, since this is a real-world scenario, there should also be some way for the accuser to provably show that the data points it is using for the test are in fact from its own training set. Also, it should be made clear what level of overlap in the datasets used is problematic. This parameter would vary from problem to problem and thus cannot be one fixed value. Nonetheless, it should be a parameter of the testing process.\n\n- The term 'dataset inference' is slightly misleading. In some portions, the authors say \"we propose to identify stolen copies by showing that they were trained on the same dataset\". In other places, they imply that the adversary used a \"subset\" of that knowledge. It is important to both be consistent in this definition, and to quantify it: nowhere in the paper is it mentioned how much overlap is acceptable as \"sheer chance/possible via domain knowledge\", and how much is \"definitely stealing\".\n\n- Key implementation details, such as how the datasets were split to be used by the target and adversary to train their models are omitted. \n\n- CIFAR-10 and CIFAR-100 are highly related datasets (same images). Claims made in the paper would hold more value if they can show similar trends on some other datasets. The authors could try some high-dimensional datasets such as tiny-Imagenet, or even something as simple as MNIST.\n\n- In several places in the paper, authors claim that watermarking models does not work in the sense that trying to copy a model does not copy the watermarks, since there is a difference in distributions. This is not true: recent works have shown honeypot attacks that transfer watermark behavior even when someone tries to copy a model using normal data. Please refer to Section 3.2.2 of [this work](https://arxiv.org/pdf/2009.12153.pdf) \n\n- When using sample points to assert model stealing, does the party arguing for their case release them all together, or one by one. One may argue that the sequential release of sample points could potentially be games (a malicious party could easily look at the target model and accuse them wrongfully). What are the countermeasures against such a scenario? It seems the final judgment can only be made by a third party by the other two parties revealing large subsets of their training sets and algorithms. This might not be possible in real-world scenarios, where datasets used to train models are bound by certain privacy-protection laws (GDPR, etc).\n\n- Section 2 \"V suspects theft\": what are the criteria here? Is it based on similarity in model performance, or pure suspicion? Since this section sets up the threat model, it might be beneficial to be as exact as possible.\n   (a) \"may gain access to a subset of\". As asked above, how big of a subset is a problem? Since the subset is of Kv and not Sv, it may be possible that they are not overlapping at all, or barely overlap. Would it still be a problem? \n   (b) Definition 1: \"prove that some knowledge\" since this is a definition, it should not be hand-wavy. In my opinion, the \"some\" here should be quantified perhaps by a parameter that captures the overlap in the two datasets and integrates them in the definition. \n   (c) Definition 2: Please rewrite the last point as a conditional assignment. It is hard to read in its current form\n\n- Section 3.3, 'Failure if Membership Inference' refers to two labels: b=1 when the sample is from S, or b=0 when the sample is from D. However, S is a subset of D and thus the first case will always be true. Assuming you mean 'sample is from S - D', please fix this.\n\n- Insufficient literature review. There are many works in the field that deal with the same problem and similar approaches, even though they do not define them as \"dataset inference\". Please search for some works on 'property inference': the current problem can be posed as differentiating between the dataset suspected to be used, versus some other dataset.\n\n- Although the white-box method looks at finding the 'closest' data points, the black-box method samples random directions and takes steps. Assuming this is intended to be a proxy of the white-box setting, it should take the form of sampling random initial directions and then, over all the samples, picking the closest directions.  Also, the white-box setting may give significantly different results if either of the models were trained with an adversarial robustness objective. It would be interesting to see how the attack performs in that case.\n\n- Section 4.2 claims that membership inference attacks do not take confidence into account and that this is a key difference from the proposed method in this work. This is not true: ROC curves for such attacks can be analyzed to set rejection regions for such claims. \n \n- Section 4.2, last line: \"publicly available data that is not used for training of Fv\" Is there a specific reason to use data that was not used to train Fv?\n\n- Page 8, 'White-box access is not essential to DI' claims the Blind-Walk approach is 'non-targeted'. This can also be made true for the white-box setting by adjusting the minimization objective to be non-targeted, instead of making it targeted and targeting the top-k classes.\n\nPlease address and clarify the cons above ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed to defend against model stealing attacks by dataset inference. The paper received unanimous rating of "Good paper" and "accept". The reviewers praise this paper insightful and well written. There are active discussion between the reviewers and authors, which further clarify some of the issues. Given the positive review and overall rating, the AC recommends it to be an spotlight paper.
Decision: Accept (Spotlight)