Increasing the Cost of Model Extraction with Calibrated Proof of Work
Authors: Adam Dziedzic
Muhammad Ahmad Kaleem
Yu Shen Lu
Nicolas Papernot
Venue: ICLR 2022 Spotlight (2022)
Keywords: model extraction
model stealing
model functionality stealing
proof-of-work
adversarial machine learning
trustworthy machine learning
deep learning

Abstract: In model extraction attacks, adversaries can steal a machine learning model exposed via a public API by repeatedly querying it and adjusting their own model based on obtained predictions. To prevent model stealing, existing defenses focus on detecting malicious queries, truncating, or distorting outputs, thus necessarily introducing a tradeoff between robustness and model utility for legitimate users. Instead, we propose to impede model extraction by requiring users to complete a proof-of-work before they can read the model's predictions. This deters attackers by greatly increasing (even up to 100x) the computational effort needed to leverage query access for model extraction. Since we calibrate the effort required to complete the proof-of-work to each query, this only introduces a slight overhead for regular users (up to 2x). To achieve this, our calibration applies tools from differential privacy to measure the information revealed by a query. Our method requires no modification of the victim model and can be applied by machine learning practitioners to guard their publicly exposed models against being easily stolen.

Published as a conference paper at ICLR 2022
INCREASING THE COST OF MODEL EXTRACTION WITH CALIBRATED PROOF OF WORK
Adam Dziedzic
Muhammad Ahmad Kaleem
University of Toronto and Vector Institute adam. dz i edz ceu.oooonooooooo.oooooooo
University of Toronto and Vector Institute ahmad. ka eem@mail www.uotoroooo...
Yu Shen Lu"
Nicolas Papernot
Stanford University
University of Toronto and Vector Institute nicolas, papernot ca
yushealu@stanford..com
ABSTRACT
In model extraction attacks, adversaries can steal machine learning model exposed via public API by repeatedly querying it and adjusting their own model based on obtained predictions. To prevent model stealing, existing defenses focus on detecting malicious queries, truncating, or distorting outputs, thus necessarily introducing a tradeoff between robustness and model utility for legitimate users. Instead, we propose to impede model extraction by requiring users to complete proof-of-work before they can read the model's predictions. This deters attackers by greatly increasing (even up to 100x) the computational effort needed to leverage query access for model extraction. Since we calibrate the effort required to complete the proof-of-work to each query, this only introduces slight overhead for regular users (up to 2x). To achieve this, our calibration applies tools from differential privacy to measure the information revealed by query. Our method requires no modification of the victim model and can be applied by machine learning practitioners to guard their publicly exposed models against being easily stolen.
INTRODUCTION
Model extraction attacks (Trmerr et al. 2016 Jagielski et al. 2020: Zanella-Beguele et al. 2021 are a threat to the confidentiality of machine learning (MI.) models. They are also used as reconnaissance prior to mounting other attacks, for example, if an adversary wishes to disguise some spam message to get it past target spam filter (Lowd & Meek 2005), or generate adversarial examples (Biggio) et al. 2013 Szegedy et al. 2014) using the extracted model (Papernot et al. 2017b). Furthermore, an adversary can extract a functionally similar model even without access to any real input training
data (Krishna al. 2020 Truong et al. 2021 Miura et al., (021] while bypassing the long and expensive process of data procuring, cleaning, and preprocessing. This harms the interests of the model owner and infringes their intellectual property.
Defenses against model extraction can be categorized as active, passive, or reactive, Current active defenses perturb the outputs to poison the training objective of an attacker (Orekondy et al. 2020).
Passive defenses try to detect an attack (Juuti et al., (2019) or truncate outputs (Tramèr et al. 2016).
but these methods lower the quality of results for legitimate users. The main reactive defenses against model extraction attacks are watermarking (Jia et al.) (2020)), dataset inference (Maini et al. 202
and proof of learning (Jia et al., 202T). However, reactive approaches address model extraction post hoc, i.e., after the attack has been completed.
We design a pro-active defense that prevents model stealing before succeeds. Specifically, we aim to increase the computational cost of model extraction without lowering the quality of model outputs. Our method is based on the concept of proof-of-work. (PoW) and its main steps are presented as block diagram in Figure I Our key proposal is to wrap model outputs in PoW problem to force users to expand some compute before they can read the desired output. The standard PoW techniques,
Work done while the author was at the University of Toronto and Vector Institute.

---

Published as a conference paper at ICLR 2022
Attacker
Defender
Send SendeQueries Queries
Information
Unlabeled
Leakage
Data
Estimator
Client
Generate
Server PoW
Cost
PoW
Puzzle
Queries
Attacker Model
Solve
Send
Verify
Victim Model
Puzzle
Solution
Solution
8 Predict
Labels
Return
Labels
Figure I: Machine Learning API with calibrated proof-of-work. Answering queries with PoW consists of the following steps: An attacker sends queries drawn from unlaheled dataset machine learning service. The information leakage estimator computes the privacy cost of the queries. The privacy cost of queries to the proof-ofwoodd (PoW) server, which adds the
to the total user's (attacker's) query cost. The PoW server generates puzzle, whose difficulty is based on the total cost incurred by the user. The attacker solves the puzzle using PoW Client and sends the solution back the server. The PoW server verifies the solution and if it is correct, it notifies the victim model, which 8 predicts the labels. The victim model returns the labels to the client which can potentially use the returned labels train its own model.
which initially were used in the security domain to prevent server from Senial-of-Service attacks (DoS) provide users with a small puzzle to be solved before giving access to the service (Dwork & Leatte--eannnnn-----ee
Naor 1993). In the of the label predicted by a model) can similarly be released to user once his or her computational or other system resources are spent. This can prevent a malicious user from obtaining enough examples labeled by victim model to steal the model. Intuitively. our method shifts the trade-off between the quality of answers and robustness, that was introduced by previous defenses, a trade-off between the computational cost and robustness to model stealing.
(MLaaS), the answer
query (e.g..
To minimize the impact of our defense on legitimate users, we the difficulty of the PoW prob- lem (Back) 2020) to an estimate of how much information about model has been extracted so far by a given user. We compute the privacy cost per query as an information leakage metric used in the differential privacy literature (Dwork et al. 2014)). We choose work with the privacy based metric because is (1) agnostic to the learning task and model architecture, and (2) practical to compute for the model owner. We tune the per-query difficulty of the required proof-of-work based the measured information gained by a user. PoW does not deteriorate the accuracy of answers and adds only small increase (less than 2x) in the cost of querying public models for legitimate users. The cost is usually much higher (up to 100x) for attackers.
To summarize, our contributions are as follows:
class of pro-active defenses against model stealing attacks that proof- of-work mechanisms to adaptively increase the computational cost of queries with a negligible increase in the cost for legitimate users, and high increase in cost for many attackers. Our method is the first active defense strategy that leaves the model's accuracy entirely intact.
We introduce
We propose novel application of differential privacy budgets way measure the amount information leakage from answered queries. We calibrate the difficulty of the proof-of-wort. problem based user's 's query cost estimated through its differential privacy budget.
Our rigorous evaluation on four datasets and eight attacks validates that adversaries attempting to steal model issue queries whose privacy cost accumulates up to 7x faster than for benign queries
In DoS attack, adversaries flond a server with superfluous requests, overwhelming the server and degrading service for legitimate
2

---

Published as a conference paper at ICLR 2022
Table I: Taxonomy of model extraction attacks. We use IND for in-distribution and OOD for out-of-distrbuttion data. The MixMatch attack is the best performing one with very few queries run against the MIL API and a high accuracy of the extracted model, mainly due the abundant and efficient usage of the problem domain (i.e., task-specific) data. At the other end of the spectrum is DataFree attack with only synthetic OOD data.
ATTACKS
DATA
OF ATTACKER'S QUERIES
ADVERSARY'S
VICTIM
TYPE
MNIST
SVHN
CIFAR
GOAT.
RETURNS
MIXMATCH
PROBLEM DOMAIN
<TOK
<TOK
<TOK
TASK ACCURACY
LABELS
ACTIVE LEARNING
PROBLEM DOMAIN
IK 10K
1K-25K
IK 9K
TASK ACCURACY
LABELS
KNOCKOFF
NATURAL OOD
1K 30K
IK S0K
1K- 50K
TAAK ACCURACY
LOGITS
COPYCAT CNN
NATURAL OOD
IK -5K
IK 50K
IK 50K
ACCURACY/FIDELITY
LABELS
JACOBIAN/-R
LIMITED IND
10K 80 K
1OK 150 K
1OK - 150 K
FIDELITY
LABELS
DATAFREE
SYNTHETIC OOD
20M
ACCURACY
LOGITS
issued by legitimate users. Thus, our proof-of work mechanism can deter attackers by increasing their computational cost significantly at minimal overhead to legitimate users.
We design adaptive attacks against our defense and show that they require simultaneous optimization of two opposing goals: (1) minimizing the privacy cost of queries run against the ML API and (2) maximizing the information leakage incurred by the queries. These conflicting targets render such attacks less effective, making them less attractive to adversaries.
2
BACKGROUND AND RELATED WORK
2.1 MODEL EXTRACTION ATTACKS
Model extraction aims to replicate some functionality of a victim model f(x; 0), where a denotes an input and 0 are parameters of the model. An attacker takes advantage of the ability to query the victim and observe the model's outputs. which are used to train a substitute model f'(z; 8¹). We test our defense against common forms of fidelity and task accuracy attacks. The fidelity extraction is given as input some target distribution Dy and a goal similarity function S(P1.P2). It maximizes the target similarity between the victim and stolen models: max PrawDy S(f'(x:; J 0)). The Jacobian- based attack (Papernot et 2017b) is a high fidelity type of model extraction because aims to align the gradients of the extracted model with the victim model's gradients. The task accuracy extraction aims at matching (or exceeding) the accuracy of the victim model. For the true task distribution D DA the goal is defined as: max Pr(ng)= (arg (f = y]. The task accuracy attacks include Knockoff Nets (Orekondy et al. 2019). DataFree (Truong et al. 2021) and MixMatch-based (Berthelot et al. 2019) model extraction.
We taxonomize these attacks in Table I from the perspective of vietim who observes the queries as well as the model's response on these queries. defender wants to defend against any of these attacks, regardless of their intent, while not causing harm to legitimate users. Attackers with problem domain data need fewer queries and access to labels only is sufficient for them to steal a model (c.g.. MixMatch extraction from Dagielski et al. (2020). ) On the other hand, the lack of such data increases the number of required queries and might call for access to logits (Truong et al. 2021).
2.2 DEFENSES AGAINST MODEL EXTRACTION
We taxonomize the space of model defenses into three main classes. First, active defenses perturb the outputs from a defended model to harm training of the attacker's model. They trade-off defended model accuracy for better security. Second, passive defenses either truncate the outputs, which also lowers accuracy, try to detect an attack, which makes them stateful since some form of information has to be stored about each user's queries and this might cause scalability issues. Other defenses are stateless. Third, reactive defenses try to prove model theft rather than preventing the attack from happening. We propose new class of defenses that are pro-active. We trade-off compute of the querying party for better security. We neither degrade the accuracy of the defended model nor require any changes to the serving model. Our defense is stateful but we only store privacy cost per user in a form of single float number. Next, describe specific examples of defenses from each class.
3

---

Published as a conference paper at ICLR 2022
Active defenses. Lee et al. (2019) perturb a victim model's s final activation layer, which preserves that top-1 1 class but output probabilities are distorted. [Orekondy et al. (2020) modify the distribution of model predictions to impede the training process of the stolen copy by poisoning the adversary's s gradient signal, however, it requires a costly gradient computation. The defenses proposed by Kariyappa & Qureshi (2020); Kariyappa et al.] (202T) identify if query is in-distribution or out-of- distribution (OOD) and then for OOD queries sends either incorrect or dissimilar predictions. The defenses introduced by Kariyappa al. assume knowledge of attackers' OOD data that are generally
hard to define a- a-priori and the selection of such data can easily bias learning (Hsu et al. 2020).
Passive defenses. PRADA by Juuti et al. (2019) and VarDetect by (Pal et al., 2021) are stateful and analyze the distribution of users' queries in order to detect potential attackers. The former assumes that benign queries are distributed in consistent manner following a normal distribution while adversarial queries combine natural and synthetic samples to extract maximal information and have, e.g., small l2 distances between them. The latter presumes that an outlier dataset, which represents adversarial queries, can be built incrementally and differs significantly from in-distribution data. However, PRADA does not generalize to attacks that utilize only natural data (Orekondy et al.) 2019) Pal et al. 2020 Follow-up works (Atli et al., 2020) have specifically targeted model extraction attempts that use publicly available, non- non-problem domain natural data, but they assume access to data from such attacker.
Reactive defenses. Watermarking allows a defender to embed some secret pattern in their model during training (Adi al.) 2018) or inference (Szyller al. 2019). Jia et al. (2020a) proposed entangled watermarks to ensure that watermarks are not easy to remove. Dataset inference (Maini al. 2021) identifies whether a given model was stolen by verifying if a suspected adversary's model has private knowledge from the original victim model's dataset. Proof of learning (Jia et al. 2021) involves the defender claiming ownership of a model by showing incremental updates of the model training. Unfortunately, reactive defenses are effective after a model theft and require a model owner to obtain partial or full access to the stolen model. This means that if an attacker aims to use the stolen model as proxy to attack the victim model, or simply keeps the stolen model secret, it is impossible for the model owner to protect themselves with these defenses.
PROOF-OF-WORK AGAINST MODEL EXTRACTION
Our pro-active defense is based on the concept of proof-of-work (PoW) and requires users to expand some computation before receiving predictions. We reduce the impact on legitimate users by adjusting the difficulty of the required work based on the estimation of information leakage incurred by users' queries. The cost of queries calculated using the privacy metric via the PATE framework (Papernot al., 201 7a 2018). standard approach to differential privacy in machine learning. Each user sends a query and receives a PoW puzzle to solve. The higher privacy cost incurred by a user, the more time it takes to solve the puzzle. The PoW server verifies whether the solution provided by the user is correct, and the outcome is positive, the server releases original predictions.
3.1 THREAT MODEL
We assume that an attacker has black-box access to the victim model. That is, the attacker can only choose the query sent to the victim and observe the response in the form of label or logits. The attacker has no access to the victim model parameters or training data. This is representative of most existing MLaaS systems. We use similar architectures for the attacker and victim it is shown that attacks with shallower models can result in sub-optimal results (Juuti et al.) 2019 Zhang et al., 202T). An attacker can have information about the strategy of our defense. For the purpose of our experiments, we assume that a legitimate user sends queries that are randomly selected from an in-distribution dataset unseen by the victim model and receives labels.
3.2 PROOF-OF- WORK PUZZLE
We propose to extend the scope of the PoW techniques to protect ML models from being easily stolen. One of the most popular oroof-of-wort. (PoW) algorithms is HashCash (Back) 2002), which is used as mining function in bitcoin (Back 2020).
4

---

Published as a conference paper at ICLR 2022
The PoW cost-function classification scheme presented in Back (2002) defines six characteris- ties: 1) efficiency of verifiabilitty (0 verifiable but impractical, I/2 practically verifiable. 1 efficiently verifiable), 2) type of the computation cost (0 fixed cost (most desirable). 1/2 bounded probabilistic cost, unbounded probabilistic cost), 3) interactive or not, 4) publicly auditable or not, 5) does the server have trapdoor in computing the function, 6) parallelizable
not. For our purpose, the required properties are 1) (with value 2 1/2) and 3). The efficiency of verifia- bility has be at least 1/2 create practical defense. The PoW-function for MLaaS has to be interactive 3) since we want to control the difficulty of the PoW-function based on how much information leakage is incurred by given user or globally after all answered queries. The remaining properties might be present or not. The HashCash function has unbounded probabilistic cost 2). It is not necessary for the PoW cost-function to be publicly auditable 4) if there is no conflict of interest and the service provider is not paid for more answered queries by an external 3-rd party, but directly by user who sends query. However, this property might be useful to verify how many queries a given server answered. It is not necessary but desirable that the server does not have trapdoor 5) (a shortcut in computing tokens) because it reduces complexity and administrable overhead (e.g.. a requirement of a periodic key replacement). The non- non-parallee parralleliizabl PoW cost-functions 6) are of questionable practical value. In general, the interactive HashCash has probabilistic cost (which in practice is acceptable) while not having the trapdoor, thus can be used as PoW cost-function for MLaaS.
32000
1000
Laainggo00
Figure 2: Binary HashCash.
With non- con-interactive PoW cost-functions, client chooses its own challenge. In our case, we use the interactive version, where server decides on the difficulty of the puzzle based on the information leakage incurred by a user. The puzzle in HashCash is to find a suffix that can be appended to challenge string generated by the server (in the interactive mode) so that the message digest has a specific number of leading zeros. The levels of difficulty in the original implementation are rather coarse-grained since the hexadecimal representation is used. For the required minimal 5 leading zeros (in hexadecimal notation), the computation time is usually < 1 sec, for 6 leading zeros it can increase to 20 sec more, and for 7 leading zeros it grows to even 511 seconds. Thus, we change the HashCash from comparing the leading zeros in the hexadecimal representation to the binary representation. One more zero in the hexadecimal format requires four zeros in the binary format. Hence, tracking zeros in the binary representation is more fine- tineggrined and allows us to better control how much time is spent to solve a puzzle. This enables a tighter management of time spent by a given user on the computation before we reveal the final label.
In Figure 2 we present the dependency between the computation time and the required number of leading zero bits for the binary HashCash (aggregated across 8 runs). The average work time required is exponential the number of zero bits and can be verified by executing a single hash (Nakamoto) 2008). The number of trials before user achieves success is modeled by a geometric distribution, which is the probability distribution over the number of required Bernoulli trials until the success. The expected number of trials for & leading zeros is 2* its variance is 2* 1), and approximate standard deviation is 2 (usually 2$ >>>>> 1). The official implementation is -on-interactiee, coarse- grained, and written in Python 2.3 (Back 2020). We extend the original version of the code and implement the HashCash cost-function, in both interactive and non-interactive modes, make it fine-grained, and port it to Python 3.8.
3.3 CALIBRATION OF PUZZLE DIFFICULTY
After specifying the proof-of-work method, we use to generate puzzles for a given user. We calibrate the difficulty of PoW puzzles which is characterized by the number of leading zeros in the message digest that needs to be returned by the client to demonstrate they solved the puzzle. Our calibration is based on the cost of each query: the cost should reflect the information leakage about the training data as result of the answers released for queries. Thus, we naturally turn to literature on reasoning about privacy in ML, and in particular differential privacy.
Evaluating the cost of queries with differential privacy. We define model information leakage as the amount of useful information user is able gain from set of queries run against a victim model.
5

---

Published as a conference paper at ICLR 2022
We use the information leakage based on the user's queries to inform our defense and ultimately help limit the amount of information that malicious users can gain. Since our defense works on a per user basis, it requires the identification of users. Our main method is to measure the information leakage based on the privacy budget incurred by a given user, which is known technique from the differential privacy literature (Dwork al., 2014). There are two canonical approaches obtaining privacy in deep learning: DP-SGD and PATE. DP-SGD is not applicable in our case as it is an algorithm level mechanism used during training, which does not measure privacy leakage for individual test queries. On the other hand, PATE is an output level mechanism, which allows us to measure per-query privacy leakage. PATE creates teacher models using the defender's 's training data and predictions from each teacher are aggregated into histogram where in indicates the number of teachers who voted for class The privacy cost computed based on the consensus among teachers and is smaller when teachers agree. The consensus is high if we have low probability that the label with maximum number of votes is not returned. This probability is expressed as: 1 Eigg erfe " ). where i' is an index of the correct class, erfe is the complementary error function, and is the privacy noise (see details in Appendix D). Then a student model is trained by transferring knowledge acquired by the teachers in a privacy-preseriin
way. This student model corresponds to the attacker's model trained during extraction. Papernot et al. (2017a) present the analysis (e.g., Table 2) of how the performance (in terms of accuracy) of the student model increases when a higher privacy loss is incurred through the answered queries.
From the privacy cost to the puzzle difficulty. The reference cost is for a standard user who is assumed to label in-distribution data that sent to the server in random order. The goal is to find
a mapping from a user's privacy cost to the number of leading zeros using the available data: (1) privacy cost, number of queries, and timing for legitimate users, and (2) timing to solve a puzzle for number of leading zeros. Based on the data from legitimate users, we create a model that predicts the total privacy cost from the number of queries to the privacy cost. This first model extrapolates to any number of queries. The more data on legitimate users we have access to, the more precise the specification of the dependency between the number of queries and the cumulative privacy cost. We interpolate the second dataset by creating a linear model that predicts the number of leading zeros for the desired time to solve the puzzle. We take the logarithm of the PoW time, which transforms this exponential dependency to linear one. To allow for the additional overhead of PoW, we set legitimate user's ' total timing to be up to 2x longer than the initial one. The query cost and timing are limited for legitimate user in comparison to attacker and so the models allow us to handle arbitrarily small or large privacy costs and timings.
Answer queries with PoW. When user sends new batch of queries for prediction, the first model maps from the user's query number to the expected accumulated query cost for a legitimate user. We take the absolute difference r between the actual total query cost of the user and the predicted legitimate query cost. If the costs for legitimate users differ from each other substantially then, instead of using difference privacy costs. Next, we map from the difference x to the timing for the PoW using, for example, an exponential function: fz) = a", where a is a parameter set by a model's owner. In our experiments 1.0075 so that the standard user's cost does not increase beyond 2x and the cost for the attackers maximized. A legitimate user should stay within a low regime of the difference and an attacker's difference should grow with more queries thus requesting more work within PoW. Then, we use the second model to map from the desired timing of PoW to the number of leading zero bits that have to be set for the puzzle. We find that using simple linear regression is sufficient for both models.
single baseline, we could define upper and lower bounds for the expected
4 EMPIRICAL EVALUATION
We evaluate our defense against different types of model extraction attacks. We show that our privacy- based cost metric is able to differentiate between the information leakage incurred by attackers vs legitimate users (Figure 3). The accuracy gain as measured on adversary's model against a victim's privacy loss assesses how efficiently our defense can counter a given attack. Our results show that no adversary can achieve substantial gain over user with legitimate queries (Figure 4 Finally, perform an end-to-end evaluation (Table 2). which indicates that run time for a legitimate user is only up to 2x higher whereas it is substantially increased (more than 100x) for most attackers.

---

Published as a conference paper at ICLR 2022
4.1 EXPERIMENTAL SETUP
We use MNIST, FashionMNIST (in the Appendix), SVHN, and CIFARIO datasets to test our defense. To evaluate the accuracy of the extracted model, subset of the corresponding test set is used, which does not overlap with an attacker's queries. In our experiments, for MNIST and FashionMNIST, we use victim model with the MnistNet architecture (shown B.2) and test set accuracies of 99.4% and 92.2% respectively. ResNet-34 with test set accuracies of 96.2% and 95.6% is used for SVHN and CIFARIO, IO, respectively. Next, we describe the model stealing attacks used in our experiments.
Jacobian and Macobian-TR (targeted) attacks use 150 samples from the respective test sets for the initial substitute training set. A value of 0.1 is used for the dataset augmentation step and the attacker model is trained for 20 epochs with learning rate of 0.01 and momentum of 0.9 after each augmentation step.
DataFree is run in the same way and with the same parameters as in the original implementation from Truong et al. (202I). Specifically for MNIST and SVHN, we use a query budget of 2M while for CI,ARIO, we use budget of 20M. The attacker's 's model is trained using ResNet- 18 in all cases.
Entropy (Active Learning) based attack uses an iterative training process where 1000 queries are made based on the entropy selection method. This is followed by the attacker model being trained for 50 epochs with learning rate of 0.01 and momentum of 0.9 on all queries that have been asked till that point. We then repeat this process.
MixMatch based attack queries a victim model using examples randomly sampled from a test set. Active learning approaches were tried but had negligible effect on the accuracy of the stolen model. After the labeling, the same number of additional unlabeled samples are taken from the test set. The attacker's model is trained using Wide-ResNet-28 (Oliver et 2018) for all datasets and using 128 epochs. The other hyperparameters are kept the same as in the original paper (Berthelot et al. 2019).
Knockoff queries model from out-of-distribution dataset, in our case it is Imagenet (Deng et al. 2009) dataset for MNIST, while SVHN and CIFAR100 are used for CIFAR1O. We use the Random and Adaptive querying strategies in our experiments. After the querying process has been completed, an attacker model is trained for 100 epochs with a learning rate of 0.01 and momentum of 0.9.
CopyCat considers two cases, namely where the attacker has access to problem domain data and where the attacker has access to non-problem domain data (Correia-Silvaa et al.) (018). We only consider the case where the attacker has access to non-problem domain data (in this case ImageNet) as the other case is the same method as our standard (random) querying.
EntropyRev and WorstCase are adaptive attacks designed to stress test our defense.
Since the attacks considered originally had highly different setups, we made adjustments to unify them. For example, for MixMatch, we use test sets select queries as opposed to train sets because other attacks assume no access to the train sets. For the active learning attacks, or Jacobian approaches, the number of epochs and hyperparameters used for training are kept the same. We train the models starting from random initialization.
4.2 INFORMATION LEAKAGE
We compare the privacy cost measured on the victim model for different attacks. To compute the privacy cost, we use PATE (Papernot et al. (018)) with 250 teachers for MNIST and SVHN, and 50 teachers for CIFAR10. The scale of the Gaussian noise set to for MNIST and SVHN, and to for CIFAR10. Results for Standard query selection represent legitimate user. From Figure we find that the privacy metric correctly shows higher values for the attackers compared to a Standard user. Privacy cost increases much more for DataFree, CopyCat, and Knockoff attacks that use OOD data, compared to the attacks that use in-distribution data. The Jacobian attack starts from a small seed of in-distribution samples and then perturbs them using the Jacobian of the loss function and therefore shows lower cost values. However, despite its lower privacy cost, the Jacobian attack achieves much lower accuracy of the extracted model than other attacks (Figure
7

---

Published as a conference paper at ICLR 2022
MNIST
SVHN
CIFAR1D
Data Free
7000
7000
Standard/Mishathh
35000
6000
6000
EntropyRev
20000
Privacy
WorstCase
Jacobian
4000
4000
Knoykort/CopyCat
20000
3000
3000
15000
1000
1000
5000
Querles
Querles
Querles
Figure 3: Privacy Cost vs Number of Queries. We measure the privacy cost against the number of queries for various query selection strategies the MNIST, SVHN and CIFAR10 datasets.
100
80
Data Free
Stainderd
70
Accuracy
EntropyRey
60
WorstCase
Jecobian
40
40
CopyCat
40
KrockoffRandom
20
20
MixMatch
KrockoftAdaptive
10
2000
4003
5000
8000
15000
2000
4000
6300
8003
10000
5000 12000
13000 20000 25000 20200 Privacy
Privacy
Privacy
Figure 4: Accuracy vs Privacy Cost. We measure the accuracy against the privacy cost for various query selection strategies the MNIST, SVHN and CIFARIO datasets. An attacker cannot achieve better tradeoff than user with in-distribution data.
4.3 PROOF-OF- SROOF-OF-WORK AGAINST ATTACKS
The execution time for legitimate user is longer by less than 2x after our defense is applied while the execution times of Knockoff and DataFree attacks are up to 100x longer (see Table and Figure The 2x longer execution is chosen as an acceptable difference for legitimate users that makes the attacks significantly more expensive. The cost of execution the DataFree attack totally dominated by PoW due to its high privacy cost incurred by many queries with almost random content. This is because this attack has no access to in distribution data.
In fact, the results for MixMatch in Figure 4 and Table show how valuable is the access to in- distribution data combined with semi-supervised learning. The cost of querying the victim model much lower while the task accuracy of the stolen model is substantially higher than for other attacks. Nevertheless, the MixMatch extraction requires unlabeled in-distribution samples, which might not be feasible to collect for the adversary.
For attacks consisting of synthetic queries, such as DataFree, even if an attacker attempts to reduce their cost, the usage of the synthetic data, which out-of-distribution. causes metrics to increase much faster than for legitimate users and thus this approach is not able to bypass our defense. Furthermore, trying minimize the cost for attack involving synthetic data requires the attacker perform more computation because there is no set of examples from which they can choose to query and so instead a larger set of possible queries needs be generated.
4.4 ADAPTIVE ATTACKS
We design and evaluate attackers who are aware of our defense and actively try to adapt to it. Their goal is to minimize the privacy cost calculated by the victim while maximizing the accuracy of the stolen model. EntropyRee Rev attack uses entropy to approximate the cost computed by victim and selects queries with minimal instead of maximal entropy, which is a reversed strategy to the entropy-based active learning attack. The achieved entropy cost (and also the privacy and gap) is
8

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of the attacks against the PoW-based defense. The notation: IND- in- distribution, SYN- Symthetic. IND denotes limited in-distribution data with additional data generated using the Jacobian augmentation. MixMatch uses as many additional unlabeled training examples for the semi-supervised training as the number of answered queries. The results for Standard, EntropyRev, and WorstCase are the same so we report only Standard. Time PoW is after applying our defense.
DATASET
ATTACK
DATA
# QUERIES
PRIVACY PATE
TIME (SFC)
TIME PoW
ACCURACY
FIDELITY
TYPE
(%)
(%)
MNIST
MIXMATCH
IND
SK-
2047
58.9
67.1
99.5
99.6
MNIST
STANDARD
IND
10K
2576
73.6
98.4
98.7
MNIST
KNOCKOFFRANDOM
IMAGENET
10K
5301
73.6
21283
98.7
99.0
MNIST
KNOCKOITADAPTIVE
IMAGNNT
5251
73.6
21235
89.7
90.0
MNIST
COPYCAT
IMAGENET
10K
5301
73.6
21283
92.7
92.9
MNIST
JACOBIAN
IND
10K
2421.6
73.6
91.1
81.4
81.5
SVHN
MIXMATCH
97.4
SVHN
STANDARD
IND
24K
7918
833.3
859,4
95.6
95.3
SVHN
KNOCKOFFRANDO
IMAGENET
24K
15656
833.3
33413
92.5
92.5
SVHN
KNOCKOFAADAPIVE
IMAGENET
24K
15195
833.3
33372
43,9
44.7
SVHN
COPYCAT
IMAGENET
24K
15636
833.3
33413
87.6
87.8
SVHN
JACOBIAN
IND
K
8763
833.3
6424
26.0
26.6
SVIN
DATAFREE
SYN
2M
1.52E6
68355
330769
95.3
97.2
CIFARIO
MIXMATCH
IND
SK
22378
131.8
163
91.7
92.2
CIPARIO
STANDARD
IND
9K
25147
148.3
182.4
65.7
66.0
CIFARIO
KNOCKOWRANDOM
CIFARI00
148.3
23921
CI
KNOCKOP ADAPTIVE
CIFAR 100
9K
27927
148.3
23528
32,0
32.3
CI
COPYCAT
CIFARI00
29584
148.3
23921
54.2
54.8
CIFARIO
IND
CIFARIO
KNOCKOFF
CIFARIOO
30K
98355
494.4
86185
87.8
88.7
CIFARIO
COPYCAT
CIFARIOO
30K
98355
494.4
86185
78.1
77.7
CIFARIO
DATAFREE
SYN
20M
7.14E7
3,22E5
2.9556
87.15
88.0
lower than for a standard (random) querying, however, the accuracy the stolen model also drops in most cases (Figure Appendix). WorstCase attack uses in di-sstriibution data (we also test it on OOD data Figure and has access to (or can infer) the exact victim's 's privacy cost to minimize the cost of adversarial queries. Privacy cost does not differ much between in-distribution queries this attacker cannot game our system.
A more realistic scenario of powerful adaptive attacker is that a victim also releases scores (e.g.. softmax outputs) that are used to obtain an estimation of victim's 's privacy cost. However, the estimation of the cost is known only after the query is answered. thus it cannot be used by the attacker to order the queries sent for prediction. Another adaptive adversary can attack our operation model with multiple fake accounts instead of funneling all their queries through single account there can be multiple colluding users. Our approach not only defends against this because the attacker still needs to solve the proof-of-work for each of the of identifying colluding users. We note that one appealing aspect of our defense is that relies on the privacy cost that can easily be summed across users because of the composition property of differential privacy. We leave the implementation and assessment of this idea for future work.
accounts, but also simplifies the task
5 CONCLUSIONS
Model extraction is a threat to the confidentiality of machine learning models and can be used as reconnaissance prior to mounting other attacks. It can leak private datasets that are expensive to produce and contain sensitive information. We propose the first pro-active defense against model stealing attacks that leaves the model's accuracy entirely intact. We wrap original model predictions in proof-of-wood -ork (PoW) to force users to expend additional compute solving PoW puzzle before they can read the output predicted by the model. To minimize the impact on legitimate users, we tie the difficulty of the puzzle to an estimate of information leakage per user's account. We show that the information leakage incurred by users' queries as measured by the privacy cost using PATE reliable indicator of which user tries to extract more information from model than expected. We thus believe that our approach can be successfully applied by practitioners to increase the security of models which are already publicly deployed.

---

Published as a conference paper at ICLR 2022
6 ETHICS STATEMENT
One possible ethical concern regarding our defense is that the application of Proof-of-Work (PoW) requires additional computation, which could increase energy usage and thus also the CO2 emission. However, since the increase in computation is very little for benign user, the effect of PoW will be minimal whereas for an attacker, extra compute to train a model, often also involving the use of GPUs, is required and this outweighs any effect of the computation required for our PoW based defense.
To reduce the energy wastage, approach be adapted to rely PoET (Proof-of-Elapse--Tnge
for example, as implemented by Hyperledger Sawtooth] instead of PoW. The users' resource (e.g.. CPU) would have be occupied for specific amount of time without executing any work. At the end of the waiting time, the user sends proof of the elapsed time instead of proof of work. Both proofs can easily verified. PoET reduces the potential energy wastage in comparison with PoW but requires access to a specialized hardware, namely new secure CPU instructions that are becoming widely available in consumer and enterprise processors. If user does not have such hardware on premise, the proof of elapsed time could be produced using a service exposed by cloud provider (e.g., Azure VMs that feature TEE
In our defense, legitimate user's cost (2X) is not prohibitive but also non-negligible. Previous work (Orekondy et al. 2020 Kariyappa & Qureshi 2020) and our results suggest that there is no free lunch the active defenses against model extraction either sacrifice the accuracy of the returned answers or force users to do more work. The principal strength our active defense approach is that it does not sacrifice accuracy, which comes at the expense of computational overhead.
Another aspect is that users with more powerful hardware (c.g., CPUs with a higher frequency) could compute the puzzle from PoW faster than other users with less performant chips. We note that this is directly related to the design of PoW cost-functions. For example, HashCash is difficult to parallelize, however, [Back] (2002) argues that protection provided by non-parallizable PoW cost-functions is marginal since an adversary can farm out multiple challenges as easily as farm out a sub-divided single challenge.
7 REPRODUCIBILITY STATEMENT
We submit code in the supplementary material. In the README. file we provide the main commands needed to run our code. We also describe the pointers to the crucial parts of the code that directly reflect the implementation described in the main part of the submission. We describe the attacks used in our experiments thoroughly in the Appendix Section A.I
ACKNOWLEDGMENTS
We would like acknowledge our sponsors, who support our research with financial and in-kind contributions: DARPA through the GARD project, Microsoft, Intel, CIFAR through the Canada CIFAR AI Chair and AI catalyst programs, NFRF through an Exploration grant, and NSERC COHESA Strategic Alliance. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring Finally, we would like to thank members of
the Vector Institute https://eecorrisiiuteeaaaapppppp CleverHans Lab for their feedback.
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into strength: Watermarking deep neural networks by backdooring, In 27th USENIX Security Symposium (USENIX Security 18), pp. 1615-1631. Baltimore, MD, August 2018. USENIX Association. ISBN 978- 1-1-9999333--- 000-939913334---- URL htw.hhttss...
isenixe https//iiii.ggg/ooooffeeeennnne.
usen b
Azure VMs that feature TEE: http:///ii.ly/333BBrr
10

---

Published as a conference paper at ICLR 2022
Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, and N Asokan. Extraction of complex dnn models: Real threat or boogeyman? In International Workshop on Engineering Dependable and Secure Machine Learning Systems, pp- 42-57. Springer, 2020.
Adam Back. Hashcash A Denial of Service Counter-Measure 2002.
Adam Back. Hashcash.
2020. [Online; accessed 8-July-2021]
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šmdićć Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Glenn W Brier et al. Verification of forecasts expressed in terms probability. Monthly weather review, 78(1):1-3, 1950.
Varun Chandrasckaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan. Ex- ploring connections between active learning and model extraction. In 29th USENIX Security Sym- posium (USENIX Security 20), pp. 1309-1326, USENIX Association, August 2020. ISBN 978-1- 939133-17-5. URL https::///cc.. www usenix.en.......... wttp://wwww.....ooooo./.eeeeeeaa
presentation//haannn hwt.aaaii...o/aaaaaaaaaaaaaaa
Jacson Correia-Silva, Rodrigo Berriel, Claudine Badue, Alberto De Souza, and Thiago Oliveira- Santos. Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data. 06 2018.
Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami Beach, Florida, USA, June 2009.
Cynthia Dwork and Moni Naor. Pricing via processing or combatting junk mail. In Ernest F. Brickell (ed.), Advances in Cryptology - CRYPTO' 92, pp. 139-147, Berlin, Heidelberg, 1993. Springer Berlin Heidelberg. ISBN 978-3-540-48071-6.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory cryplography conference, pp. 265-284. Springer, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations differential privacy. Foundations and Trends in Theoretical Computer Science, (33-4):111-40,, 2014.
Yen- Chn-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from distrri-itiittoonn data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Hyperledger Sawtooth. PoET (Proof-of-Elapsee--imee http://mmennntttss://sswttttott http://wwwtttthhh.ypeeeeeeeegg 19-Septemmeer22211.
[Online; accessed
Matthew Jagielski and Nicolas Papernot. In Model Extraction, Don't Just Ask 'How?': Ask 'Why?'. http::/ww// www.wehaweeee.......e. is httpa//www.....e///////////aaaaa 2020. [Online; accessed 4-July-2020J.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papemnt. High accuracy and high fidelity extraction of neural networks. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1345-1362, 2020.
11

---

Published as a conference paper at ICLR 2022
Hengrui Jia, Christopher A Prooquette-Choo and Nicolas Papernot. Entangled watermarks defense against model extraction. arXiv preprint arXiv:2002. 12200, 2020a.
Hengrui Jia, Christopher A. Choquette-Choo, and Nicolas Papernot, Entangled watermarks as defense against model extraction. CoRR, http/2002.122000. 2020b. htlpttsss//axiv.oor
abs/2002. 12200
Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. arXiv preprint arXhv:2103.05633, 2021.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 312-527. IEEE, 2019.
Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adaptive misinformation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 770-778, 2020.
Sanjay Kariyappa, Atul Prakash, and Moinuddin K Qureshi. Protecting [dnn]s from theft using an ensemble of diverse models. In International Conference on Learning Representations, 2021.
URL httpps://ppeeeeviiiittee net htte//ooormmdd-nacceee
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit lyyer. Thieves on sesame street! model extraction of bert-based apis. In International Conference on Learning Representations, 2020. httsstss:: http///pwrrrreeeeeeeeero/rrrrraa
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against neural network model stealing attacks using deceptive perturbations. In 20/9 IEEE Security and Privacy Workshops (SPW), pp. 43-49, 2019. doi: 10. 1111///W..2019.00222
Daniel Lowd and Christopher Meek. Adversarial learning. In Proceedings of the Eleventh ACM SIGKDD International Conference Knowledge Discovery in Data Mining, KDD '05, pp. 641-647, New York, NY, USA, 2005, Association for Computing Machinery. ISBN 1595931355 135X. doi: 10.1145//011870.11 1081950. URL
1tt.://.org//11111//1111111 100.01115/11111//00 001111777111110000 1081950
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot, Dataset inference: Ownership resolution in machine learning. In Proceedings of ICLR 2021: 9th International Conference on Learning Representationsn, 2021.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275, IEEE, 2017.
Takayuki Miura, Satoshi Hasegawa, and Toshiki Shibahara. Megex: Data-free model extraction attack against gradient-basee explainable ai. 2021.
08909
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Satoshi Nakamoto. Bitcoin: A peer-to-peer electronic cash system. Decentralized Business Review, pp. 21260, 2008
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In S. Bengio, H. Wallach, H. Larochelle, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL httpttppsss/ppoceeeinngggegeees neurips
K. Grauman, N.
cc/ cc/paaper/018/ B/tiB//fll////// fea c44 8080 7 9w 72020000020000000226266662222223
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4954 4963, 2019.
12

---

Published as a conference paper at ICLR 2022
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stealing attacks. In International Conference on Learning Representations. 2020. URL
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy, A framework for the extraction of deep neural networks by leveraging public data. arXiv preprint 15t::1995009165. 2019.
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. Activethief: Model extraction using active learning and unannotated public data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 865-872, 2020.
Soham Pal, Yash Gupta, Aditya Kanade, and Shirish K. Shevade. Stateful detection of model extraction attacks. CoRR, abs/2107.05166, 2021. URL httpthtttss/si..ccocm
05166
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Confer- ence Track Proceedings, 2017a.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the ACM on Asia conference on computer and communications security, pp. 506-519, 2017b.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 17X.v 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6/ 2014.
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N Asokan. Dawn: Dynamic adversarial watermarking of neural networks. arXiv preprint arXXiv//906 99930,083,, 2019.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th {USENIX] Security Symposium ((USENIX) Security 16), pp. 601-618, 2016.
Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papernot. Data-free model extrac- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Santiago Zanella- Zanella-Beguelin, Shruti Tople, Andrew Paverd, and Boris Köpf. Grey-box extraction of natural language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. preas/v139/zanel] www.lllbbbbeeeaaee i2lla .tmll
PMLR, 18-24 Jul 2021. URL httpp:::pprceeeiggs... mll
Xinyi Zhang, Chengfang Fang, and Jie Shi. Thief, beware of what get you there: Towards under- standing model extraction attack. arXiv preprint 10Xivv2004.05921, 2021.
Yuqing Zhu, Xiang Yu, Manmohan Chandraker, and Yu-Xiang Wang. Private-knn: Practical differen- tial privacy for computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

Published as a conference paper at ICLR 2022
Table 3: Comparison of the attacks against the PoW-based defense. (Extended results from 2) We present the number of required queries, gap, entropy, privacy cost PKNN (as computed for alternative metrics from Appendix [B6), time before (Time) and after PoW (Time PoW), and accuracy (ACC) for different datasets and attacks. The results are aligned according to the highest aceuracy achieved by the attacks.
DATASET
ATTACK
DATA
# OF QUERIES
GAP
ENTROPY
PRIVACY PKNN
TIME (SEC)
TIME PoW
Ace
TYPE
(%)
MNIST
MIXMATCH
IND
100
0.17
0.19
0.36
0.69
0.97
98.0
MNIST
MIXMATCH
IND
IK
4.51
2.8
1.01
8.61
98.9
4K
MNIST
8K
46.3
MNIST
STANDARD
IND
10K
59.7
37.5
4.58
73.4
88.92
98.4
IMAGENET
45.99
MNIST
COPYCAT
IMAGENET
10K
4539
4367
50
73,4
42513
93.2
MNIST
JACOBIAN
IND
10K
41.9
46.2
3.46
73.4
88.9
82.4
1747972
1915677
MNIST
ENTROPYREV
IMAGENET
10K
3750
3295
62.2
73.4
58078
91.03
MNIST
WORSTCASE
IMAGENET
10K
444.6
704
73.4
24474
95.3
SVHN
MIXMATCH
IND
250
8.95
10.45
0.96
2.19
2.64
90.8/95.88
SVHN
MIXMATCH
30.3
35.9
8/96.87
SVHN
MIXMATCH
IND
4K
138.4
156.5
97.3/97.07*
SVHN
MIXMATCH
IND
8K
280.1
314.9
6.39
82.5
97.2
SVHN
STANDARD
IND
25K
868
975
15.3
218.9
257.4
94.7
SVHN
IMAGENET
SVIN
COPYCAT
IMAGENET
25K
10594
10918
168
218.9
247506
87.6
SVHN
JACOBIAN
IND
25K
1411
1429
23.0
218.9
3121
44.6
SVHN
795382
SVHN
ENTROPYREV
IMAGENET
25K
10467
10771
171.4
218.9
224868
86.3
WORSTCASE
IMAGENET
5789
6907
26.1
254.1
CIFARIO
MIXMATCH
7.41
2.24
70,4/87
CIFARIO
IK
CI
MIXMATCH
IND
4K3
118.23
89.33
3.44
28.8
35.25
89. 1/9..99.
CIFARIO
MIXMATCH
IND
8K
231.88
178.40
5.25
58.8
70
91.7
CIFARIO
STANDARD
IND +TINYIMAGES
30K
704.9
588.3
15.9
215.6
262.5
89.6
CI
KNOCKOFF
CIFARI00
30K
6889
5656
146
215.6
290377
87.8
CIPARIO
COPYCAT
CIFARI00
30K
6889
5656
146
215.6
290377
75
CI
JACORIAN
IND
10K
1060
783
23.1
71.9
2165
28.2
CIFARIO
DATAFREB
SYN
20M
7,65E8
6.84B6
146540
143612
2.29E11
87.15
CIFARIO
ENTROPYREV
CIFARI00
30K
7003
5675
148.9
213.6
290672
70.9
CIFARIO
Table 4: Taxonomy of model extraction defenses.
PRADA
PASSIVE
COMPARE DISTRIBUTIONS
STATEFUL
PRESERVED
VARDETECT
PASSIVE
COMPARE DISTRIBUTIONS
STATEFUL
PRESERVED
STATELESS
ADAPTIVE MISINFORMATION
ACTIVE
PERTURE OUTPUTS
STATELESS
BOUNDED
WATERMARKINO
REACTIVE
SECRET
DATASCT INCERENCE
REACTIVE
RESOLVE OWNERSHIP
N/A
N/A
PROOF OF LEARNING
REACTIVE
TRAINING LOGS
N/A
N/A
PoW
PRO-ACTIVE
POW PATE
STATEFUL
PRESERVED
A MODEL EXTRACTION ATTACKS
A.1 DETAILED DESCRIPTION ATTACKS USED
The latest attacks that we implement and check against using methods follows:
1. Jacobian (Jacobian matrix based Dataset Augmentation), which was originally proposed by Pa- permot et al. (2017b), is an attack that uses synthetic data to generate queries. The substitute training process involves querying for the labels in the substitute training set Sp, training model and then using augmentation on the substitute training set S,, to generate Sp+1. For each sample my E Sp, generate new synthetic sample xy Sp+1 by using the sign of the substitute
14

---

Published as a conference paper at ICLR 2022
MWST
CIRAR10
Accurocy
Aycurncy
standard
1080
4000 S000
100
800
20W
1924
5000
8003
5000
2000 SE0
4003
7603
500
Queriies
Querles
Queries
NMBST
SWHI
CFA430
336
esthppyye
Entropp
Querfes
Queries
Querles
NMB1I
SWHN
CPA-30
Enbopy
S103Py
1350 200
1383 258
1003 1000
1.00
3003 4230
entrpy
erbopy enEappew
intrippeer
Aoniicais Jocooinn
Jasitiin
Pitracy
15000
3300
383
283
7510
1000 1510 940
2005 FR2E Querles
0095
3001
1923 4010 1103 ADD 7001 K52E 5000 Querles
1090
2010
9030 4.00 1100 PR2E 0009
Dueelies
NWST
CIAA3O
Acouray
ertrgg stavtedd
aibopy entopper
1520 7018 2003
2002
2001
1500
1000 1100 1900 28m
vire weo 4000
Figure 5: Adaptive attacks using in-distribution data.. We observe a tradeoff between attackers' goal to bypass defense and the resulting accuracy they achieve. Accuracy, Entropy, Gap. and Privacy costs of the victim model vs number of Queries and Accuracy vs Privacy for MNIST, SVHN and CIFARI0 datasets. We note that for the Accuracy Privacy graph, the final privacy cost of all methods is the same. This is because at the end of the querying process all methods have used all the available samples.
DNN's Jacobian matrix dimension JF(x) corresponding to the label assigned to input æ; by the oracle O.: z = x; + A (gn(J]n(n(([ (Oxz))) Note that the set Sp+1 is the union of Sp and the new points generated. These steps then repeated. The aim of the attack is to identify directions in which the model's output is varying around an initial set of points so that the decision boundary be approximated. Jacobian is used as model extraction attack, however, it was designed
15

---

Published as a conference paper at ICLR 2022
MAIST
SU4N
CFAA30
ACCurocy
Cosyrot
Duta Froe
Dara Fioe
inscouff
2038
10:0
3000
13:03
2000
25000
1508
2000
1800
Quertes
Querles
Queries
1503
HAAST
1.03
Sy-1
3000
CI-A811
190
Steccuff
0000
010/000
5000
6200
5000
Inedeff
Ineccof
erbopprce
2900
scriccas
Queries
Querfes
Queries
1.03
NMS51
7543
Shrt
6000
CI-AR:0
1900
4.03
1103
360
2003
Lesyol
kneckef
1303
Data Free
2500
683
3000
1250
1908
21000
118
1990
3303
3800
Queries
copyott
copysat
Brockaff
bnoccof
lnaccuff
worilcas
Data Froe
Daca Free
Cata Free
1.89
2003
3303
2009
3000
6103
I000
3005
Y900
125
10014
1990
20303
2000
12:0
12:00
10002
2000
2900
Querries
Querties
Querress
WAST
SV
CFAR30
kredeff
Ineccuff
antrppreey
Accua
extippeite
2502
19901 12N0E
19000 11000 2000
250 2000 1800
1906
1100 2500
3000
200
10000
1200
Figure 6: Adaptive attacks using out-of-distributtin data. Accuracy, Entropy, Gap, and Privacy of the victim model number of Queries and Accuracy Privacy for the MNIST, SVHN and CIFARIO 10 datasets.
to optimize for the transferability of adversarial examples between models, which is different objective. This is called a reconnaissance motivvated adversary who is more interested a high delity extraction reconstruction whereas deet-mootvvtted adversary optimizes for high accuracy extraction (Jagielski & Papernot 2020).
Jacobian-TR variation of Jacobian proposed by Juuti et al. 2019). Here a new sample x/ Sp+1 is generated from existing sample I; Sp using the sign of the Jacobian matrix dimension for a random target class as: xf = 2; X sgn (V()), where yr is a random class not equal the oracle's prediction O(x:).
16

---

Published as a conference paper at ICLR 2022
3. MixMatch is semi semi-supervised learning approach, which was originally proposed by Berthelot et al. (219)) and applied by Jagielski et al. (2020) for high fidelity and accuracy extraction. This model extraction method shows that is possible to steal model even with small number of labeled examples (Jagielski et al. 2020). This attack uses an initial set of labeled data points along with an equally sized set of unlabeled examples. The algorithm produces processed sample-label pairs for both the labeled and unlabeled data points through steps involving data augmentation (random horizontal flips and crops) that leave class semanties unaffected, label guessing (average prediction over augmentations), sharpening (to produce lower entropy predictions), and Mix Up (a linear interpolation of two examples and their labels). The ablation study indicates that MixUp (especially unlabeled points) and sharpening are the most crucial steps. This model trained using a standard semi-supervised loss for the processed batches, which is a linear combination of the cross-entropy loss used for labeled examples and Brier's score (squared lg loss on predictions) (Brier et al. 1950) for the unlabeled examples.
4. DataFree ree (Data Free Model Extraction) is an attack proposed by Truong et al. (2021) that does not require surrogate dataset and instead uses a strategy inspired by Generative Adversarial Networks (GANs) to generate queries. A generator model creates input images that are difficult to classify while the student model serves discriminator whose goal during training is to match the predictions of the victim (oracle) on these images, The student and generator act as adversaries, which respectively try to minimize and maximize the disagreement between the student and the victim. DataFree assumes access to the softmax values from the victim model and requires gradient approximation train the generator, thus many queries are needed to be issued, e.g., in the order of millions when attacking models trained on SVHN or CIFAR10 10 datasets.
5. AI (Active Learning) methods for model extraction help the attacker to select set of points from a pool of in-distribution unlabeled examples. The active learning methods serve as heuristics that are used to compute scores for queries based on the trained substitute model (a.k.a. stolen model). The attacker uses obtained labels to improve the substitute model and can iteratively repeat the selection and query process. AL methods assume that an unlabeled in-distribution dataset d and classification model with conditional label distribution Po x:) are provided. We implement the following heuristies from the active learning literature, particularly from Pal al. (2019) :
Entropy: select points which maximize the entropy measure:
x² = arg max Σ; (Bx x²) log (M z), where y! ranges over all possible labels.
Gap: select points with smallest gap in probability of the two most probable classes:
x² = arg min (Dn x)- - Pa( æ²). where ý and J are respectively the most and second
most probable classes for xr according to the currently extracted model.
Greedy: run k-nearest neighbour algorithm by selecting labeled points as cluster centers that minimize the furthest distance from these cluster centers to any data point. Formally, this goal is defined as ming; SUDISk max; minjesup A(x, Xy), where D is the current training set and S is our new chosen center points. This definition can can be solved greedily as shown in (Sener & Savarese 2017).
DeepFool: select data points, using the DeepFool attack proposed by (Moosavi-Deziooli et al., 2016), that require the smallest adversarial perturbation to fool the currently extracted model. Intuitively, these examples are the closest ones to decision boundaries. The victim model is queried with such unperturbed examples.
The attacks that use active learning methods include ActiveThief (Pal et al. 2020 and model extraction with active learning (Chandrasekaran et al. 2020).
6. Knockoff Nets is two step approach model functionality stealing proposed by Orekondy ct al.] 2 The first step involves querying set of input images to obtain predictions (sofumax output), followed by the second step where knockoff (illegal copy of the original model) is trained with the queried namely a Random strategy where the attacker randomly samples queries (without replacement) from different distribution compared to the victim's 's training set and novel Adaptive strategy where policy is learnt based on reinforcement learning methods (e.g., gradient bandit algorithm) to improve the sample efficiency of queries and aid the interpretability of the black-box victim model.
pairs. Two strategies are presented for the querying process,
7. CopyCat CNN is a method proposed by Cooreia--iile Silva et al. (011)) which aims to extract a model using random non labeled data. This attack also consists of two steps where a fake dataset is first.
17

---

Published as a conference paper at ICLR 2022
generated using random queries to the model which is then followed by CopyCat training. The paper considers two cases, namely where the attacker has access to problem domain data and where the attacker has access attacker has access to non problem domain data (in this case ImageNet) as the other case the same method our standard (random) querying.
problem domain data. We only consider the case where the
Another attack related to the above methods is EAT which we briefly describe below:
1. Extended Adaptive Training (EAT) is an extraction strategy applied by Chandrasckaran et al. (2020) that relies on active learning methods. The training of attacker's 's model similar the approach we use for our active learning methods with an iterative process that retrains a model on the newest generated points. The exact strategy used for the SVM-based models is active selection. This approach selects to query the point that is the closest to decision boundary.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 DETAILS ON THE EXPERIMENTAL. SETUP
Our experiments were performed on machines with Intel@Xcon@Sivver 4210 processor, 128 GB of RAM, and four NVIDIA GeForce RTX 2080 graphics cards, running Ubuntu 18.04.
B.2 MNISTNET ARCHITECTURE
The architecture used for MNIST dataset consists of two convolutional layers followed by two fully connected layers. Below is the detailed list of layers in the architecture used (generated using torchsummnry).
LeNet style architecture for MNIST:
Layer type
Output Shape
Param
Conv2d-1
[-1, 20, 24, 24]
520
ReLU-2
MaxPoo12d-3
Conv2d-4
[-1, 50, 8, 8]
25, 050
ReLU-5
MaxPoo12d-6
Linear-7
[-1, 500]
400, 500
ReLU-8
Linear-9
[-1, 10]
5, 010
Total params: 431, 080
Trainable params: 080
Non-trainable params:
Input size MB: 0.00
Forward/backwarc pass size MB: 0.12
Params size MB: 1.64
Estimated Total Size MB: 1.76
B.3 PROOF OF WORK
We present the accuracy vs timing of attacks Figures and g We also include similar graphs when using PkNN to compute the privacy cost and these graphs shown in Figures 9 and 10
18

---

Published as a conference paper at ICLR 2022
CIFAR10
90
80
70
Accuracy(%o
60
mode
standard
50
standard pow copycat
40
copycat pow
30
jacobian jacobian pow data free
20
10
data free pow
100
100
1002
1003
Time (sec)
Figure 7: PoW against model extraction attacks for the CIFAR10 dataset.
B.4 SYNTHETIC + IN IN-DISTRIBUTION DATA FOR ATTACKS
Apart from attackers who try to minimize their costs to avoid high computational cost of PoW, consider attackers who call this attack InOut. Here an attacker can first query with out-of-distribution methods and then have intervals where distribution data (low in quantity) is used to prevent the privacy continually increasing at much higher rate than a normal user. The usage of both in distribution and out of distribution can also help speed up the extraction process. In Figure II we compare this method with CopyCat CNN which randomly queries only using out-of-distriban distribution data and random querying which only uses in distribution data. We observe that the method which targets our attack is able to achieve a lower privacy cost than CopyCat CNN which uses OOD data while also being able reach higher accuracy, but still significantly worse than the random method.
mixture of in-distribution and out-of-distribution data. We
from
B.5 PATE PRIVACY COST VS PKNN COST
For the privacy cost, we also consider privacy accounting in the case when there are multiple models behind the ML API which are then used to both make the predictions returned to the model and to compute the privacy. Figure shows the experimental results for multiple models behind the API. We observe that the number of models in the ensemble increases, the effectiveness of the worstcase method also decreases which is likely because with more models behind the API, the difference the privacy costs of queries also decreases and therefore attacker selecting queries with the minimum privacy cost is not able to achieve a better accuracy privacy tradcoff.
In the case of single model behind ML API, we can also a form of PATE proposed by Zhu et al. (202)). First, it computes the representations from training data points obtained the last (softmax) layer of neural network. Second, for each new data point to be labeled, its representation found in the last layer. Finally, in the representation space, we search for the nearest neighbours from the training set. These nearest points become the teachers whose votes their ground-truth
19

---

Published as a conference paper at ICLR 2022
MNIST
100
80
Accuracy(%o)
mode
standard standard pow copycat copycat pow jacobian jacobian pow data free data free pow
60
40
20
100
1002
1002
Time (sec)
SVHN
80
Accuracy(%)
mode
60
standard
standard pow
copycat copycat pow jacobian jacobian pow data free data free pow
40
20
1000
1001
1002
Time (sec)
Figure 8: PoW against model extraction attacks for MNIST and SVHN datasets.
labels. The remaining steps the in the standard PATE. Similar results obtained when
20

---

Published as a conference paper at ICLR 2022
CIFAR10
90
80
mode
70
standard
Accuracy(%o)
standard pow
60
knockoff
50
knockoff pow data free data free pow entropyrev entropyrev pow worstcase worstcase pow
40
30
20
10
100
100
1002
1003
1004
100
Time (sec)
Figure 9: PoW against model extraction attacks for the CIFARIO dataset using the PkNN based privacy cost.
using this method with the main difference being that larger value of the parameter a is needed as result of the lower individual privacy costs and thus lower differences obtained with the pkNN cost.
B.6 OTHER QUERY COST METRICS
For completeness, we also harness metrics from information theory and --iirii-iittiittioonn detection methods. The entropy and gap metrics give similar estimations of the information leakage and they correspond directly to the entropy and gap methods used for the active learning types of attacks (as described Section 5 We also use the scores for queries from Generalized ODIN (Hsu et 2020). which is one of the state-of- satte-of-theaan fi-thh--rt out-of- -uittf-diisriuution detectors. Note that using the number of queries only instead of the other metries ineffective since after the same number of queries, we observe that attacks such Knockoff (Orekondy et al. 2019), which query victim model using out-of aut-of-distribution data, can extract model of the same accuracy as legitimate user querying with in-distribution data. However, Knockoff incurs much higher privacy cost than legitimate user. Note that the entropy or gap can be computed using the probability output for a query while the computation of privacy requires access to the whole training set, which makes the privacy metric more secure.
B.7 ACTIVE LEARNING
This section shows the results of running the active learning methods described in Section 5 on the MNIST, SVHN and CIFAR1O datasets. An iterative training process whereby the attacker model retrained
queries is used for the active learning methods whereas for random querying, all queries were done at once after which the model was trained. 50 epochs were used for training MNIST and SVHN models while 200 epochs were used for training CIFAR1O 10 models.
The active learning experiments for the entropy and gap metrics were also replicated on the imagenet dataset (Figure 14 to test the effectiveness of our metrics larger datasets.
21

---

Published as a conference paper at ICLR 2022
MNIST
100
mode
80
standard standard pow knockoff knockoff pow data free data free pow entropyrev entropyrev pow worstcase worstcase pow
Accuracy(%)
60
40
20
1000
100
1002
1003
100
Time (sec)
SVHN
100
mode
80
standard standard pow knockoff knockoff pow data free data free pow entropyrev entropyrev pow worstcase worstcase pow
Accuracy(%o
40
20
100º
100
1002
1003
1004
Time (sec)
Figure PoW against model extraction attacks for MNIST and SVHN datasets using the PkNN based Privacy Cost. Here the worstcase and entropyrev methods use out of distribution data.
B.8 DATA FREE MODEL EXTRACTION
Figure 15 shows the accuracy of the attacker model as the number of queries increases for the DataFree attack.
22

---

Published as a conference paper at ICLR 2022
MARST
CRARCO
/// //////// Mamm
Accuracy
Copyc TOOEEO:
capyort
rosddl
1000
3000
2000
4010
6000
18000
2000
4403
200
1800
Quertes
Querles
Quarles
NUST
SWh-W
2303
OF4R10
1423
Irouudirr
133
112.00.l
190
Findion
2003
2383
rundonn
2.89
135
0303
1.29
1.09
Queries
Queries
Queries
4.00
MM151
Swi
2403
1980
3250
6433
3000
2703
1322
1500
2003
6000
1850
oappe rosddi:
oapyce
rosdii
Pitrac
22
3910
2000
8001
7900
2080
421C
6000
12200
4.00
K000
5005
1800
Querles
Quee
Figure 11: InOut attack: Accuracy, Entropy, Gap, and Privacy of the victim model on all the answered queries number of Queries for MNIST, SVHN and CIFAR10 datasets.
CHARLS
Accuraty
0m3m 29
vid
vidin
2004
400
6009
9008
3000
2000
4008
230CE 1200 LEE
10000
2000 3400
4000
500W
Figure 12: AI attack: Accuracy of the attacker model vs number of Queries for MNIST, SVHN and CIFAR10 datasets, respectively. The accuracy of the student model resembles the increase in accuracy for standard training. The higher the model accuracy, the more queries are needed (higher entropy is incurred) further improve the model.
23

---

Published as a conference paper at ICLR 2022
SVNN
MANSS
CEA1LO
3TP
1233
rarobum
1303
432
233
2048
488
1931
1991
1986
1290
1400
18500
QUcric
Queries
Quertts
VMSST
690
SWH
04/02
randem
2703
249
1223
3.03
333
Querics
Queries
Querits
NW ST
12
SVHN
eytropy
enbagy
910
Queries
Qu
Outres
Figure 13: AL attacks: Entropy, Gap, Privacy costs of the victim model on all the answered queries vs number of Queries for the MNIST, SVHN and CIFARIO datasets, respectively. The active learning methods and random querying have similar trends in the privacy metries. In this case, the gap-based attack helps more to achieve higher accuracy with fewer queries than the entropy attack. The values of the three grow similarly with more queries.
240
100
Srtr99
2.80
130
2003
Querees
Querees
Quertes
Figure 14: AL attacks: Entropy, Gap, Privacy costs of the victim model on all the answered queries vs number of Queries for the Imagenet dataset. It is easy to differentiate between the active learning methods and random querying using all three metrics. Based on these empirical trends, the privacy cost resembles more to entropy than the gap metric.
B.9 JACOBIAN/ JACOBIAN-TH
Figure shows the accuracy of the attacker model as the number of queries increases for Jacobian and Jacobian- TR attacks. Figure 17 shows the values of the cost metrics against the number of queries.
24

---

Published as a conference paper at ICLR 2022
HR:5T
SVY-
CHARLS
Accuracy
Querles
Queries
34t
Figure DataFree attack: Accuracy (of the student/attacker model) vs number Queries for MNIST, SVHN, and CIFAR1O 10 datasets, respectively. The accuracy of the student model resembles the increase accuracy for a standard training. The higher the model accuracy, the more queries are needed (higher entropy is incurred) to further improve the model. DataFree needs 2 mln queries to either match the accuracy the victim model (for SVHN) or 1.5 mln for MNIST and 20 mln for CIFARIO to achieve its highest possible accuracy (which is few percentage points lower than the accuracy of the victim model).
390
JUSE an
20
1i0 2017
18008 2009 300 2000 SERE 1390 1226 10000 2008 2018 +3003 20300 50065 75506
2000 4000 XEE 8399 10089 12899 14000
Queries
Querles
Querles
Figure 16: Jacobian attacks: Accuracy (of the student/attacker model) vs number of Queries for MNIST, SVHN, and CIFARIO datasets, respectively. The accuracy the student model increases by large amount the beginning when using small number of initial queries and then remains relatively constant (when there are more Jacobian augmented queries). This likely because after the initial subset of images, the others generated based on Jacobian augmentation and thus eventually will not lead to sufficient new information for the model to learn from. Jacobian attack achieves lower accuracy of the extracted model than the other attacks (only about 80% for MNIST and 20% for SVHN as well as CIFAR10).
B.10 MIXMATCH
Figure 18 shows the accuracy of the attacker model over time when MixMatch is used for training and Figure 19 shows the corresponding values of the cost metries.
B.11 KNOCKOFF NETS AND COPYCAT CNN
We note both of these attacks have the same costs both methods query randomly from task- specific but an muitr--distribution dataset, this case ImageNet. Therefore S the same result for the Knockoff Nets and CopyCat. The accuracy of these methods is compared against other attacks that use distirriibution data in Figure
B.12 FASHION MNIST DATASET
We also similar experiments on the Fashion-MNIST dataset for which the results are shown in
Figures 20 2T and 22 We note that we have mostly similar trends with the main difference being
that the entropy AL method gives a lower cost than random querying. This is likely result of the attacker's s estimation of the costs on the victim side not being as accurate as the other methods which causes the selected queries to not have the smallest cost when measured the victim's 's side. In
25

---

Published as a conference paper at ICLR 2022
MRIST
Ixcebar
2403
CIARRU
jicetarry
12:
653
13.6
Qutrles
Queries
Queres
MhisT
joceiin peeeborr
238
CIARR0
288
158K
0.3
170
Queries
Queries
Queer Ms
HMSST
SV1N
2900
CIRARDO
patib on
bcothn
[ecoclarzz
1330
paincy
1250
530
1006
10.6
2000
300
12000
2500 5000
7505 1005 1240
150M 1756) 2000
2003 SEER
4530 1230 1930 DEEE LARCE
Figure 17: Jacobian attacks: Entropy, Gap, Privacy costs of the victim model on all the an- swered queries vs number of Queries for the MNIST, SVHN and CIFARIO datasets, respectively. Jacobian methods incur higher costs for SVHN and CIFARIO while for MNIST, random has slightly higher cost due to not enough difference of the generated queries from the initial query points.
HO.ST
SA
CITARLE
mibmacch
Ukt sccurdy
Vidin aSOUEO))
1500
7033 8x
2000 2018 2103
1210
1503 2000
1530
Figure 18: MixMatch attack: Accuracy (of the student/attacker model) vs number Queries for MNIST, SVHN, and CIFARIO datasets, respectively. A higher number of queries leads to a higher accuracy. The test accuracy corresponding to the epoch with the maximum validation accuracy is used for these plots. MixMatch is the best performing attack in terms of the task accuracy goal. It requires only around a few hundreds queries for MNIST, 1000 for SVHN, and 8000 for CIFAR10, to almost match the accuracy of the victim model.
other words, there is mismatch between the cost estimated on the attacker's side and the real cost computed the victim's side.
26

---

Published as a conference paper at ICLR 2022
NMST
SWeM
CPMRL
Ho-eosh
234
Quedees
Querles
Querles
NMBT
CFAALE
Meteach
150
Haloar
Viraath
225
19
54
190
2022
(00)
100)
7030
3000
135
283
2001
40.0
6001
1900
1000
12
283
108
4001
100)
152
7423
Bandon
Rondon Hosoac
20085
kandin Mbirath)
2.0
Privace
142
2500 2009
209 1020 5009 Querics
5303
1005
0000
1090 2009
2003
0006 Querees
1809
5009
70.5
$500
1303 2007
2035
-938
1006
0351
300
1000
Quarics
Figure 19: MixMatch attack: Entropy, Gap, and Privacy costs of the victim model on all the answered queries number of Queries for MNIST, SVHN and CIFARIO datasets, respectively. The costs are very similar because the MixMatch extraction also uses random querying to create the labeled set.
Rushion HLL
Figure 20: AL attacks: Accuracy Entropy, Gap, and Privacy of the victim model on all the answered queries vs number of Queries for the Fashion-MNiST dataset. Random queries can provide higher accuracy of the extracted model and lower costs incurred by the queries on the victim's side than the simple AL methods.
Cuerte
300
100
2505
300
100
1200
Oueris
1900
Figure 21: Jacobian attacks: Accuracy Entropy, Gap, and Privacy of the victim model all the answered queries vs number of Queries for the Fashion-MNIST dataset.
27

---

Published as a conference paper at ICLR 2022
Taston-8157
OLerto
Oueto
Oueres
Figure 22: DataFree Attack: Accuracy, Entropy, Gap, and Privacy of the victim model on all the answered queries number of Queries for the Bushion-MNISS dataset.
Table 5: Compare computation times (in seconds) of querying the API when only inference is performed or with the additional computation of the query cost when labeling samples.
DATASET
OF LABELED SAMPLES
INFORENCE
PRIVACY
ENTROPY
GAP
0.000
0.214
0.001
0.001
SVHN
0.250 0.000
2.167
0.012
0.250
0.001
0.250
0.001
CIFARIO
64
0.249 0.001
1,031 = 0.006
0.230 = 0.001
0.249
0.001
C COMPUTING THE COST OF QUERIES
We compute the cost of queries using different measures, such as differential privacy, entropy, or gap. We compare the overheads of each these methods in terms of their execution time Table
D PRIVACY COST WITH PATE
Differential privacy (DP) canonical framework for measuring the privacy leakage of a randomized algorithm (Dwork et al., 2006). It requires the mechanism, the training algorithm in our case, to produce statistically indistinguishable outputs on any pair of adjacent datasets, i.e., datasets differing by only one data point. This bounds the probability of an adversary inferring properties of the training data from the mechanism's outputs.
Definition (Differential Privacy) A randomized mechanism M with domain D and range R satisfies (s, d)ffferrret privacy for any subset S C and any adjacent datasets e D, i.e. l|d - d'Il S 1, the following inequality holds: Pr Pr[M( e S] s e (M(')) E S] +
PATE (Private Aggregation of Teacher Ensembles) by (Papernot et al. 2018) uses RDP (Rényi Differential Privacy) that generalizes pure DP (i.e., when = 0) by using the Réni-divergence to compare the mechanism's 's output distributions and obtains the required randomization for privacy by sampling from a Gaussian distribution and
Definition 2 Rényi Differential Privacy (Mironov 2017 We say that a mechanism M is (A, e)- RDP with order E (1,00) for all neighboring datasets X, X':
[ (MIX)(()) ) (MMXX)
Dx(M(X)IM(X')) - log E
SE
is convenient to consider RDP in functional form as (A), which the RDP & of mechanism M at order X. RDP analysis for Gaussian noise is particularly simple.
Lemma RDP-Gaussiar mechanism. Let f : X have bounded &g sensitivity for any two neighboring datasets X, X', i.e., (I - f(X'll2 < A2. The Gaussian mechanism M(X) =
f(X) + +0(,00² obeys RDP with (M()) 2n
AA3
Another notable advantage of RDP over (c, 8DP is that it composes naturally.
28

---

Published as a conference paper at ICLR 2022
Lemma 2 RDP- COP-Composition (Mironov 2017). Let mechanism M (Mi...., where M. can potentially depend on outputs of Mi,.... M...---------... M obeys RDP with eM()) = Σi=1 e.M, (.)-
PATE is model-agnostic approach to DP in ML (Papernot et al., 2017a). PATE trains an ensemble of models on (disjoint) partitions of the training set. Each model, called a teacher, is asked to predict on query and to vote for one class. Teacher votes are collected into histogram where n, (x(x) indicates the number of teachers who voted for class i, x is an input sample. To protect privacy, PATE relies the noisy argmax mechanism and only reveals noisy aggregate prediction (rather than revealing each prediction directly): argmax n. (2(2)) + (0, o?) where the variance o of the Gaussian controls the privacy loss of revealing this prediction. Loose data-independent guarantees obtained through advanced composition (Dwork et al. 2014) and tighter data-dependent guarantees are possible when there is high consensus among teachers on the predicted label (Papernot et al. 2018). To further reduce privacy loss, Confident GNMax only reveals predictions that have high consensus: max; (z) is compared against a noisy threshold also sampled from Gaussian with variance o². A student model is trained in a semi-supervia ensemble. Each returned label incurs privacy loss, which we use in our method measure the query cost. We train the ensemble of teachers on the defender's s training set. We do not reveal the PATE predictions, skip the confident GNMax since the privacy loss has to be computed for each query, and do not train a student model. The predictions are returned using the original defender's 's model.
fashion on noisy labels returned by the
E MORE INFORMATION ON OTHER DEFENSES
Active defenses can be further categorized into accuracy-preservvnng. which do not change the where the drop in accuracy is bound but allows
predicted top-1 class, and for a higher level of perturbations. Lee et al. (209)) introduce accuracy-prexerrvin defense which perturbs a victim model's final activation layer. The predicted classes are preserved but output probabilities are distorted so that an adversary is forced to use only the labels. Accuracy- construined defenses can also alter the returned label. Orekondy et al. (2020) modify the distribution of model predictions to impede the training process of the stolen copy by poisoning the adversary's gradient signal. however, it requires costly gradient computation. The adaptive misinformation defense (Kariyappa & Qureshi 2020) identifies if a query is in-distribution or -uttof-distribution (OOD) and then sends incorrect predictions for OOD queries. If an attacker does not have access to in-distribution samples, this degrades the accuracy of the attacker's clone model but also decreases the accuracy for benign users. This defense uses OOD detector trained on both in-distribution and OOD data. Kariyappa et al. (2021) also propose to train an ensemble of diverse models that returns correct predictions for in in-distribution samples and dissimilar predictions for OOD samples. The defense uses a hash function, which is assumed to be kept secret, to select which model from the ensemble should serve given query. To train either OOD detector or the ensemble of models, the defenses introduced by Kariyappa et al. assume knowledge of attackers' OOD data that are generally hard to define a-priori and the selection of such data can easily bias learning (Hsu al. 2020).
Reactive defenses are applied after a model has been stolen. Watermarking allows a defender to embed some secret pattern in their model during training (Adi et al. 2018) inference (Szyller et al. 2019). For the defense to function, any extracted model obtained by the attacker should inherit this secret pattern. However, these watermarks are easy to remove because they are from a different distribution than the training data of the model. Subsequently, Jia et al. (2020a) proposed entangled watermarks to ensure that watermarks are learned jointly with the original training data, which requires changes to the training of the defended model and exhibits trade-off with model accuracy. Rather than encode additional secret information in the model, two recent lines of work leverage secret information already contained in the model. Dataset inference (Maini et al.) 2021 identifies whether a given model was stolen by verifying if a suspected adversary's 's model has private knowledge from the original victim model's dataset. Proof of learning (Jia et al. (021) involves the defender claiming ownership of a model by showing incremental updates of the model during training process, which is difficult for the attacker to reproduce. Unfortunately, reactive defenses are effective after a model theft and require a model owner to obtain partial or full access to the stolen model. This means that if an attacker aims to use the stolen model as proxy to attack the victim model, or simply keeps the stolen model secret, it is impossible for the model owner to protect themselves with these defenses.
29


Reviews:
[{"review_id": "h5jcDSQcnlT", "reviewer": "Reviewer_hR7i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Interesting idea of applying proof-of-work to machine learning as an entry barrier.\n- The novelty of the paper seems to reside in the application of existing concepts, like PATE for privacy cost estimation and HashCash as challenge, as components of a theft defense method. I find this novel enough and quite ingenious. Moreover, the existing concepts were appropriately adapted for the task at hand.\n- The experimental section is extensive, and the experimental protocol seems appropriate.\n- The paper is clear, well organized and pleasant to read.\n- Good quality code base allowing to reproduce the experiments.\n\nWeaknesses:\n- The proposed defense does not seem effective against attackers that have access to data from a similar distribution to the training one (e.g., MixMatch baseline).\n- The Knockoff attack does not seem to use its most effective querying strategy (Random is used instead of Adaptive).\n- Doubling computation cost for a legitimate user is not a prohibitive cost, but is also non-negligible, both from an effort and computation time perspective, as well as the CO2 emissions mentioned in the ethics statement of the paper.\n\nOther comments / questions:\n- With the proposed approach, does it mean that legitimate users would incur higher query cost when submitting many queries?\n- The paper states multiple times that it proposes the first defense to leave model accuracy intact, but I would argue that it is the first *active* defense to do so.\n\n[Update post-discussion] I am raising my rating following the exchanges below.", "summary_of_the_paper": "This paper proposes a novel defense against model extraction attacks. The proposed approach slows information leakage by asking all users to answer a puzzle before receiving the response to their query (proof-of-work). The difficulty of the puzzle allows to keep the computation overhead low for legitimate users, while rendering information leakage prohibitively expensive for attackers. The puzzle difficulty is calibrated based on an estimate of how much information each user has already acquired. This is based on PATE, a differential privacy metric. Experiments are performed on multiple datasets, opposing the proposed defense to a wide range of attacks, including adaptive adversaries.", "summary_of_the_review": "Novel active strategy to defend against model extraction, good experimental results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cNcCvwkhLED", "reviewer": "Reviewer_LLkZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- S1: This paper proposes a novel defense focusing on delaying the model stealing process.\n- S2: The proposed defense is tested against 6 attacks on 3 datasets.\n\nWeaknesses\n- W1: If the goal of the proof of work is just to slow down the process, the server can simply choose to delay the response, unlike the distributed block chain. Thus, the proof of work does not look necessary.\n- W2: The defense does not work against an attacker using in-distribution data, such as a subscribed user using the model normally until enough data is accumulated.\n- W3: The use of PATE as the information leakage estimator is not justified. PATE is devised to protect sensitive data, and how sensitive data is connected to the performance of a machine learning model is unclear. This component is rather the most important part of the framework as it dictates the user experience and the attacker's success. A detailed reasoning or comparison to other options should improve the paper.", "summary_of_the_paper": "This paper proposes a new defense to the model stealing problem that an attacker queries a supervised machine learning model service to have data labeled to use as training data and copy the functionality of the model. The proposed method is proactive defense that slows down the attacker in obtaining the labeled data, based on the information leakage estimator. The evaluation shows that existing attacks using out-of-distribution data can be slow down more than a regular user querying in-distribution data.", "summary_of_the_review": "This paper is in general well written, and the experiment was done to show the strengths and the weaknesses of the proposed defense. The defense against model stealing is inherently difficult problem where the attacker in the end can steal the model using the queried data and returned label pairs unless architecture or resource is no object (e.g., if not GPT-3). The main idea proposed in the paper against such a strong attack looks interesting, but the key technique used here is the proof of work. I am not convinced this is necessary at all since the original model is hosted and controlled centrally, and response time can be fully controlled without using the proof of work. Just delaying the response would have much better control of information leakage per time unit than the proof of work that a powerful machine can do faster, or simultaneously processing multiple queries with multiple machines reaching up to the speed of a regular user. Rather the important piece is the information leakage estimator which is somewhat overlooked in the paper, especially in terms of why PATE has to be the right choice here. I think focusing on this can lead to a better paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SrNOizC9BuU", "reviewer": "Reviewer_emnb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a defense against model extraction attacks by forcing users to do a proof-of-work (PoW) puzzle before they receive the labels from the victim model. To minimize the impact on legitimate users, the defense tunes the per-query difficulty based on the extent of information leakage. The privacy cost is evaluated via differential privacy budget. Compared to other defense methods, this method will not reduce the accuracy of answers, and it can protect the model before the model stealing happens (unlike watermark schemes).\n\nMy comments are as follows.\n\nFirstly, I think the structure of the paper can be improved. The first two sections of the paper take up a lot of space, so that the discussion in section 3 is not clear enough. Some experimental results are placed in the appendix, and some tables interspersed in the reference list.\n\nSecondly, section 3.3 shows how to use differential privacy to evaluate the privacy cost. Though the authors explain \"We compute the privacy cost of queries using the PATE framework\",  \"The privacy cost is computed based on the consensus among teachers and the amount of privacy noise added to the histogram\" and  \"more details can be found in appendix D\", I would suggest to show the main formulas and conclusions here to help the readers to follow.\n\nThirdly, the authors create two models. The first one is used to predict query time from the privacy cost, the second one is used to predict the number of leading zeros for the desired time. So the whole mapping flow is : privacy cost → query time → the number of leading zeros. The authors find that using a simple linear regression is sufficient for both models. Firstly, Figure 2 shows the dependency between the computation time and the number of leading zeros is exponential, is it contradictory? Secondly, I think there should be a clear mapping from privacy cost to the number of leading zeros, which will help practitioners easily to tune parameters to control the planned results, such as \"Time POW\" in Table 2. Thirdly, I can't find the results about these linear regression models in the experiment part.\n\nFinally, the difference in privacy cost between attackers and legitimate users is the main evaluation metric. Figure 3 shows a higher privacy leakage for the attackers compared to a standard user, but for the CIFAR10 dataset, this distinction is not very obvious, and it causes the computational cost for attackers to increase a little bit (In Table 2, line 16, for the attack COPYCAT on CIFAR10, time changes from 148.3 to 326). This situation may be caused by the ambiguous mapping from privacy cost to computational time, and it may pose a risk of defense failure. So more analysis is needed here.\n", "summary_of_the_paper": "This paper proposes a novel defense to prevent model stealing by requiring users to solve proof-of-work puzzles. The authors evaluate their method against different types of model extraction attacks. The results show that the defense will result in the attacker costing higher computational time (100x) than legitimate users (2x).\n", "summary_of_the_review": "see main review", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0j6Z8UB_V7K", "reviewer": "Reviewer_aHDe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper discussed the issue, and the background works well. The purpose is clear, and\nthere are a satisfying amount of experiments for validating the proposed solution. However,\nthe privacy scoring section is not clear. I cannot find out why PATE is the best option for\ncalculating privacy metrics and the exact functionality of this section is not as straightforward\nas other parts. Moreover, identification of users is essential since, as mentioned, the difficulty of\nPOW will increase gradually, but this concern is not well pointed in the paper.", "summary_of_the_paper": "The paper discusses ML model extraction attacks while using APIs for accessing them on the\npublic networks. Although some methods exist for preventing or making the attacks hard, all of\nthem have substantial impacts on legitimate users’ experience while using the system, including\nthe slower models or lower accuracy in results. This paper proposed a method for dissuading\nthe attacker by increasing the cost of the attack. Solving a puzzle for all users before getting\nthe final response (POW) would be the solution noting that the difficulty will be increased if\nthe system identifies any adverse behaviors. This method requires no modification of the victim\nmodel and can be applied by machine learning practitioners to guard their publicly exposed\nmodels against being easily stolen.", "summary_of_the_review": "The authors should make the privacy scoring module more clear and add more discussion about identifying malicious nodes in continuous requests", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: the paper proposed a novel idea of  requiring users to complete a proof-of-work before they can read the model's prediction to prevent model extraction attacks. Reviewers were excited about the paper and ideas. Some misunderstanding raised by reviewers were sufficiently clarified by the authors in the rebuttal.
Decision: Accept (Spotlight)