HODA: Protecting DNNs Against Model Extraction Attacks via Hardness of Samples
Authors: AmirMahdi Sadeghzadeh
Faezeh Dehghan
Amir Sobhanian
Rasool Jalili
Venue: ICLR 2022 Submitted (2022)
Keywords: Trustworthy Machine Learning
Model Extraction Attacks
Hardness of Samples

Abstract: Model Extraction attacks exploit the target model's prediction API to create a surrogate model in order to steal or reconnoiter the functionality of the target model in the black-box setting. Several recent studies have shown that a data-limited adversary who has no or limited access to the samples from the target model's training data distribution can use synthesis or semantically similar samples to conduct model extraction attacks. As the training process of DNN-based classifiers is done in several epochs, we can consider this process as a sequence of subclassifiers so that each subclassifier is created at the end of an epoch. We use the sequence of subclassifiers to calculate the hardness degree of samples. In this paper, we investigate the hardness degree of samples and demonstrate that the hardness degree histogram of a data-limited adversary's sample sequences is distinguishable from the hardness degree histogram of benign users' samples sequences, consisting of normal samples. Normal samples come from the target classifier's training data distribution. We propose Hardness-Oriented Detection Approach (HODA) to detect the sample sequences of model extraction attacks. The results demonstrate that HODA can detect the sample sequences of model extraction attacks with a high success rate by only watching 100 samples of them.

HODA: PROTECTING DNNs AGAINST MODEL Ex- TRACTION ATTACKS VIA HARDNESS OF SAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Model Extraction attacks exploit the target model's prediction API create sur- rogate model in order to steal or reconnoiter the functionality of the target model in the black-box setting. Several recent studies have shown that a data-limited ad- versary who has no or limited access to the samples from the target model's 's train- ing data distribution can use synthesis or semantically similar samples to conduct model extraction attacks. As the training process of DNN-based classifiers is done in several epochs, we can consider this process as a sequence of subclassifiers so that each subclassifier is created at the end of an epoch. We use the sequence of subelassifiers to calculate the hardness degree of samples. In this paper, we inves- tigate the hardness degree of samples and demonstrate that the hardness degree histogram of a data-limited adversary's 's sample sequences is distinguishable from the hardness degree histogram of benign users' samples sequences, consisting of normal samples. Normal samples come from the target classifier's ' training data distribution. We propose Hardness-Oriented Oriented Detection Approach (HODA) de- tect the sample sequences of model extraction attacks. The results demonstrate that HODA can detect the sample sequences of model extraction attacks with a high success rate by only monitoring 100 samples of them.
INTRODUCTION
Deep Neural Networks (DNNs) have shown impressive performance in various tasks in recent years that have encouraged the industry to deploy DNN-based models in a variety of real-world applica- tions. Since the training process of DNNs and collecting training data is an expensive and tedious process, models are considered the intellectual property of organizations, and they must be kept secure. Therefore, models are often securely deployed on cloud servers, and only the creators can access the model parameters. Users are only allowed to query the model via a prediction API and receive predictions. Recent studies Tramèr et al. (2016); Papernot et al. (2017); Juuti et al. (2019); Orekondy et al. (2019); Jagielski et al. (2020) demonstrate that an adversary can exploit the predic- tion API of target model to create a surrogate model in order to steal or reconnoiter the functionality of the target model. Such attacks are called model extraction attacks, and they violate the intellec- tual property of model owners. Furthermore, the surrogate model can be leveraged to conduct other attacks on the target model in black-box setting, such as adversarial example attacks Papernot et al. (2017); Juuti et al. (2019) and membership inference attacks Shokri al. (2017).
Most of the model extraction attacks use the target model's prediction API to label an unlabeled dataset create the surrogate model's training set. In most real-world settings, the adversary has no or limited access to samples from the target model's 's training data distribution, which is called nor- mal or in-distribution samples. Hence, most proposed attacks in the previous studies use some form of Out-Of-Distributt or semantically similar samples to the target model's training set Orekondy al. (2019); Pal et al. (2020) to conduct model extraction attacks. We focus on such attacks in this paper. There are two main approaches to defend against model extraction attacks, manipulating the target model outputs to prevent adversary from producing high-quality surrogate model Lee et al. (2019); Orekondy et al. (2020); Kariyappa& & Qureshi (2020); Kariyappa et al. (2021b) and detecting the sample sequences of model extraction attacks kesarwani al. (2018); Juuti al. (2019). Juuti al. (2019) propose PRADA to detect samples sequence of model extraction attacks based on the distance among suc- cessive samples. We propose Hardness-Orientted Detection Approach (HODA), new approach to
(OOD) samples, such as synthesis Papernot et al. (2017); Juuti et al. (2019)

---

detect sample sequences of model extraction attacks, which outperforms PRADA by large margin and has significantly less computational overhead.
Generally, the training process of DNN-based classifiers is done in several epochs, and the resulted classifier at the end of the last epoch is considered the final classifier. We can consider the training process of DNN- ANN-based classifiers created at the end of the 1th epoch. HODA uses a subset of subclassifiers to compute the hardness degree of samples. The hardness degree of a sample is equal to the index of the subclassifier that all subsequent subclassifiers agree with its predicted label for that sample. It is important to note that we must save subclassifiers the training phase of a target classifier in order to use their predictions to calculate the hardness degree of new samples. We demonstrate that the hardness degree histogram of normal sample sequences distinguishable from the hardness degree histogram of model extrac- tion attack sample sequences, and HODA uses this property to detect sample sequences of model extraction attacks. For each user, HODA calculates the distance between the hardness degree his- tograms of the user's samples and normal samples, and if the distance is greater than threshold, JBR the user is detected as an adversary. HODA can detect JBDA Papernot et al. (2017), JBRAND Juuti
a sequence of subclassifiers in which the 1th subclassifier is
et al. (2019), and Knockoff Net Orekondy et al. (2019) attacks with high success rate by only monitoring 100 samples of attack. We demonstrate that HODA is also highly effective when the target classifier is trained using transfer learning.
Contributions. (i) We demonstrate that the hardness degree of a sample for a classifier pertains to the training data distribution of that classifier. (ii) We indicate that the hardness degree histogram of normal samples is distinct from the hardness degree histograms of model extraction attack samples. (iii) We propose HODA detect the sample sequences of model extraction attacks.
2 RELATED WORK
Model Extraction Attacks: Primary model extraction attacks try extract the exact value of pa- rameters Lowd & Meek (2005); Tramèr et al. (2016) and hyperparameters Wang & Gong (2018) of shallow models. In recent years, the proposed attacks mainly aimed to steal reconnoiter the functionality of deep neural networks by querying them in the black-box setting. It is often sensibly assumed in the literature that the adversary has no or limited access samples from the training set distribution of target classifier. In order to overcome this issue, attacks generally use some form of
samples, such as synthesis or semantically similar samples, to create the sur- rogate classifier's 's training set. Knockoff Net Orekondy et al. (2019), Active Pal et al. (2020), and Copycat CNN da Silva et al. (2018) use semantically similar datasets to the target model's 's train- ing set to train surrogate classifier. In another line of studies, Papernot et al. (2017), Juuti et al. (2019), Yu et al. (2020), Truong et al. (2021), Kariyappa et al. (2021a), and Barbalau et al. (2020) use synthetic data to create the surrogate classifier's 's training set.
Model Extraction Defenses: Existing defense methods against model extraction attacks generally distribute into two branches: perturbation- based and detection- based. Perturbation baased defenses Lee et al. (2019); Orekondy et al. (2020); Kariyappa & Qureshi (2020) attempt to prevent adver- saries from producing high-quality surrogate classifiers by adding perturbation the target clas- sifier outputs. Recently, Kariyappa et al. (2021b) proposed a new defense with the same goal as verturbation-basse defenses, which does not perturb the target classifier outputs. Their approach employs an ensemble of diverse models to produce discontinuous predictions for out-of-distribution samples. Detection-based defenses attempt detect the occurrence of model extraction attacks by observing successive input queries to the target classifier. Kesarwani et al. (2018) present a method to detect extraction attacks against Decision Tree models. PRADA Juuti et al. (2019) is the first proposed iirtection-based defense for DNN models. We propose new defense detecting the sample sequences of model extraction attacks via hardness of samples.
Atli et al. (2020) demonstrate that several OOD detection approaches, such as Baseline Hendrycks & Gimpel (2017) and ODIN Liang et al. (2018), have poor performance in detecting Knockoff Net attack samples. Hence, they propose new OOD detection approach that leverages classifier to detect OOD samples. However, their approach only rejects OOD samples, and it does not have any detection mechanism to detect adversaries. Besides, the OOD detector is trained the same distribution used by the adversary to conduct Knockoff Net attacks, which is an unrealistic assumption in practice. Concurrent with our work, Zhang al. (2021) and Pal et al. (2021) propose
samples from
2

---

SEAT and VarDetect to detect sample sequences of model extraction attacks, respectively. SEAT aims to detect model extraction attacks that use several similar samples to extract target model, such jacobian- based attacks (Papernot et al. (2017); Juuti et al. (2019)). Hence, SEAT is ineffec- tive when an adversary uses natural samples that are not similar to each other, such as Knockoff Net attack. VarDetect uses Variational Autoencoders (VAs) and Maximum Mean Discrepancy (MMD) to detect model extraction attacks. VarDetect has only been evaluated on low-dimensional datasets. Re- garding that VarDetect uses VAs and MMD, it is unclear how well it performs on high-dimensional datasets. Besides, it uses the ImageNet dataset to extract target classifiers trained on very struc- turally different datasets, such as F-MNIST and SVHN. HODA is evaluated on harder attacks using attack datasets that are structurally similar to the target classifier's dataset, such as using ImageNet dataset to extract Caltech256 target classifier. HODA can detect both jacobian-based and Knockoff Net attacks, and it performs well on high-dimensional datasets, such as Caltech256 and CUB200. Furthermore, unlike other work Atli et al. (2020); Kariyappa & Qureshi (2020); Kariyappa et al. (2021b), HODA only needs access to in-distribution samples to detect model extraction attacks.
3 MODEL EXTRACTION ATTACKS
The model extraction attack is one of the most serious threats against machine learning-based clas- sifiers remote servers, such as Machine Learning as a Service (MLaaS). The adversary's s goal is to create a surrogate classifier fa that imitates a target classifier ft on task T. Most model extraction attacks exploit target model ft label unlabeled samples to create the surrogate model's training set. The adversary sends sample z; to the target model and receives its output fe (a;)), and then she uses pair fe(z)) to train surrogate classifier fs. The output type of target model can be label, label confidence, top-k values in probability vector, or the entire probability vector. We only con- sider label fe (x:)) and the entire probability vector (x:) as the output type of target classifiers in our experiments. There are two primary intents for adversaries to conduct model extraction attacks, stealing and reconnaissance.
Stealing: Producing high performance classifier is an expensive and time-consuming process and requires computational resources and experts. Besides, given that DNNs need large number of training samples, collecting data and labeling them is a complex and costly procedure for most real- world applications. Therefore, adversaries are motivated to take advantage of a target classifier to reduce the cost of creating new classifier. The adversary's 's goal in stealing is to maximize the accuracy of surrogate model on data distribution Dr. Hence, the adversary's 's goal is:
Maximize P(..ii>,,,s(
Recomnaissance: The model extraction attacks can be used to conduct other attacks in the black- box setting, such as adversarial example attacks Papernot al. (2017); Goodfellow et al. (2015) and membership inference attacks Shokri et al. (2017). The adversary's goal in reconnaissance is to maximize the fidelity among surrogate and target classifiers in order to increase the success rate of black-box attacks. Similar to Jagielski et al. (2020), we consider label agreement among surrogate and target classifiers as the fidelity metric on data distribution Dr. Hence, the adversary's 's goal is: Maximize f(ax))-),ffzI)) (z)
(2)
Proposed model Extraction attacks create the surrogate classifier training X, - ((,,, (xx))))
by various methods, where B is the attack budget. The attack budget determines the number of samples that an adversary is allowed to send to the target classifier and receive their associated predictions. After creating X,, the adversary trains surrogate classifier f. to minimize empirical loss on Xg. We suppose that the adversary knows the architecture and hyperparameters of the target classifier and uses them to train the surrogate classifier. It is noteworthy that our proposed defense independent of surrogate classifiers' training process.
OUR PROPOSAL: HARDNESS-ORIENTER DETECTION APPROACH
4.1 HARDNESS DEGREE OF SAMPLES
The training process of ONNNbased classifier can be considered sequence of subclassifiers called F y that each subclassifier is created at the end of epoch. Suppose that classifier f. is
3

---

DenseNet121
ResNet18
MobileNet
of samples
2K
2K
2K
IK
0
20
40
60
80
100
0
20
40
60
80
100
20
40
60
80
100
(a) CIFARIO
4 of samples
IK
.5K
.5K
0
20
40
60
80
100
20
60
80
100
20
40
60
80
100
Hardness Degree
Hardness Degree
Hardness Degree
(b) CIFARIOO
Figure 1: The hardness degree histograms of CIFARIO and CIFARIO0 test samples for DenseNet121, ResNet18, and MobileNet classifiers.
trained for epochs. The training process of classifier fo can be represented as the following sequence of subclassifiers:
F = fo St. ftm
where subclassifier fí is created the end of the 7thh epoch. We say sample x, is learned in epoch when ft is the first subclassifier that its predicted label is equal to all subsequent subclassifiers' predicted labels. Generally, as the number of epochs increased, the performance of classifier fe is improved so that easier samples are learned in the early epochs, and harder ones are learned in the last epochs. Therefore, the hardness degree of sample x; for classifier fc. which is displayed by Φ, (zz). directly relates to the epoch number that x; is learned by fc. Hardness degree of sample a; for classifier fe is defined as follows:
Φ (z:) =e s.t.
1], Je (x:)= (()),, (1( F
(4)
The hardness degree domain is dependent on the number of subclassifiers, and since we have m subclassifiers, the hardness degree of a sample is in the range [0, m 1]. Since we want to calculate the hardness degree of samples in inference time, we need save subclassifiers at the end of each or several epochs in the training phase of target classifiers to use them in the in- ference time. When new sample arrives, is fed to all loaded subclassifiers, and using their predictions, the hardness degree of that sample is computed. not use the true label of samples calculate their hardness de- gree. As seen in equation (4), the hardness degree of samples is calculated only by the prediction of subclassifiers It noteworthy that do not compute the hardness degree of the target classi- fiers' training samples throughout the paper and only compute the hardness degree of normal (test) or attack samples. Algorithm in Appendix G describes computing the hardness of samples using Fsubety in the inference time.
Table
The accuracy of classi- fiers on CIFARIO and CIFAR100 test sets.
ResNet18
DenseNet121
MobileNer
CIAARI
94.35
94.92
93.59
77.57
important to note that we do
We train three various types of classifiers, including 1enseNet121 Huang et al. (2017). 18 He et al. (2016), and MobileNet Sandler et al. (2018), 3), on CIFAR10 and CIFAR 100 training sets for 100 epochs (details of datasets in Appendix A). All classifiers are trained using stochastic gradient descent with momentum 0.9 and batch size 128. The learning rate is 0.1 and it is scheduled to be decreased in each epoch by a constant factor 0.955. The accuracy of classifiers Table 1. We save all 100 subclassifiers in the training phase of each classifier and use them to cal- culate the hardness degree of samples. Figure shows the hardness degree histogram of CIFARIO I0 and CIFARI00 test samples for various classifiers, The figure demonstrates that a large fraction of CIFAR1O test samples are easy, and many samples are learned in the first few epochs. However, the learning of CIFAR10 100 test samples is distributed over various epochs, and the number of hard samples is more than CIFARIO. Figure 6 in Appendix B demonstrates a strong positive correlation between the hardness degree of samples and the misclassification rate. As ResNet18 architecture
presented in
4

---

achieves strong performance on both datasets at reasonable computational cost, this archi- tecture for target classifiers in the rest of the paper. We conduct various model extraction attacks on two CIFARIO and CIFARI00 target classifiers in the next subsection to depict the hardness degree histogram of their samples.
4.2 MODEL EXTRACTION ATTACKS SETUP
In line with prior work (Orekondy et al. (2020); Kariyappa & Qureshi (2020); Kariyappa et al. (2021b)), we select JBDA Papernot et al. (2017), JBRAND Juuti et al. (2019), and Knockoff Net Orekondy et al. (2019) model extraction attacks to evaluate our defense method. These attacks broadly represent two main strategies (synthesis or semantically similar samples) to conduct model extraction attacks. Jacobian-Based Dataset Augmentation (JBDA) Papernot et al. (2017) and its improvement (JBRAND) Juuti et al. (2019) assume that the adversary has access to a limited number of samples from the target classifier's training data distribution called seed samples, and they aim to augment seed samples using adversarial examples to increase the fidelity of the surrogate classifier to the target classifier. Orekondy et al. (2019) propose Knockoff Net (K.Net) attack that uses large public datasets that is semantically similar to the target classifier dataset to increase the accuracy of the surrogate classifier. We consider two versions of K.Net attack, K.Net CIFARX, and K.Net TIN. K.Net CIFARX attack uses CIFARI00 training set to extract CIFARIO target classifier and vice versa. K.Net TIN employs Tiny ImageNet training set to extract target classifiers. More details about attacks and their implementations are presented in Appendix D.
To evaluate the performance of model extraction attacks, we use two ResNet18 classifiers being trained on CIFAR1O and CIFAR 100 training sets as the target classifiers and con- duct all four attacks on them. The de- fault value of the attack budget in our experiments is 50000 (B=50K). Table shows the accuracy and the fidelity of surrogate classifiers created by var- ious model extraction attacks on CI- FARIO and CIFAR100 test samples. The results demonstrate that K.Net attacks have significantly better per- formance than jacobian- based attacks (JBDA and JBRAND), and when the output of target classifiers is probabil- ity vector, the performance of attacks is considerably increased.
The Accuracy (Acc) and the Fidelity (Fid) of sur- rogate classifiers being created by four various model traction attacks on two target classifiers CIFARIO and CI- FARI00. The output type of target classifiers can be Label or Probability Vector (Prob. Vec.).
Table 2:
1.
Metric
Ourpus type
JBDA
JBRAND
N.Ner CIFARX
K.Ne: TIN
Prob. Vec.
41,00
43.33
79.86
78.86
CIBARIO
Acc
Label
34,57
34.35
66.88
71.29
ResNet18
Prob. Vec.
41.16
43.63
$1.36
80.18
(Ace: 94.36%
Fide%)
Label
34.45
67.98
72.43
Prob. Vec.
16.44
18.78
51.09
60.36
CIPARIOO
Acc(())
Label
8.62
8.07
23.20
32.88
ResNet18
Prob.
19.13
54.59
(Ace: 76.33%)
Fide%)
Label
8.91
8.20
24.72
34.58
4.3 HARDNESS DEGRER OF MODEL EXTRACTION ATTACK SAMPLES
Figure depicts the hardness degree histogram of 50000 samples generated by various attacks for CIFARIO and CIFARI00 target classifiers. In this experiment, the architecture of target classifiers ResNet18. We also present the hardness degree histogram of attack samples when the architecture of target classifiers is Densenet121 in Appendix E. Figure demonstrates that the samples generated by various attacks have very small number of easy samples, and most samples have medium and high hardness degrees. However, Figure indicates that a high number of normal samples that are from the same distribution as the target classifier's training set are easy.
To investigate more on the hardness degree of attack and normal samples, Figure 3 displays two- dimensional visualization of CIFAR10 10 test samples using t-SNE. Figure 3a uses the logits of the CIFARIO 10 classifier to visualize CIFAR 10 test samples, and the color of each sample is determined by its label. This figure has ten sample clusters where most samples of each cluster are from one class. Figure 3b illustrates the hardness degree of CIFAR10 10 test samples for CIFARIO 10 target classifier and demonstrates that most of the easy samples are in the high-density regions inside clusters, and most of the hard samples are in the low-density regions at the borders of clusters. Figure 3c is similar to Figure 3b, but the hardness degree of each sample is calculated via CIFAR 100 target classifier.
5

---

JBDA
JBRAND
K.Net CIFARX
K.Net TIN
1000
10K
4K.
# of samples
3K.
3K.
2K
4K
4K
(a) CIFAR10
100%
10K
4K
# of samples
6K
3K
3K
2K
2K
4%
4K
1K
Hardness Degree
Hardness Degree
Hardnesss Degree
Hardness Degree
(h) CIFAR 100
Figure 2: The hardness degree histograms of samples of four various model extraction attacks for CIFARIO and CIFARIOO target classifiers. The budget of model extraction attacks is 50000.
0.3
0.5
0.5
0.5
Airplane Automobile
Foog
Horee Ship Truck
Hardnwes Degrne
Hardroes Degros
(b)
(c)
Figure 3: (a) Visualization of CIFARIO test samples. (b) Hardness of CIFARIO test samples for CIFARIO 10 classifier. (c) Hardness of CIFARIO test samples for CIFARI00 classifier.
This figure demonstrates when the training data distribution of the classifier being used to calculate the hardness degree of samples becomes different from the distribution of CIFARI0 test samples, hardness degree of a high number samples changed. Figure 3c shows hard and medium samples distributed among clusters, and the number of easy samples is very small. Similar to Figure 3, we visualize CIFAR 100 test samples and their hardness for CIFARIO and CIFAR target classifiers in Appendix F. Based on experiments, the empirical distribution of the target classifier's training data called Pdata- Easy samples are from the high probability region, and hard samples are from the low probability region of the input space. For example, if we suppose that P Pdato is a Gaussian-like distribution, easy samples are closer to the center of the distribution, and hard samples that easy and hard normal samples lie in the high- and low densiey region of the input space. Figures 2 and 4 demonstrate that the number of easy samples among model extraction attack samples (OOD samples) is very small, which means that attack samples are from the low probability region of the input space. However, Figures and 4 show high number of normal samples are easy, which means that they are from the high probability region of the input space. We use histogram rather than hardness degree histogram in the rest of the paper for simplicity.
consider hardness degree
estimator of
in the tail of the distribution. Figures 3 and 8 show
4.4 HARDNESS-ORIENT ORIENTED DETECTION APPROACH
We propose Hardness- Hairdness-Oriente Detection Approach (HODA) to detect sample sequences of model extraction attacks. HODA requires normal histogram H,, representing the histogram of normal

---

samples. When new sample 2; from user arrives, HODA caleulates its hardness degree Φ (2:), and the histogram belongs to that user H,, is updated. After the number of samples sent by user reaches specific number num,, HODA calculates Pearson distance between the histograms of normal samples H, and user samples Hu. and if the distance is greater than a threshold 6, the user is detected as an adversary. Pearson Distance (PD) between two random variable X and Y is defined as follows:
PD(X,Y) = XY
(5)
PxPY
where Cov(X, X Y) is the covariance between random variables X and Y. and px is the standard
deviation of random variable X. The output of Pearson distance is in the range [0,2]. To calculate the Pearson distance between two histograms, HODA first transforms histograms into probability vectors by dividing the value of histogram bins by the total number of samples in the histogram (Hn/sum(FF) H))) and H../sum(H.)) and then calculates the Pearson distance between them.
HODA uses normal sample set SHODA create Hn and calculate ö. It randomly selects numseg sample sequences with size num,, from the sample set SHODA and for each sample sequence, produces a histogram and adds it to the histogram set isstStt. The normal histogram Hn, is the average of all histograms in HístSet, and ō is the maximum Pearson distance between H,, and all histograms in Hist.Set. Since is independent of attacks and only relies on normal samples, HODA is not dependent on any attacks. Notably, HODA does not need to save samples of each user or their hardness degrees. It only keeps vector (H,) that indicates the values of histogram bins for each user. Algorithm 2 in Appendix G describes HODA in details.
5 SETUP AND EVALUATION
Two normal sample sets SHODA and S. are required to evaluate the performance of HODA. S., is used to simulate benign users. We randomly select 40% and 60% of test samples of each dataset for SHODA and S.,. respectively. We randomly select numm 40000 sequences with size num, from SHODA to create H, and calculate ò. To evaluate the performance of HODA against model simulate 10000 benign users and 10000 adversaries for each attack. Each be-
extraction attacks, nign user sends a sequence of num, samples randomly selected from S... and each adversary sends sequence of numm, samples randomly selected from 50000 samples of attack in the order they were generated. So far, we have used 100 subclassifiers to calculate the hardness degree of samples. How- ever, it may not be possible to classify each sample by high number of subclassifiers in practice. So in order to reduce the computational cost of HODA, we use subset of subclassifiers called F F'sube// to compute the hardness degree of samples. HODA only uses 11 subclassifiess to calculate the hard- ness degree of each sample, and these subclassifiers are saved in the training phase of target classifier fi at the end of each 10 epochs F Faubelf =< fo. Je, f f220 f20 5:00 f fca f79 Sr. 1 >. Algorithm 1 in Appendix G describes how hardness degree is computed using F Fsubelf in details. Since the hardness degree domain depends on the number of subclassifiers, the hardness degree of a sample in HODA is in the range [0,10].
We compare the detection rate and the false-positive rate of HODA with PRADA. PRADA Juuti et al. (2019) declares that the histogram of min- imum La distance between a new sample and all previous samples of benign user follows a Gaussian dis- tribution. Hence, it uses the Shapiro- Wilk normality test to determine that a sample sequence belongs to a be- nign user or an adversary. Similar to HODA, PRADA also uses thresh-
Table 3: The detection rate and False Positive Rate (FPR) of PRADA and HODA against four various model extraction attacks on CIFAR 10 and CIFAR 100 target classifiers.
Detection Rate Altack((S)
(FR/%)
JBDA
JBRAND
K.Net CIFARX TIN
100
0.818
0.01
0
PRADA
CIFARIO
500
0.08
96.7
50
0.290
0.02
100
100
99.92
99.73
IIODA
1001
0.154
0.02
100
100
500
0.550
0.01
0
CIFARUM
PRADA
1000
0.953
0.03
67.3
73.5
old ó to detect sample sequences of model extraction attacks, and 8 is
0.715
0.02
94.65
90
89.06
HODA
100
0.349
0.02
100
100
100
100
the only parameter of PRADA. Since
PRADA needs to save each user's
samples and calculate La distance between them, it has a high computational overhead. Table
7

---

Test, set
K.Net ILSVRC12
Test: set
K.Net ILSVRC12
1,000
9K
5K
# of samppees
of semples
900
8K
800
8K
600
800
<K
<K
200
2K
200
2K
0-
00
80
100
20
80
60
100
00
100
Harduess
Degree
Harduess
Degree
Harduess Degroe
Hardness Degree
(a) CUB200
(b) Caltech256
Figure 4: The left histogram in subfigures and b shows the hardness degree histogram of CUB200 and Caltech256 test samples, respectively. The right histogram in each subfigure indicates the hard- ness degree histograms of K.Net ILSVRC12 attack samples on CUB200 (a) and Caltech256 (b) target classifiers.
indicates the detection rate and False Positive Rate (FPR) of PRADA and HODA against four var- ious model extraction attacks on CIFARIO and CIFARI00 target classifiers. We evaluate HODA when it only monitors 50 and 100 samples of each user (num, 50 and num, 100), and since PRADA needs to monitor more samples to detect attacks, we use larger num, to evaluate PRADA. PRADA and HODA have very low false-positive rates. Palst-Positive Rate (FPR) indicates the per- attack. The results demonstrate that HODA is very effective against model extraction attacks, and it outperforms PRADA by large margin. Since HODA does not rely the distance between samples, it can detect knockoff Net attacks that use natural samples. HODA also has better performance on jacobian-bas basee attacks. The runtime and the number of samples that need to be stored by PRADA depend ertheless, for reum, = 500 and CIFAR 10 target classifier, the average runtime of PRADA for each user is 0.47 seconds (prediction time not included) for each user on average. For each user, the average runtime of HODA is 0.0012 seconds (prediction time not included), and it only stores a vector with size 11 representing a hardness degree histogram. Although HODA requires the predictions of 11 models to calculate the hardness degree of each sam- ple, there no sequential relationship between models, and they can predict in parallel, so HODA does not increase the prediction time of target models. Appendix I indicates the Pearson distance histogram of benign users and adversaries for all model extraction attacks. Appendix H introduces HODA-5 that uses five subclassifiers to calculate the hardness degree of samples. Table 6 shows the performance of HODA-5 against various model extraction attacks. Table in Appendix L reports the accuracy of the K.Net attacks' surrogate classifiers for defended adversary by HODA.
centage of benign users' sample sequences wrongly detected
the attack. Nev-
Tesla K80 GPU, and 471 samples are stored
5.1 TRANSFER LEARNING
Transfer learning is a machine learning technique that initializes the parameters of the target task classifier us- ing the parameters of a pre-trained source task classi- fier. We train two new target classifiers CUB200 and Caltech256 datasets using transfer learning (details of datasets in Appendix A). The training process new tar-
Table 4: The detection rate and False Positive Rate (FPR) of HODA against K.Net ILSVRC12 attack.
Ditection %)
Target Midel
nam,
6
(PR(S)
K.Ne; ILSV&C12
get classifiers is the same as CIFAR 10 and CIFAR 100 tar-
CUB200
0.973
0.393
97.50
100
get classifiers (Section 4.1). We initialize the parameters of target classifiers from pre- pre-trained ImageNet Deng
Caltech256
301
0.694
u01
9508
100
0,152
001
100
al. (2009) classifier and train all layers of target clas-
sifiers. Orekondy et al. (2020) indicate that jacobian-based model extraction attacks have very poor performance on high dimensional datasets. Thereby, we only evaluate the performance of target classifiers against K.Net ILSVRCI2 attack. K.Net ILSVRCI2 is the Knockoff Net attack that uses ILSVRC12 dataset as the surrogate classifier's training set. The budget of K.Net ILSVRCI2 is 50000, and the output of target classifiers is the entire probability vector. The accuracy of CUB200 target classifier and its surrogate classifier is 73.7% and 59.3%, respectively, and the accuracy of Caltech256 target classifier and its surrogate classifier is 77.2% and 72.2%, respectively.
Figure depicts the hardness degree histogram of CUB200 and Caltech256 test sets on the associ- ated target classifier and also the hardness degree histogram of K.Net ILSVRC12 samples for both
8

---

Detection Rate (56)
Detecttion Rate (50
50 100 DIC
50
500
4%
100
4K
(a) P. 25%
(b) P. 50%
(c) P. 75%
(d) P. 90%
JBDA JBRAND Ket CIFARX K.Ner. TIN Tarpott model:
CIFARIO
Target modell: CIFAR100
Figure 5: The detection rate of HODA for various percentages of normal samples Pn over different
values of num...
target classifiers. The figure demonstrates that the majority number K.Net ILSVRCI2 attack samples are hard (hardness degree > 70), and the number of easy samples (hardness degree < 30) is very small. We replicate the experiment of the previous section to evaluate the performance of HODA against K.Net ILSVRC12 attack with the same parameters. Table 4 shows the performance of HODA against K.Net ILSVRC12 attack on both target classifiers. The results demonstrate that even the starting point of target classifiers' parameters is random, HODA is very effective in detecting K.Net ILSVRC12 attack.
6 DISCUSSION ON ADAPTIVE ADVERSARY
An adaptive adversary who is aware of HODA must send her queries based the hardness degree histogram normal samples to evade HODA. We consider two scenarios for an adaptive adversary to conduct model extraction attacks. In the first scenario, the adversary has access to normal samples, and she only can use synthetic or semantically similar samples to extract the target model. There are two reasons why such attacks are hard conduct. First, the adversary needs samples with various degrees of hardness; however, since the adversary has no access to the target classifier, she can not determine the hardness degree of her samples for the target classifier. Second, the adversary has no access to the histogram of normal samples to generate her samples based on it.
In the second scenario, we assume the adversary has access to a limited number of normal samples, and she can use normal samples to make her hardness degree histogram more similar to the hardness degree histogram of normal samples. To evaluate HODA this scenario, we suppose that the adversary has access to 1000 normal samples from Suser and she sends sample sequence of which P. is filled by normal samples, and the rest is filled by model extraction attack samples. Notably, when the number of normal samples in the sequence exceeds 1000, the adversary sends duplicate normal samples. is important to note that the cost of attack is increased by a factor of (-(P./100) in this scenario. Figure 5 shows the detection rate of HODA for various Pa over different values of num,. The palse-positive rate of all experiments is less than 0.2%. The figure demonstrates that increasing num, improves the detection rate of HODA. Except for K.Net attacks on CIFAR10 10 target classifier in P, 90%, HODA can detect all attacks with a high success rate by increasing num,- Due to the dataset limitation, we can not evaluate HODA for num,, > 4000. However, detection rate HODA against K.Net attacks on CIFARIO target classifier in P. = 90% will be improved for num, 4000. Altogether, think the main challenge of an adaptive adversary to evade HODA to collect easy samples, which are very rare in out-of-distribution samples based on experiments.
think the
CONCLUSIONS
This paper demonstrates that the hardness degree of samples is important in trustworthy machine learning. We investigated the hardness degree of samples and demonstrated that the hardness degree histogram of model extraction attack samples is different from the hardness degree histogram of nor- mal samples. Using this observation, to detect sample sequences model extraction attacks. HODA can detect the sample sequences of model extraction attacks with high success rate by only monitoring 100 samples of attacks.
Hardnees-OOienttd proposed Hardness- Oriented Detection Approach (HODA)

---

REFERENCES
Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. 27th USENIX Security Symposium, pp- 1615-1631, August 2018.
B. G. Atli, S. Szyller, M. Juuti, S. Marchal, and N. Asokan. Extraction of complex dnn models: Real threat or boogeyman? In Engineering Dependable and Secure Machine Learning Systems, pp. 42-57, 2020.
R. Baldock, H. Maennel, and B. Neyshabur. Deep learning through the lens of example difficulty. CoRR, abs/2106.09641 2021. URL https: arxiv. org/ 22102006662 09647.
A. Barbalau, A. Cosma, R. T. Ionescu, and M. Popescu. Black-box ripper: Copying black-box models using generative evolutionary algorithms. In Advances in Neural Information Processing Systems, volume 33, pp.
2020.
L. Batina, S. Bhasin, D. Jap, and S. Picek. CSI NN: Reverse engineering of neural network architec- tures through electromagnetic side channel. In 28th USENIX Security Symposium, pp. 515-532, August 2019.
N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
V. Chandrasekaran, K. Chaudhuri, I. Giacomelli, S. Jha, and S. Yan. Exploring connections between active learning and model extraction. In 29th USENIX Security Symposium, pp. 1309-1326, 2020.
K. Chen, S. Guo, T. Zhang, X. Xie, and Y. Liu. Stealing deep reinforcement learning models for fun and profit. Proceedings of the 202/ ACM Asia Conference on Computer and Communications Security, pp. 307-319, 2021.
A. Coates, A. Y. Ng. and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15, pp. 215-223, 2011.
Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proceedings of the 37th International Conference on Machine Learn- ing. pp. 2206-2201221 2020.
R. C. da Silva, R. F. Berriel, C. Badue, A. F. de Souza, and T. Oliveira-Santos Copycat CNN: stealing knowledge by persuading confession with random non-labeled data. In 2018 Interna- tional Joint Conference on Neural Networks, pp. 1-8, 2018.
J. Deng, W. Dong, R. Socher, J. K. L.. and F. F. Li. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 248-255 255, 2009.
J. Frankle, D. J. Schwab, and A. S. Morcos. International Conference on Learning Representations, 2020.
The early phase of neural network training. In 8th
J. J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, 2015.
G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. 2007.
G. Hacohen, L. Choshen, and D. Weinshall. Let's agree to agree: Neural networks share classifi- cation order on real datasets. In Proceedings of the 37th International Conference on Machine Learning, pp. 3950-3960, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
X. He, Jía, M. Backes, N. Z. Gong, and Y. Zhang. Stealing links from graph neural networks. In 30th USENIX Security Symposium, August 2021.
10

---

D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and ples in neural networks. In 5th International Conference on Learning Representations, 2017.
exam-
Hong, M. Davinroy, Y. Kaya, S. N. Locke, R., K. Kulda, D. Dachman-Soled and Dumitras. Security analysis of deep neural networks operating in the presence of cache side-channel attacks. CoRR, abs/1 abs/1810.. 810.03487, 2018. URL http: aaaiiiiiiiv wwtpaatw.vvaaa..aaa.........l.tt h/gg///b//// /11011111111 10110.000 03487.
G. Huang. Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2261- 2269, 2017.
M. Jagielski, N. Carlini, D. Berthelot, A. Kurakin, and N. Papernot. High accuracy and high fidelity extraction of neural networks. 29th USENIX Security Symposium, pp. 1345-1362, 2020.
H. Jia, C. A. Choquette- defense against model extraction. In 30th USENIX Security Symposium, August 2021.
V. Chandrasckaran, and N. Papernot. Entangled watermarks as
M. Juuti, S. Szyller, S. Marchal, and N. Asokan. PRADA: protecting against DNN model stealing attacks. In JEEE European Symposium on Security and Privacy, pp. 512-527, 2019.
Kariyappa and M. K. Qureshi. Defending against model stealing attacks with adaptive misin- formation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 767-775, 2020.
Kariyappa, A. Prakash, and M. Qureshi. Maze: Data-free model stealing attack using zeroth- order gradient estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13814-13823, 2021a.
S. Kariyappa, A. Prakash, and M. K Qureshi. Protecting dnns from theft using an ensemble of diverse models. In International Conference Learning Representations, 2021b.
M. Kesarwani, B. Mukhoty, V. Arya, and S. Mehta. Model extraction warning in mlaas paradigm. In Proceedings of the 34th Annual Computer Security Applications Conference, pp. 371-380, 2018.
K. Krishna, G. Singh Tomar, A. P. Parikh, N. Papernot, and M. lyyer. Thieves on sesame street! model extraction of bert-based apis. In 8th International Conference Learning Representa- tions, 2020.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. 2015.
T. Lee, B. Edwards, I. M. Molloy, and D. Su. Defending against neural network model stealing attacks using deceptive perturbations. In 2019 IEEE Security and Privacy Workshops, pp. 43-49, 2019.
S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. 6th International Conference on Learning Representations, 2018.
D. Lowd and C. Meek. Adversarial learning. In Proceedings of the eleventh ACM SIGKDD inter- national conference Knowledge discovery in data mining, pp. 641-647, 2005.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, 2018.
K. Mangalam and V. Prabhu. Do deep neural networks learn shallow learnable examples first? In Workshop on Identifying and Understanding Deep Learning Phenomena at 36th International Conference Machine Learning, 2019.
T. Orekondy, B. Schiele, and M. Fritz. Knockoff nets: Stealing functionality of black-box models.
In JEEE Conference on Computer Vision and Pattern Recognition, pp. 4954 9954-4963. 2019.
T. Orekondy, B. Schicle, and M. Fritz. Prediction poisoning: Towards defenses against DNN model stealing attacks. In 8th International Conference Learning Representations, 2020.

---

S. Pal, Y. Gupta, A. Shukla, A. Kanade, S. Shevade, and V. Ganapathy. Activethief: Model extrac- tion using active learning and unannotated public data. Proceedings of the AAAI Conference on Artificial Intelligence, pp.
Apr. 2020.
S. Pal, Y. Gupta, A. Kanade, and S. K. Shevade. Stateful detection of model extraction attacks. CoRR, abs/2107 107.051666666 2021. URL https: pesttss:/arxxxe //arxvv/aiiiv////aa http/aiiiviggaaaaa///// org/abs/ aaa.aass/ 2217 05166,
N. Papernot, P. D. McDaniel, I. J. Goodfellow, S. Jha, Z. Berkay Celik, and A. Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference Computer and Communications Security, pp. 506-519, 2017.
I. Pliushch, M. Mundt, N. Lupp, and V. Ramesh. When deep classifiers agree: Analyzing relations between learning order and image statistics. CoRR, abs/2105.08997, 2021. URL https: / arxiv org/ alos/ 2205..00.0 08997.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510 4520, 2018.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, pp. 3-18, 2017.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, 2014.
Szyller, B. Atli, Marchal, and N. Asokan. DAWN: dynamic adversarial watermarking of neural networks. pp. 4417-4425, 2021.
F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Stealing machine learning models via prediction apis. In 25th USENIX Security Symposium, pp. 601-618, 2016.
J. B. Truong, P. Maini, R. Walls, and N. Papernot. Data-free model extraction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4771 2021.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report
1-001, California Institute of Technology, 2011.
B. Wang and N. Z. Gong. Stealing hyperparameters in machine learning. In 2018 IEEE Symposium on Security and Privacy, pp. 36-52, 2018
M. Yan, C. W. Fletcher, and J. Torrellas. Cache telepathy: Leveraging shared resource attacks to learn DNN architectures. In 29th USENIX Security Symposium, pp. 2003-20222 2020, August 2020.
H. Yu, K. Yang. T. Zhang, Tsai, T. Ho, and Y. Jin. Cloudleak: Large-scale deep learning models stealing through adversarial examples. In 27th Anmual Network and Distributed System Security Symposium, 2020.
Zhang, Z. Gu, Jang, H. Wu, M. P. Stoecklin, H. Huang, and I. Molloy. Protecting intellee- tual property of deep neural networks with watermarking. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security, pp. 159-172, 2018
Z. Zhang, Y. Chen, and D. Wagner. Seat: Similarity encoder by adversarial training for detecting model extraction attack queries. In Proceedings of the 14th ACM Workshop Artificial Intelli- gence and Security, pp. 37-48, 2021.
Y. Zhu, Y. Cheng, H. Zhou, and Y. Lu. Hermes attack: Steal DNN models with lossless inference accuracy. In 30th USENIX Security Symposium, August 2021.
12

---

DenseNet121
RosNet18
MobileNet
100
100
20
20
20
10,0]
(10. (0)
dor
ja,0)
toj
ja0 [a0-30)
(40 (40.49)
(50,00)
en)
(ro.70) (o0.00)
1000 (o)
(n,0] dign in de Mesnn
top on)
yu (n sode hume sng
(a) CIFARIO
80
80
60
50
40
40
40
20
20
20
[0.]
[1019)
(2129)
(30,8)
(40.9)
(89.99)
(e0.99)
(90,99)
(90
[09]
(1019)
(2)259
(30.5)
(40.6)
(82.59)
(40.99)
(90,99)
(88.9)
(92)99)
[ao]
(10.9)
(2725))
(30.9)
(40.9)
(40,99)
(40.99)
190779)
(80.89
190399)
Hardness Degree Range
Hardness Degree Range
Hardness Degree Range
(b) CIFARI00
Accuracy(%}
Misclassification Rato(5) - Data Percentago($)
Figure 6: Blue and red bars show the percentage of test samples in each range of hardness de- grees, which are correctly or wrongly classified, respectively. For each range of hardness degrees, Data Percentage indicates the percentage of CIFAR1O and CIFAR100 test samples whose hardness degrees are in that range.
A DATASETS
CIFARIO Krizhevsky (2009): CIFAR-I0 dataset consists of 60K 32 x 32 color images in 10 classes, including airplanes, cars, birds, cats, deer, dogs, frogs. horses, ships, and trucks. It has 6K images per class, where 5K images in the training set and IK images is in the test set.
CIFARI00 Krizhevsky (2009): CI0AR100 dataset consists of 60K 32 x 32 color images in 100 classes. It has 600 images per class, where 500 images the training set and 100 images is in the test set.
Tiny ImageNet Le Yang (2015): TinylmageNet is a subset of ILSVRCI2 Deng et al. (2009) dataset, and contains 200 image classes. It has 500 training samples and 50 test samples for each class. The size of images 64 x 64. We resize all images to 32 32.
CUB200 Wah et al. (2011): CUB200 dataset contain 200 classes of bird categories. It consists of about 6K training and about 6K samples. The size of images 224 224.
Caltech256 Griffin et al. (2007): Caltech256 dataset contain 256 classes of common objects cate- gories. It consists of about 24K training and about 6K test samples. The size of images is 224 x 224.
ILSVRC12 Deng et al. (2009): ILSVRC12 uses subset of ImageNet and consists of 1.2 million training images, 50,000 validation images, and 100,000 test images. The dataset has 1000 classes and the size of images is 224 224.
STL10 Coates et (2011): STL10 dataset consists of 13K 96 color images in 10 classes, including airplanes, cars, birds, cats, deer, dogs, monkeys, horses, ships, and trucks. It has 1.3K images per class, where 0.5K images is in the training set and 0.8K images is in the test set. We resize all images to 32 x 32.
B RELATIONSHIP BETWEEN THE ACCURACY OF CLASSIFIERS AND HARDNESS DEGREE OF SAMPLES
To assess the relationship between the hardness degree samples and the misclassification rate, compute the hardness degree of CIFAR10 and CIFAR100 test samples and then partition them into ten groups based on their hardness degree. is important to note that the number of samples in each group is different. Afterward, we calculate what percentage of samples in each group is
13

---

classified incorrectly. Figure demonstrates that the misclassification rate is increased by increasing the hardness degree of samples. In other words, there is a strong positive correlation between the hardness degree of samples and the misclassification rate. The figure indicates the percentage of samples in each hardness degree range by a green curve.
For example, the hardness degree of 40.65% of CIFARIO test samples (4065 samples) is in the range [0,9] for MobileNet classifier, from which 99.88% is classified correctly, or the hardness degree of 7.4% of CIFARIO test samples (740 samples) in the range [90,99] for MobileNet classifier, from which 55.27% is classified correctly. More than 99% and 95% of samples being learned in the first 30 epochs (hardness degree < 30) are correctly classified in CIFARIO and CIFAR test samples, respectively. On the other side, less than 55% and 36% of samples being learned in the last 10 epochs (hardness degree 2 90) are correctly classified CIFAR1O and CIFAR 100 test samples, respectively.
C HARDNESS TRANSFERABILITY
In this section, we indicate that the hardness of samples is relatively transferable among vari- ous classifiers. We use three classifiers created in Section 4.1 and new ResNet18 classifier in this experiment. Table 5 displays the Pear- son correlation coefficients between hardness degree of CIFARIO and CIFARIO 100 test sam- ples for various pairs of classifiers. The re- sults demonstrate a positive and strong corre- lation between the hardness degree of samples for various pairs of classifiers. Therefore, the hardness of samples is relatively transferable between different classifiers. On the other side, it implies that the hardness degree of samples is relatively independent of the architecture of classifiers.
Table 5: Pearson correlation coefficients between hardness degree of CIFARIO and CIFAR100 test samples for various pairs of classifiers.
Pearson Correlation Coefficient
CIFARIO
CIFARI00
BesNee11-RResNell ResNet18- De1seNet121
0.784
0.687
0,775
0.685
ResNet18- M--Miblleee
0.765
0.688
MaaaeNecl2IMMobileMe
0.769
0.706
D DETAILS OF MODEL EXTRACTION ATTACKS
Jacobian- Dataset Augmentation (JBDA) Papernot al. (2017): The goal of JBDA attack is to increase the fidelity of the surrogate classifier to the target classifier in order to produce ad- versarial examples for the target classifier in the black-box setting. The authors assume that the adversary has access to limited number of normal samples called seed samples. JBDA augment seed samples using adversarial examples to improve the performance of surrogate model. The aug- mentation process is conducted in multiple rounds. In the first round, surrogate training set X, is initialized by seed samples, and surrogate model fs is trained on X,, In the next rounds, sample set S with size K. is randomly selected from X,, and for each x E S, adversarial example x' is created using the following equation:
x' = x + (
(6)
where À is step size and J is the Jacobian function. Afterward, new adversarial examples are labeled by the target model, and they are added to X.. Lastly, surrogate model Js trained on X,. The attack is implemented with À 0.1 and x 2000. The seed samples selected from the test set of datasets. We use 500 (50 for each class) and 1000 (10 for each class) samples of CIFAR 10 and CIFAR100 test sets for seed samples, respectively.
Jacobian- Random Target (JBRAND) Juuti et al. (2019): The goal of JBRAND is to im- prove the performance of JBDA. It perturbs each sample in multiple iterations to generate more powerful adversarial examples and generates targeted adversarial examples with random targets. We generate three adversarial examples with random targets for each sample and the same seed samples as JBDA. Each sample is perturbed in five iterations with 725x5' mented with À
61 The attack is imple-
64 and 2000.
255
Knockoff Net (K.Net) Orekondy et al. (2019): Knockoff Net attack uses large public datasets that semantically similar to the target model's training samples create the surrogate model's 's train-
14

---

JBDA
JBRAND
K.Net CIEARX
K.Net TIN
10K
# of samples
8K
6IC
8K
4K
4K
2K
20
40
60
60
100
20
40
60
60
100
20
40
60
100
20
40
60
100
(a) CIFARI0
10K
10K
6K
* of samples
8K
6K
6K
4K
4K
2K
20
40
60
80
100
20
10
80
100
20
40
60
100
0
20
40
80
100
Hardness Degree
Hardness
Degree
Hardness Degree
Hardness Degree
(b) CIFARIO0
Figure 7:
The hardness degree histograms of samples of four various model extraction attacks on CIFAR10 and CIFARIO 100 target classifiers. The budget of model extraction attacks is 50000. The architecture of target classifiers DenseNet121.
0.5
asmplex
80
20
Hardooee
Hardroess
a
Figure 8: (a) Visualization of CIFARI00 test samples. (b) Hardness of CIFAR 100 test samples for CIFAR100 classifier. (c) Hardness of CIFAR100 test samples for CIFARIO classifier.
ing set. has adaptive and random strategies to select the surrogate classifier's training set, which both use semantically similar samples. Since the adaptive strategy has very marginal benefits, we only consider the random strategy to implement this attack. K. Net randomly selects a subset of a public dataset and labels them using the target classifier create Xg. Finally, it uses X, to train surrogate classifier fs-
E DENSENET 121 TARGET CLASSIFIERS
Figure 7 shows the hardness degree histograms of samples of four model extraction attacks on CIFAR1O 10 and CIFARIO0 100 target classifiers. The architecture of target classifiers is DenseNet1211 As seen in Figure 7, the hardness degree histogram of model extraction attack samples is distinguishable from the hardness degree of normal samples (Figure 1) for DenseNet1 21 target classifiers.
VISUALIZATION OF CIFAR 100 SAMPLES AND THEIR HARDNESS
Figure displays a two-dimensional visualization of CIFAR100 test samples using t-SNE. Figure 8a uses the logits of CIFAR 100 classifier to visualize CIFARI00 100 test samples, and the color of each sample determined by its label. Figures 8b and 8c show the hardness degree CIFAR100 test samples for CIFARION 100 and CIFAR10 target classifiers, respectively.
15

---

G HODA ALGORITHM
Algorithm Hardness Degree Computation
Inputs: sample and / is a sequence subelassifiers
Outputs: degree the bardness degree of sample
1: function GETHABBDEESDEGGEEEE Fauheif)
2: labe! <- None
3: for + -,,te((Foatet)) do
4: pred_werter t- Fautets (1) // Faubets [8] the xh suhclassifier sequence Foubel
5: pred label
vector)
6: if predJabel & label then degree =
7:
8:
label pred
9:
degree
10:
end function
Algorithm 2 Hardness- Oriented Detection Approach (HODA)
Inputs: SHODA a of normal samples, nam,, is size sample sequences, num. the number sample sequences, Faubely subclassifier sequence lurgel model, NewQuery y the newest query being received target model,
UserID is the identifier of owner of New-Quer..
Outputs: H,, histogram of normal samples, is the attack detection threshold, Attack Alarm declares occurence attack.
1: function PEARSONDISTCH, H.))
2: return PD(H,, Sam(H.)) (m.))m(HH))
3:
end function
4: function HODAINITAALPAATTNNSSOOO... aNm,, numbee F Foubelf)
5: WistSet +-
6:
for + 0, num do
7: aeg Randomly select num, samples SHOPA
8: Hist t- 0
9:
sng
10:
HD GotHardnssDDerood,, Faubeif)
11:
H) +1
12:
HistSet HistSot U Hist
13:
Нu e- Aeg(HistSet)
14:
Dist
15:
for Hist in HistSet do
Diet append PHARSONDIST(H,, Hiat))
Moz (oz(tist)ss))
18:
return H..8
19:
end function
20:
function HIODA(NewQuerg UserID, Ha, num., (anbeig)
21:
Attack
False
22:
H., + GetUssctHioogrmm/UU eer
23:
HD <- DerFaaddesDDerreeNeeneeeeeee Foubelf)
24:
M]
25:
if I.Summ (.,) - Num, then
26:
PEARRONNDIST(H, H.)) >>>>>>>>>>>>
27:
AttackAAlarm +- True
28:
SaveUserHistogam(Ef H., I
29:
AtlackAlarm
30:
end function
H HODA-S (FIVE SUBCLASSIFIERS)
In Section 5, HODA uses 1l subelassifiers calculate the hardness degree of sam- ples. This section introduces HODA-5, which uses five subclassifiers (Fsubelf = < f1,, f39 f29 f50 f7" , f >) to calculate the hard- ness degree of samples. Hence, the domain of hardness degree is in the range [0.4]. Table 6 shows the performance of HODA-5 against model extraction attacks. The table demon- strates that although HODA-5 needs to monitor more samples (num, = 200) to reach the per- formance of HODA (11 subclassifiers) against K.Net attacks, still very effective against model extraction attacks.
Table 6: The detection rate and False Positive Rate (FPR) of HODA-5 against four various model ex- traction attacks CIFARI0 and CIFAR100 tar- get classifiers.
Dlsuction of AttudeeS)
FFR(S)
K.Ne
KNN::
CHPAAO
0.120
0.01
100
100
75.87
75.54
HODDA-5
0.01
0018
0.07
59
$9.5
CHUNRCO
50
11
001
90.17
100
30.0
8063
FX00AA4
0,000
0.01
16

---

JBDA
JBRAND
KeNet CIFARX
K.Net TIN
STL10
10K
# of 1oees
0.5
CIFARIO
10k
of 1oers
8K
0K
3 o 1800
25
PD(H,, H.)
PD H.)
PD(H,,. II.)
PD( H.)
PD(I,, II))
(b) CIFARIO0
Benign user Adversary a STLI0 User
Figure 9: The histogram of Pearson distance between H₂ and 10000 benign users' hardness degree histogram and Hn and 10000 adversaries' hardness degree histogram for various attacks. STL10 users are benign users for CIFARIO target classifier and are adversaries for CIFAR target classi- fier.
I PEARSON DISTANCE HISTOGRAM
HODA creates hardness degree histogram for each user, called H,, and calculates Pearson distance between Hy and normal hardness degree histogram Hn (PD(Hn, H). In section 5, we simulated 10000 benign users and 10000 adversaries for each attack. Figure 9 indicates the histogram of Pear- son distance between H,, and benign users' hardness degree histogram and also H,, and adversaries' hardness degree histogram for num, 100. Notably, Pearson distance is in the range [0,2]. For CIFAR 10 target classifier, Pearson distance between H7, and hardness degree histogram of all 10000 benign users is less than 0.2, and as seen in Figure 9, Pearson distance between H,, and hardness degree histogram of all 10000 adversaries of each attack is more than 0.2. For CIFAR 100 target classifier, Pearson distance between H, and hardness degree histogram of all 10000 benign users less than 0.35, and Pearson distance between H,, and hardness degree histogram of all 10000 ad- versaries of each attack is more than 0.35. However, the confidence HODA on CIFAR100 users is less than CIFARIO users. We also consider 10000 STL10 users. STLI0 users randomly select their samples from STLIO dataset (details Appendix A). STL dataset has been inspired by the CIFARI0 10 dataset, and its images are obtained from the ImageNet dataset. It has the same classes as the CIFAR1O dataset, except instead of the frog class, it has a monkey class. We remove monkey images from STL10 dataset. Since classes of modified STL10 dataset are a subset of CIFAR10 classes, STL.00 users use in-distribution samples for the CIFAR10 10 target classifier. However, similar to K.Net CIFARX attack CIFAR 100 target classifier, STL users ples for CIFARION 100 target classifier. Since we suppose that only adversaries use out-of-distribue samples to extract target model, STL10 users are adversaries for CIFARIOO 100 target classifier and are benign users for CIFARI0 target classifier. The detection rate of HODA for STL10 users is 100% for CIFAR100 target classifier and 5.27% for CIFAR1O target classifier num, 100. Hence, the dulse-positive rate of HODA for STL10 users, which are considered benign users, is 5.27% for CIFAR target classifier. By choosing a larger delta value (e.g., = 0.25), the false-positive rate of HODA for STL.10 users can be reduced to almost zero with almost no change in the detection rate of attacks.
-istr--iistiitttion sam-
J ADVERSARIAL EXAMPLES (AEs)
Adversarial examples (AEs) are maliciously crafted inputs that cause the target classifier to misclas- sify them. There numerous methods to generate adversarial examples such L-BFGS Szegedy
17

---

FGSM
C&:W
PCD
AA
2,500
1,000
1,000
1,000
# of samples
800
600
1,500
600
600
400
1,000
200
20
600
1.00
0
40
60
80
100
201
40
80
100
20
401
50
100
(a) CIFARIO
2,500
1,000
1,000
of samples
2,000
800
000
1,500
600
600
400
1,000
400
400
*
200
60
100
40
40
Hardness Degrec
Hardness Degree
Harduess Degroc
Hardness Degree
(b) CIFARI00
Figure 10: The hardness degree histograms of samples of four various adversarial example attacks on CIFARIO and CIFAR 100 target classifiers. Each attack uses 10000 natural samples in the test set associated with the target classifier dataset to create 10000 adversarial examples.
al. (2014), FGSM Goodfellow al. (2015), C&W Carlini Wagner (2017), PGD Madry et al. (2018), and AutoAttack (AA) Croce & Hein (2020). To investigate the hardness of adversarial exam- ples, we use FGSM (e = 0.1), C&W (L2), PGD (e = 8/255, a = 3/255), and AA (Loo, = 8/255) attacks to generate adversarial examples CIFAR10 and CIFARIO0 test sets. The adversarial examples are created in the white-box setting, and they are untargeted. Figure indicates the hard- ness degree of adversarial examples CIFARIO and CIFAR 100 target classifiers. Although the distance between normal samples and adversarial examples is very small, Figure 10 demonstrates that the hardness degree histograms of adversarial examples (Figure I). Most adversarial examples generated by FGSM and C&W are harder than adversarial examples generated by PGD and AA. We think this because the size of perturbations added by FGSM and C& W is larger than PGD and AA. An intriguing observation is that almost all adversarial examples being generated by PGD are hard (hardness degree 70). AA has relatively the same behavior, and the number of its hard adversarial examples is very small.
very different from normal samples
K PERFORMANCE ANALYSIS OF MODEL EXTRACTION ATTACKS
To give new insight into model extraction attacks, we investigate the performance of model extrac- tion attacks on normal samples with various levels of hardness. For this purpose, the test sets of CIFAR10 and CIFAR100 datasets partitioned into 10 hardness groups based hardness de- gree samples. The hardness group consists of samples that their hardness degree in range i 10, + 1) 10). Hence, the first hardness group consists of the easiest samples, and the last hardness group consists of the hardest ones. Figure 11 shows the accuracy and the fidelity of attacks over 10 hardness groups when the output of target classifier is the entire probability vector. The results demonstrate that the accuracy and the fidelity of all attacks are decreased as the hardness of samples is increased. We know from Figure 6 that the accuracy of target classifiers is decreased by increasing the hardness of samples. Figure 11 indicates the surrogate classifiers also have the same behavior.
The results demonstrate that the distance between the accuracy of target classifiers and surrogate classifiers (specially K.Net attacks) increased by increasing the hardness of samples in the first hardness groups. However, the accuracy of the K.Net surrogate classifiers approaches the accuracy of target classifiers on the last two hardness groups. To investigate this observation, Figure shows the percentage of samples being classified correctly by both surrogate classifier and target classifier for all attacks over various hardness groups with dashed lines. The results indicate that all samples correctly classified by surrogate classifiers also correctly classified by target classifiers in the first two hardness groups. However, by increasing the hardness samples, the surrogate classifiers correctly classify some samples that not correctly classified by the target classifier,
18

---

CIFAR10
CIFARIO
CIFAR100
CIFAR100
100
1.00
100
Aoe(3))
Fid(% (%6
90
8 8
(%)
890
60
80
Acc
80
8
40
20
8
Hardnes Group
Hardness Group
Hardness Group
Hardness Group
fa JBDA
JBRAND
f:: NeN CIFARN
Ja: K.Not TIN--- Correctly classified by both and
The aceuracy and the fidelity of four surrogate classifiers over various hardness groups. The test set of each dataset is partitioned into 10 hardness groups so that hardness group consists of the easiest samples and hardness group 10 consists of the hardest samples. The dashed lines indicate the percentage of samples being correctly classified by both target classifier fr and surrogate classifier fa.
Figure 11:
and the number of such samples is increased by increasing the hardness of samples. Jagielski et al. (2020) demonstrate that labels from the target classifier are more informative than dataset labels. We think the information in the labels that come from the target model causes the surrogate classifiers to correctly classify hard samples that are not correctly classified by the target classifier.
The fidelity of all surrogate classifiers is decreased by increasing the hardness of samples, which means that the disagreement among surrogate classifiers and target classifiers is raised on harder samples. An intriguing observation is that the fidelity of surrogate classifiers to the target classifiers on correctly classified samples by target classifiers is much more than wrongly classified samples.
L THE ACCURACY OF SURROGATE CLASSIFIESS OF DEFENDED ADVERSARIES
Table shows that HODA detects all simulated adver- saries by only monitoring 100 samples (nwm, = 100) of each attack. Hence, adversaries can use the prediction of at most 100 samples to train the surrogate classifier. Ta- ble 7 reports the accuracy of K.Net surrogate classifiers on CIFAR10 and CIFAR 100 test sets when the adversary only uses 100 samples of attack to train surrogate classi- fiers. Since simulated adversaries randomly select their samples from 50000 samples of each attack, we simulate ten adversaries for each experiment and report the aver- age and standard deviation of the accuracy of ten trained surrogate classifiers for each attack. The training pro- cess and architecture of surrogate classifiers are similar to target classifiers, and the output of target classifiers is a probability vector.
Table 7: The average and standard de- viation of the accuracy of surrogate classifiers for defended adversaries by HODA.
Acc of Surrogate Classifier (%)
Je
K.Net CIFARX
KeNet TIN
CIFAR1O
17.10 +: 2.61
17.77 :: 1.76
(Ace: 94.36%)
CIFARI00
2.71 + 0.38
3.75 + 0.52
(Ace: 76.38%)
M COMPREHENSIVE RELATED WORK
Some recent studies investigate the dynamic of DNNs training process. Hacohen et al. (2020); Fran- kle et al. (2020); Mangalam & Prabhu (2019) show that DNNs learn samples that are learnable by shallow models in early epochs of training before learning harder ones. Hacohen et al. (2020) demonstrate that DNNs learn samples in both training and test sets in a similar order. Pliushch et al. (2021) discuss correlation between learning order of samples with image statisties like segment count, edge strengths, image intensity entropy, and DCT coefficient matrix. Frankle et al. (2020) indicate that the DNN-based classifiers undergo substantial changes in the first few SGD iterations. Baldock et al. (2021) demonstrate negative correlation between their measure of hardness (predic- tion depth) and learning events during training. They show that samples learned in later epochs have
19

---

higher prediction depth and confirm that neural networks learn easy samples first. In the following, we briefly review the most prominent model extraction attacks and defenses presented so far.
M.1 MODEL EXTRACTION ATTACKS
For the first time, I Lowd & Meek (2005) demonstrate the possibility of stealing simple linear ma- chine learning models through only interaction with them. Tramèr et al. (2016) show the feasibility of model extraction attacks on commercial MLaaS. Papernot et al. (2017) and Juuti et al. (2019) investigate stealing DNN-based classifiers and propose jacobian-based model extraction attacks for creating surrogate classifier in order to generate adversarial examples in the black-box setting. Chandrasekaran et al. (2020) explore the connection between active learning and model extraction attacks. They implement two query synthesis active learning algorithms to extract machine learn- ing models, such as decision trees. Jagielski et al. (2020) use semi-supervised learning methods to improve the performance of model extraction attacks. Knockoff Net Orekondy et al. (2019), Active Thief Pal et al. (2020), and Copycat CNN da Silva et al. (2018) use semantically similar dataset the target classifier's training set to create the surrogate classifier's training set. They em- play different strategies for selecting samples from attack datasets to extract more information from the target classifier. Yu et al. (2020) employ active learning. transfer learning, and a new method for generating adversarial examples to improve model extraction attacks efficiency. A line of studies Truong et al. (2021); I; Kariyappa et al. (2021a); Barbalau et al. (2020) use synthetic data to create the training set of surrogate classifiers. Although their methods do not need to have access to natural samples, they send a high number of queries to the target classifier, which makes their methods im- practical. For example, Truong et al. (2021) and Kariyappa et al. (2021a) send millions of queries to extract a CIFAR 10 target classifier. While most model extraction attacks have focused on the vul- nerabilities of image classifiers, recent studies demonstrate the vulnerability of NL. Krishna et al. (2020), Graph DNN He et al. (2021), and Reinforcement learning Chen et al. (2021) models against model extraction attacks. Another type of model extraction attack uses hardware side-channel vul- nerabilities to extract target classifier Zhu et al. (2021); Batina et al. (2019); Hong et al. (2018), 88;; Yan al. (2020). However, these attacks have a very strong threat model and suppose the adversary has access to the hardware that hosts the target classifier.
M.2 DEFENSES AGAINST MODEL EXTRACTION ATTACKS
Existing defense methods against model extraction attacks generally distribute into two branches: perturbation- and detection-based defenses. Perturbation-base defenses Lee et al. (2019); Orekondy et al. (2020); Kariyappa & Qureshi (2020) attempt to prevent adversaries from pro- ducing high-quality surrogate classifiers by adding perturbation to the output of target classifier. These methods generate the perturbation with various strategies to minimize the accuracy of surro- gate classifiers. Recently, Kariyappa et al. (2021b) proposed new defense with the same goal as perturbation-base defenses, which does not perturb the output of target classifiers. Their approach employs ensemble of diverse models to produce discontinuous predictions for samples. Proposed detection- based defenses Kesarwani et al. (2018); Juuti et al. (2019) attempt to detect the occurrence of model extraction attacks by observing successive input queries to the target classifier. Kesarwani et al. (2018) propose a method to measure adversary perceived knowl- edge from target classifier, but this method only works for Decision Tree models. PRADA Juuti et al. (2019) is the first proposed detection- -basd defense for DNN models. PRADA uses the his- togram of the minimum L2 distance among a new sample and all previous samples to detect model extraction attacks. Aside from its high computational overhead, it has been shown that PRADA is unable to detect model extraction attacks when an adversary uses natural samples Pal et al. (2020). Watermarking neural networks Jia et al. (2021); Zhang et al. (2018); Szyller et al. (2021); Adi et al. (2018) is another type of defense against model extraction attacks. These methods prove ownership of surrogate classifier instead of preventing the occurrence of model extraction attacks.
Atli et al. (2020) demonstrate that several OOD detection approaches, such as Baseline Hendrycks & Gimpel (2017) and ODIN Liang et al. (2018), have poor performance in detecting Knockoff Net attack samples. Hence, they propose a new OOD detection approach that leverages a classifier to detect OOD samples. However, their approach only rejects OOD samples, and it does not have any detection mechanism to detect adversaries. Besides, the OOD detector is trained on samples from the same distribution used by the adversary to conduct Knockoff Net attacks, which is unreal-
20

---

istic assumption in practice. Concurrent with work, Zhang al. (2021) and Pal al. (2021) propose SEAT and VarDetect to detect sample sequences of model extraction attacks, respectively. SEAT aims to detect model extraction attacks that use several similar samples to extract target model, such as jacobian-based attacks (Papernot et al. (2017); Juuti et al. (2019)). Hence, SEAT is ineffective when adversary uses natural samples that are not similar each other, such Knock- off Net attack. VarDetect uses Variational Autoencoders (VAs) and Maximum Mean Discrepancy (MMD) detect model extraction attacks. VarDetect has only been evaluated on low-dimensional datasets. Regarding that VarDetect uses VAs and MMD, it is unclear how well it performs on high- dimensional datasets. Besides, uses the ImageNet dataset to extract target classifiers trained on very structurally different datasets, such as F-MNIST and SVHN.
21


Reviews:
[{"review_id": "n1zRhSMv1D", "reviewer": "Reviewer_UQxC", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors use the ordered ensemble of checkpoints to give a numerical hardness value to image samples and argue that the distribution of benign test samples by hardness differs from attack samples. Using this, distance is computed between holdout set histogram and test time histogram. If it exceeds a threshold, an attack is detected.\n\nThe definition of hardness appears to have conceptual flaws:\n* It does not account for mis-classified samples. A mis-classified sample $x$ should intuitively be one of the most hard samples and should be assigned the highest hardness degree $m-1$ (where $m-1$ is the last epoch). Among the epoch-wise checkpoints, $x$ could get a wrong label $l$ in some epoch $e < m-1$ and the model could continue to label it as $l$ in the later epochs. So the hardness degree $\\phi_{f_t}(x) = e < m-1$, which is counter-intuitive.\n* Let $A$ be a model that is not well-trained (e.g., trained using a very small learning rate) and the original labels of the training samples do not change at all. Then all the samples will get hardness degree of $0$! In comparison, a well-trained model $B$ will have samples falling into higher values of hardness. Thus, by the hardness definition in the paper, the well-trained model $B$ would find the same set of samples \"harder\" than an ill-trained model. There does not seem to be any correlation between model accuracy on a dataset and hardness of the dataset.\n* There is no guarantee in general that model training is monotonic for each example, that is worst to better. So concluding hardness based on the latest epoch number at which its label stabilizes is not indicative of its hardness.\n\nBecause of the conceptual flaws, I am not convinced about validity of the proposed approach.\n\nThere are several limitations of the experiments:\n* The paper compares only against PRADA, which is not designed to work for attacks based on natural samples (e.g., Knockoff nets used in the paper). There are at least two defenses which are designed to detect such attacks: Atli et al. \"Extraction of Complex DNN Models: Real\nThreat or Boogeyman?\" and Pal et al. \"Stateful detection of model extraction attacks\". The authors should compare against these defenses. The paper incorrectly states that PRADA is the only defense in this space.\n* To construct benign users, the authors take a split from the original test distribution. In practice, a benign user can provide samples that could come from similar distributions and not necessarily from the same distribution. The authors should conduct experiments to check false positives for such cases, e.g., MNIST vs color MNIST.\n* The results on adaptive attacks show that HODA requires an order of magnitude more samples (50-100 to 500-1K) to detect a decent number of attacks. For the CIFAR10 model, it requires even more samples. HODA requires running each sample on multiple checkpoints, which can be expensive in the adaptive case due to increasing number of samples required for detection.\n* Appendix H gives trend over different hardness degrees for the target and extracted models. This however does not tell us the model accuracy numbers exactly. The authors should report model accuracy for these models and for models extracted when HODA is applied.\n", "summary_of_the_paper": "In this paper, the authors propose HODA (hardness oriented detection approach) to detect model extraction attacks against DNNs. They use the checkpoints during training of the model to define hardness of a sample. Hardness of a sample is the epoch from which the model assigns the same class to the sample. During inference, Pearson distance between the histograms of holdout set samples and test samples is used to detect an attack. HODA is evaluated on image datasets and models. It is shown to detect three classes of attacks and is analyzed against an adaptive attack.\n", "summary_of_the_review": "The paper is targeting a challenging problem. However, the central concept of hardness of samples as defined in the paper appears flawed. The paper has several limitations when in comes to experimental evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fqJPH2ul4k", "reviewer": "Reviewer_qQMY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points:**\n1. The method outperforms previous detection-based defense called PRADA.\n2. The method is well-described but some more intuition why it works well would be helpful. Overall, the paper is well-written.\n\n**Weak points:**\n1. The storage overhead is substantial if the method has to store checkpoints after every epoch. However, it is shown that this requirement can be alleviated by storing checkpoints only every 10th epoch and it is sufficient to maintain a high detection rate as well as the low false-positive rate of the HODA defense. Additionally, defenses usually come at a cost, so this can be justified. The authors could compare to PRADA not only in terms of storing the hardness histograms but also add the size of the saved checkpoints.\n2. The computational cost of the method is increased by a number X of saved checkpoints in comparison to the standard inference. Usually, the DNN models are large and if only a single K80 GPU is available, then HODA cannot be executed in parallel (not sufficient memory). Additionally, the GPU load + energy cost is increased by X as well.\n\n**Other points:**\n1. There are other detection-based defenses, e.g. [1] and [2].\n2. Minor inconsistent naming convention, e.g.: CIFAR10 and Cifar10.\n3. Why is MobileNet performing the best across the 3 models in Figure 2?\n4. Many transfer learning methods freeze all but the last (classification) layer. How does HODA perform in this setting (or only if few last layers are retrained)? What is the dependence between how many last layers are re-trained and the detection rate of HODA?\n5. In Appendix G, PGD examples are not hard with hardness degree **<** 70.\n\n**References**\n\n[1] Stateful Detection of Model Extraction Attacks. Soham Pal, Yash Gupta, Aditya Kanade, Shirish Shevade. https://arxiv.org/abs/2107.05166\n\n[2] Extraction of Complex DNN Models: Real Threat or Boogeyman? Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, N. Asokan. https://arxiv.org/abs/1910.05429\n", "summary_of_the_paper": "HODA is a new defense against model extraction attacks. It is a detection method that determines if a given user is adversarial based on the hardness of users' queries. For a given sample, the hardness metric is computed as the epoch number of a subclassifier after which the sample is classified correctly. The method generates histograms for in-distribution data and the histograms of each user's queries. The Person distance is computed between the histograms and if the distance is greater than a certain threshold $\\delta$ then the user is classified as malicious. The results show that the hardness degree histogram is an indicator of dataset distribution. In general, the in-distribution samples have significantly lower hardness levels than samples used by adversaries (the adversarial samples are OOD natural or synthetic data).", "summary_of_the_review": "The paper presents a new detection-based defense against model extraction attacks that performs better than the previous (directly comparable) defense called PRADA. However, the defense is expensive in terms of computation and storage usage. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KrTRlUzkC7H", "reviewer": "Reviewer_QDdd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1\\. Novelty and certain benefits\n- While there has been some recent attention to defenses via manipulation, detecting queries targeted for model stealing has receive little attention. The paper explores the latter line and demonstrates empirical improvements.\n- The defense also provides some benefits e.g., defender does not need attacker's model architecture or parameters, it can be combined with manipulation-based defense strategies.\n\n2\\. Experiments\n- I appreciate the authors comparing against multiple recent attack strategies and also evaluating in a variety of settings.\n\n3\\. Well-written\n- The paper is well-written and was easy to follow.\n\n**Major Concerns**\n\n1\\. Constraints / Severe assumptions\n- Although the results appear promising, I am concerned that these results are a product of possibly impractical assumptions and would appreciate the authors' clarification:\n- (a) examples can be precisely associated with a user $u$: what if the user $u$ performs a sybil attack and spreads the examples among multiple different user accounts? (which seems easily achievable in MLaaS scenarios.) \n- (b) to obtain hardness values of a example now requires 10-100$\\times$ as many predictions (to classify over all $f^m$ classifiers)\n- (c) distribution(benign set) = distribution (training set): but this seems highly unlikely as benign distributions can have a very specific dataset bias (e.g., objects with white-backgrounds, images under poor lighting). I would guess in this case the benign user would be considered adversaries? \n\n2\\. Hardness analysis and comparison to OOD\n- A key contribution of the paper is the concept of 'hardness' that captures some statistics of a set of examples and contrasts it with a normal set consisting of the victim's training examples. While the results using hardness appear promising, I find missing some additional analysis or discussion justifying 'hardness'. After all, the defense task is reduced to comparing two distributions and this is achievable by a variety of techniques (e.g., FIDs, Wasserstein, MMD). While the authors show that hardness is a good measure, I find lacking *why* or some in-depth analysis motivating hardness.\n- I appreciate the empirical study in Sec. 4.3. that shows hardness captures OODness of samples. This makes me wonder if there are connections to existing OOD detection approaches (e.g., ODIN ICLR '18) and if one could instead simply use OOD detection techniques (which has significantly progressed recently) to defend against model stealing attacks. \n\n3\\. Experiments / Evaluation\n- OOD baselines: Building on top of my previous concern, I would have appreciated comparison of HODA with OOD detection approaches (e.g., ODIN) or similar OOD techniques used within model stealing literature (e.g., Adaptive misinformation, Kariyappa and Qureshi CVPR '19). These seem more competitive and fairer to compare HODA with. Moreover, this seem more appealing from a defense perspective, since they typically perform example-level classification (vs. user-level) and as a result only certain examples can be rejected.\n- Table 3-4: To claim that  HODA outperforms PRADA in Table 3-4, I find overall results missing e.g., FPR vs. detection rate as a result of modifying $\\delta$, AUROC numbers. Going by the table alone, I am unsure if there exists a range of thresholds $\\delta$ where HODA performs poorly.\n- Simulating benign users: How are benign users $S_u$ simulated -- by using a held-out fraction of the victim's test set? If yes, this seems concerning as it is possible to design the approach or tune the hyperparameters and overfit to biases in the victim's distribution. One idea would be to construct the benign sample set by using images from CIFAR10 classes, but sampled from TinyImage or another dataset.\n\n\n**Minor Concerns**\n\n1\\. Large hardness values -> noisy?\n- For histograms in Fig. 1,3 wouldn't the values at higher hardness values be extremely noisy? For instance for a hardness value of 98 (at m=100 epochs), this can be achieved using only predicts at a single reference epoch (e=99). This seems to be case going by the curves where there is a big peak at higher hardness values. As a result, it might make sense to train the model instead of 200 epochs and estimate hardness values until 100.", "summary_of_the_paper": "- The paper proposes a defense 'HODA' against DNN model stealing attacks.\n- They key idea of the approach is a strategy that discriminates a set of 'benign' examples (i.e., similar to victim's data distribution) vs. 'surrogate' examples (i.e., from attacker's input distribution). \n- The strategy involves mapping an example $x_i$ (benign/adversarial) to 'hardness' value which roughly indicates at which point in the training process the intermediate model converges on prediction of $x_i$. HODA exploits a property that the hardness distribution statistics varies between benign and surrogate sets.\n- Experiments on standard datasets {CIFAR10, CIFAR100, CUB200, ...} and attacks {knockoff, JBDA, ...} indicates that HODA is able to achieve reasonable detection rates and FPR against attacks.", "summary_of_the_review": "- The proposed approach 'HODA' extends the state of defenses against model stealing attacks by exploiting an interesting property of how an attacker's distribution contrasts a benign distribution. The experimental results indicate that indeed this property can be exploited to some extent to reject malicious queries.\n- However, I am primarily concerned about (a) certain assumptions (being able to associate each example with the same user, distribution comparison) and (b) how this compares to simply performing OOD detection. In particular with (b), if the key idea to exploit that attacker's distribution is semantically different from the victim's, I would have appreciate a better comparison with OOD detection, or present some analysis/insights of the proposed 'hardness' justifying the choice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KU5J3R0FssI", "reviewer": "Reviewer_Mihr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper demonstrates that the hardness degree of samples is important in trustworthy machine learning. The authors investigated the hardness degree of samples and demonstrated that the hardness degree histogram of model extraction attack samples is different from the hardness degree histogram of normal samples.\n\nMy comments are as follows.\n\nFirst, in section 4.1, this paper states a definition of the hardness degree, which is essential in this paper. However, I didn't see any significant result from Figure 2. Also, the meaning of accuracy in the Y-axis is unclear. In the last paragraph of page 4, it is not lucid to me how the authors get the results based on this figure.\n\nSecond, in section 4.2, the authors introduce the model extraction attacks and evaluate them. Nevertheless, I am confused about how to get the attack budget and how to get these images. I think the authors should add more details about that.\n\nNext, in section 4.3, the authors calculate the hardness degree of the samples generated by attacks. However, I am wondering for the data samples that have a large hardness degree, is it easy to distinguish them from the data samples not used for span queries. As mentioned in the next section, the Pearson Distance will be very large. So, I don't think we can use the datasets that contain plenty of such samples to measure the sequences.\n\nFinally, in section 5, the authors proposed the experiment setup and evaluation. However, I would like to know more evaluation metrics in the table, not only FPR.\n\nMinor: in the last paragraph of page 7, it should FPR not PFR.\n", "summary_of_the_paper": "This paper proposes Hardness-Oriented Detection Approach (HODA) to detect the sample sequences of model extraction attacks.\n", "summary_of_the_review": "In general, I think this paper has merit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new method for detection of model extraction attacks. It is based on the intuition that typical model extraction attacks involve samples submitted by users that are harder to classify than "benign" samples submitted by users. By introducing the notion of hardness, a metric is developed for identifying malicious users submitting their samples for the purpose of model extraction. While the proposed method is original, it incurs a substantial overhead. Experimental evaluation of the proposed method also has several deficiencies, in particular, in the assessment of its overhead as well as in modeling of benign users.
Decision: Reject