Protecting DNNs from Theft using an Ensemble of Diverse Models
Authors: Sanjay Kariyappa
Atul Prakash
Moinuddin K Qureshi
Venue: ICLR 2021 Poster (2021)
Keywords: Model stealing
machine learning security

Abstract: Several recent works have demonstrated highly effective model stealing (MS) attacks on Deep Neural Networks (DNNs) in black-box settings, even when the training data is unavailable. These attacks typically use some form of Out of Distribution (OOD) data to query the target model and use the predictions obtained to train a clone model. Such a clone model learns to approximate the decision boundary of the target model, achieving high accuracy on in-distribution examples. We propose Ensemble of Diverse Models (EDM) to defend against such MS attacks. EDM is made up of models that are trained to produce dissimilar predictions for OOD inputs. By using a different member of the ensemble to service different queries, our defense produces predictions that are highly discontinuous in the input space for the adversary's OOD queries. Such discontinuities cause the clone model trained on these predictions to have poor generalization on in-distribution examples. Our evaluations on several image classification tasks demonstrate that EDM defense can severely degrade the accuracy of clone models (up to $39.7\%$). Our defense has minimal impact on the target accuracy, negligible computational costs during inference, and is compatible with existing defenses for MS attacks.

Published as a conference paper at ICLR 2021
PROTECTING DNNS FROM THEFT USING AN ENSEM- BLE OF DIVERSE MODELS
Sanjay Kariyappa
Atul Prakash
Georgia Institute of Technology
University of Michigan Ann Arbor MI. USA
Atlanta GA, USA
Saarjayyaarlyapp@gateeed
aprakash umi ch edu
Moinuddin K Qureshi
Georgia Institute of Technology Atlanta GA, USA
moin@gatech.
ABSTRACT
Several recent works have demonstrated highly effective model stealing (MS) at- tacks Deep Neural Networks (DNNs) in black-box settings, training data is unavailable. These attacks typically use some form of Out of Dis- tribution (OOD) data to query the target model and use the predictions obtained train a clone model. Such a clone model learns to approximate the decision bound- ary of the target model, achieving high accuracy on in-distribution examples. We propose Ensemble of Diverse Models (EDM) to defend against such MS attacks. EDM is made up of models that are trained to produce dissimilar predictions for OOD inputs. By using different member of the ensemble to service different queries, our defense produces predictions that are highly discontinuous in the in- put space for the adversary's OOD queries. Such discontinuities cause the clone model trained on these predictions to have poor generalization on in-distribution examples. Our evaluations on several image classification tasks demonstrate that EDM defense can severely degrade the accuracy clone models (up to 39.7%) Our defense has minimal impact the target accuracy, negligible computational costs during inference, and is compatible with existing defenses for MS attacks.
when the
INTRODUCTION
MS attacks allow an adversary with black-box access to the predictions of the target model to copy its functionality and create high-accuracy clone model, posing threat to the confidentiality of pro- prietary DNNs. Such attacks also open the door to a wide range of security vulnerabilities including adversarial attacks (Goodfellow et al., 2014) that cause misclassification, membershippinnference attacks (Shokri et 2017) that leak membership, and model-inversion attacks (Fredrikson et al.] 2015) that reveal the data used to train the model. MS is carried out using the principle of Knowledge distillation (KD), wherein the adversary uses dataset D to query the target model. The predictions of the target on are then used train clone model that replicates the target model's functionality. Since access to training data is limited in most ceal-world settings, attacks typically use some form of OOD data to perform KD. Clone models trained in this way closely approximate the decision boundaries of the target model, achieving high-accuracy on in-distribution examples.
The goal of this paper is to defend against MS attacks by creating a target model that inherently hard to steal using Knowledge Distillation with OOD data. Our key observation is the existing MS attacks (Orekondy et al. 2019a Papernot et al. 2017; Juuti et al. 2019) implicitly assume that the target model produces continuous predictions in the input space. We hypothesise that making the predictions of the target model discontinuous makes MS attacks harder to carry out. To this end, we propose Ensemble Diverse Models (EDM) to defend against MS attacks. The models in EDM are trained using a novel diversity loss to produce dissimilar predictions on OOD data. Each input query to EDM is serviced by single model that is selected from the ensemble using an input-based

---

Published as a conference paper at ICLR 2021
(a) Target Model Training data and decisions
(b) Surrogate Data used to query target medel
(c) Clone Model trained on the surrogate data
(i) Undefended
0 Surrogate
Training
ModelkA
(ii) EDM
Mo del-B
Figure I: Toy experiment comparing the efficacy of MS attacks between (i) Undefended Baseline and (ii) EDM targets. (a) Predictions and training data of the target model (b) OOD surrogate data used by the adversary to query the target (c) Clone model trained using the predictions of the target on the surrogate data. The discontinuous predictions produced by EDM make MS attacks less effective compared to undefended baseline. (Best viewed in color)
hashing function. We develop a DNN-based perceptual hashing algorithm for this purpose, which invariant simple transformations of the input and prevents adaptive attacks. Since different models in the ensemble are used to service different queries and the models are trained to produce dissimilar predictions, the adversary obtains predictions that are highly discontinuous in the input space. The clone model, when trained on these predictions, tries to approximate the complex dis- continuous decision boundary of the EDM. Our empirical evaluations show that the resulting clone model generalizes poorly on in di-diitribution data, degrading clone accuracy and reducing the efficacy of MS attacks. In contrast existing defenses that rely on perturbing output predictions, our defense does not require modifying the output and instead uses a diverse ensemble to produce discontinuous predictions that are inherently hard steal.
We illustrate the working of our defense by comparing the results of MS attacks on two target models- (i) Undefended baseline (ii) EDM trained on a toy binary-classifictiinn problem shown in Fig. The training data and predictions of the target model under attack are shown in Fig. [.. We use a set OOD points as shown in Fig. [ to query the target model and obtain its predictions. The predictions of the target model the OOD data is finally used to train the clone model to replicate the functionality of the target as shown in Fig. I.
For the undefended baseline target, which uses simple linear model to produce continuous pre- dictions, the clone model obtained by the attack closely approximates the decision boundary of the target model, achieving good classification accuracy sists of two diverse models (Model-A and Model-B) that produce dissimilar predictions on the OOD data. EDM uses input-based hash function to select the model that services the input query. As result, the target model produces highly discontinuous decisions on OOD data (Fig. [ for EDM). The clone model trained on this data fails to capture any meaningful decision boundary (Fig. [ for EDM) and produces poor accuracy on the classification task, making MS attacks harder to carry out. In summary, our paper makes the following key contributions:
in-distribution data. The EDM target
1. We propose novel Diversity Loss function train Ensemble of Diverse Models (EDM) that produce dissimilar predictions on OOD data.
2. We propose using EDM defend against model stealing attacks. Our defense creates discontinuous predictions for the adversary's OOD queries, making MS attacks harder to carry out, without causing degradation the model's accuracy benign queries.
3. We develop a DNN-based perceptual hash function, which produces the same hash value even with large changes to the input, making adaptive attacks harder to carry out.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Model Stealing Attacks: MS Attacks fall into one of three categories: (i) parameter stealing,
(ii) hyper-parameter stealing and (ii) functionality stealing attacks. Parameter stealing (Tramèr
ct al.
2016
Lowd & Meek] 2005) and hyperparameter stealing attacks (Wang & Gong 2018/01 Oh
al.
2019 focus on inferring the exact model parameters or hyperparameters used in the architec-
ture/ ture/training of the target model respectively. Our work focuses on defending against functionality stealing attacks, where the adversary's 's goal is to train a clone model that achieves high accuracy on
given task. Since the training data of the target is usually unavailable, attacks leverage alternate forms of data to query the target. Orekondy et al. (2019)); Correia-Siin from related problem domains, while other attacks (Papernot et al. [017 Juuti et al. [019 Roberts et al. 2019 Kariyappa et al. 2020) have proposed crafting synthetic examples to query the target to perform model stealing.
et al. (2018) use datasets
Model Stealing Defenses: Existing defenses that have been proposed to defend against model func- tionality stealing involve perturbing the output predictions to prevent the adversary from having liable access to the predictions of the target model. Perturbation-basee defenses modify the output of the model y f(:) produce perturbed output y' = y 5. These defenses be further sub-categorized into accuracy-preserving and accuracy-constrraiee defenses.
Preservyyn Defenses (Lee al. 2018) ensure that the class-labels of the perturbed pre- dictions y' are the same as the original prediction i.e. argmaaz(y))
argmax(yy) Since the class-labels are unchanged, the benign accuracy of the model is preserved.
contrast, accuracy- constrained defenses (Kariyappa & Qureshi] [020 Orekondy et al. (219b) allow for the usage of higher level of perturbation, trading off benign accuracy for increased security against MS attacks.
This work presents the first defense that is not reliant on adding perturbations to the predictions of the model. Instead, we aim to create an entirely new class of ensemble-based models that is inherently hard to steal with a data-limited attack, due to the discontinuous nature of the predictions produced by the model. Our method has minimal impact on the benign accuracy and offers a much higher security against MS attacks compared to single model. Additionally, our model can be used in conjunction with perturbation- based defenses to further improve robustness against MS attacks.
3 PRELIMINARIES
3.1 ATTACK: OBJECTIVE AND CONSTRAINTS
The goal of the attacker to maximize the accuracy of clone model C on a test set Drest i.c. maxée Ea [Ace(C (a; 0c), v)]. MS attacks use the principle of knowledge distillation to train a clone model. First, the attacker queries the target model using inputs {ar] 1 to obtain the target's predictions {y: Txx:). }. The labeled dataset {x;, }/} created this way can then be used to train a clone model C, completing the attack. The predictions returned from T can either be soft-label prediction, where 7 returns probability distribution over the output classes, or hard- label predictions, where only the argmax class-label returned. We consider attacks under both prediction types in this paper. Since there is fixed cost associated with querying T, assume that the number of queries is upper bounded by a query budget Q.
In most real-world settings, the adversary does not have access to the dataset used to train the target model. A data-limited attacker can use alternate forms of data to carry out model stealing attacks. KnockoffNets (Orekondy et al.) (019a) uses data from alternate surrogate dataset, which has semantic similarities with the target model and Jacobian-Basee Dataset Augmentation (Papernot et al. 2017) uses synthetic datasets to query the target model.
Data Constraints:
3.2 DEFENSE: OBJECTIVE AND CONSTRAINTS
The goal of the defense is to minimize the accuracy of the clone model i.c. min Ex.yyDi.a [Acc(C((;;e)) model or the model predictions. However, we require the defense to have minimal impact on the accuracy of the model for benign inputs to preserve the utility of the model. The defender is unaware of the kind of attack being used, the dataset used by the adversary query the target
The defense is allowed to make changes to the target
3

---

Published as a conference paper at ICLR 2021
(a) Accuracy Objective
(b) Diversity Objective
X
x,y Dyn
Dutt (auxx)
f1
f2
f1
fz
N
Figure 2: Models in EDM are jointly trained on two objectives: (a) Accuracy Objective ensures that the models make correct preditions for in-distribution data. (b) Diversity Objective encourages the models to make diverse decisions for auxiliary out-of-distribution data.
model. addition to existing attacks, the defense also needs to he effective against adaptive attacks that are tailored for the defense.
OUR PROPOSAL: ENSEMBLE OF DIVERSE MODELS
This paper proposes a novel type of model ensemble called Ensemble of Diverse Models (EDM), which is significantly of a set of diverse models, which produce dissimilar predictions for OOD data. By using an input- based hash function, EDM selects single model from the ensemble service each query. Since the models are trained to produce dissimilar predictions, EDM outputs predictions that are highly discontinuous in the input space for OOD data. The complexity of the discontinuous predictions cause the clone model trained on these predictions to generalize poorly on in-distribution examples, making MS attacks less effective. We explain how EDM models can be trained and deployed for inference in this section.
robust to MS attacks compared to single DNN model. EDM consists
4.1 TRAINING A DIVERSE ENSEMBLE
EDM leverages an ensemble of diverse models to produce discon- tinuous predictions. Let (fi) denote the ensemble mod- els used in EDM with model parameters {0.) These mod- els are jointly trained with a two-fold training objective using an in-distribution dataset of labeled examples Din and an out-of- distribution dataset of unlabeled examples Dout as explained below:
Y1
83
Minimum Angle
02
ya
32
Accuracy objective: Models in the ensemble should be trained to produce high accuracy for the examples in the training dataset of in-distribution examples Di, as shown Fig. p.. For a multi-class classification problem, let {7, f: 8,)
Cahermoe((x: Cos(A,)
}. Ji=1 be the prediction
Figure 3:
Coherence mea-
sures the cosine of the small- est angle made by any pair of vectors in a set.
probabilities of the models where (x, y) E Din is the input, la- bel pair. We use the cross-entropy loss averaged across all models:
N EN Cg) y) train the models in the ensemble to achieve high accuracy on the in on-distribution training data.
Diversity objective: The diversity objective requires the models in the ensemble produce
dissimilar predictions for examples in the out of distribution dataset Dout as depicted in Fig, 2b. If (ū. (ā: f.) \ are the output probabilities produced by the ensemble for an OOD input 5 Douts then the diversity objective requires the set of output vectors {Ø: J be misaligned. We use the coherence metric (Tropp) 2006) to measure the alignment of the output probability vectors. Coherence of a set of vectors measures the maximum cosine similarity (CS) ie. cosine of the
smallest angle, between all pairs of probability vectors in the set as shown in Fig, Coherence can be computed as follows:
coherence ((
max a.BE( abb
CS 9h)-
(1)
4

---

Published as a conference paper at ICLR 2021
Selector
Input
Perceptual Hash
#
Yout
findex
x
(())
index = (H (x)%N)
Figure 4: EDM uses input-based hash (z) to select the model that is used to service input
query z. A perceptual hashing algorithm is used to prevent adaptive attacks.
In order to increase the misalignment of probability vectors, need to reduce the value of coher- ence. Unfortunately, the mar function in Eqnl | makes coherence non-smooth, which precludes the use of first-order methods to optimize the value of coherence. To overcome this limitation, the LogSumExp function to get a smooth approximation of the maz function that allows it to be used in first-order optimization algorithms. We term the resulting loss function the diversity loss:
DivLoss ((oLoss(())))) (
loo
Σ
exp(CS((j.: Bn))
ISacDEN
Models in the ensemble are jointly trained on the combination of accuracy and diversity objectives:
N
E D.y
Σ (ce(Ú.,)) + AD Die.Loss(())))))
(3)
1=1 where ds = f((2), U. f((D.
Here Ap is a hyperparameter that dictates the ratio of importance between the diversity and accu- racy objectives. Note that the loss function described above requires access a dataset of labeled in-distribution examples Din as well not know the OOD samples that is used by the adversary a-priori, we make use of an auxiliary OOD dataset to train EDM. Similar use of auxiliary OOD datasets have been explored anomaly detec- tion literature (Hendrycks et al. 2018) to detect unseen anomalies. Our experiments in Section 5.3 suggest that the diversity objective generalizes to the unseen OOD dataset used by the adversary.
dataset of out-of-distributio examples Dout- Since we do
4.2 EDM INFERENCE
The objective of EDM is to produce predictions that are discontinuous in the input space. EDM produces discontinuous predictions by selecting a model the ensemble based on the input to service each request shown in Fig. Given input query æ, EDM uses hashing function H to compute a hash of the input, from which the index of the model that is used to service the query can be computed: index
+ (=))%N). The output prediction is given by yout Jinden (r).
4.2.1 CHOOSING THE HASI FUNCTION
To ensure that adversary is unable to launch an adaptive attack to game the hash function, we select hash function that satisfies the following properties:
Security: The hash function needs to be kept secret from the adversary to avoid the adversary from computing the hash and figuring out which model was used to service a given input.
1.
2. Trunsformation invariacce: While the hash function should map different inputs to different hash values, we want the hash value to be invariant to small transformations in the input images such scaling, rotation, translation, warping. contrast adjustment, etc. This invariance property necessary to prevent an adversary from using slightly modified versions of the original image to change the hash value and query different model. Without the invariance property, the adversary can query all the models in the ensemble and get the average prediction for given input, which is continuous in nature and allows for more effective model stealing.
Cryptographic hash function would be unsuitable for our application as they lack transformation invariance. Perceptual hashing (pHash) (Zauner 2010) class of hash functions that ensure that
5

---

Published as a conference paper at ICLR 2021
perceptually similar inputs map to the same/similar hash values by using transformation invariant features (like discrete cosine transform), making them better suited for our application. Empini- cally, we found existing pHash schemes to be less robust to various transformations of the inputs (discussed in Appendix A.2). To solve this problem, we propose a novel DNN-based perceptual hashing function that outputs the same hash value even with large changes the input. Our insight is that for any DNN that is trained on a classification problem involving natural images, the output prediction of the DNN typically invariant similarity. Given a DNN fH that is trained on a secret M-class classification problem, we can use the argmaz predictions to compute hash of the input: (xx) ar argmaz (z)). To ensure that the hash function uniformly maps inputs to different indices, we increase the output space of the hash function by subdividing each class into m sub-classes (achieving a total of M m possible
transformations of the input that retain perceptual
outputs) using the following function: Hxx) = m argnaz([n(s))+
M M=maz(((=))-11 -m] We M-1
use a small value for m (m 2) in our experiments to ensure transformation invariance is preserved. Additionally, a relatively small model can be used for fH to reduce computational overheads. The model architecture and the dataset used to train H are unknown to the adversary, making the output of the hash function hard to estimate. The security of the hash function can further be improved by training it on custom dataset that is not available in the public domain.
EXPERIMENTAL RESULTS
5.1 SETUP
Victim models and datasets: We evaluate our defense by comparing the clone accuracy achieved by various attacks on EDM and an undefended baseline model. A lower clone accuracy implies better security. We use DNNs trained on various imagc-classification datasets listed in Table 3 as the target models for performing MS attacks. For each dataset, we train two target models 1. unde- fended baseline: A single model trained on Din using cross entropy loss. 2. EDM: An ensemble of models trained on the EDM loss function (Eqn. 3). The dataset, model architecture and test accuracies
Table 1: Datasets, model architecture and accuracies of the target models.
Dataset
Model Arch.
Undel.
Accuracy (%)
MNIST FashionMNIST
Convá
99.49
99.5
Coevá
91.02
91.01
CIFAR-10
Wres1644
94.83
94:35
CIFAR-10
Wres1644
74.87
74.83
Flowers-17
ResNet-15 18
95.15
obtained by the undefended target and EDM targets are shown in Table Note that, despite the ad- ditional diversity objective, the accuracy of EDM is comparable the undefended baseline model.
We consider an adversary who is interested in stealing model functionality by training a high- alcurccuracy clone-model on the predictions of the target model. We assume that the adversary does not have access to the entirety of the training dataset that was used to train the target model. The number of queries that the adversary can make to the target model limited by the query budget Q (We use Q = 50000 for our evaluations). Additionally, we assume that the adversary is aware of the target model's 's architecture and use clone model with the same architecture as the target. We consider three recent MS attacks to evaluate our defense:
Attacks:
1. KnockoffNets (Orekondy et al., 2019a) performs model stealing by using a surrogate dataset to query the target model and training clone model C' the predictions of the target model. We use FashionMNIST, MNIST, CIFAR100, CIFAR1O, and Indoor67 (Quattoni & Torralba 2009) as the surrogate datasets for MNIST, FashionMNIST, CIFAR-10. 10. CIFAR-I00, and Flowers- 17 (Nilsback) & Zisserman 2006) models respectively. The clone model is trained using an SGD optimizer with an initial learning rate of 0.1 for 50 epochs with cosine-annealing learning-rate schedule !
2. Jacobian Based Dataset Augmentation (JBDA) (Papernot et al. 2017) uses synthetic data to query the target model. JBDA starts with small set of in-distribution "seed" examples {a:). The attack iteratively trains the clone model by performing the following steps: (i) Obtain a labeled dataset (x,, } by querying the target model (ii) Train the clone model on the labeled dataset (iii) Augment the dataset with synthetic examples x by perturbing the original input x; to change the prediction of the clone model by using the Jacobian of the loss function: 2 I4 + Bsign (V.C (C' (z,; fc) 9:)).
For Flowers-11 17, use a clone model that is pre-trained Imagenet with an initial learning rate of 0.001.
6

---

Published as a conference paper at ICLR 2021
Table 3: Comparing the clone accuracy produced by attacking undefended baseline and EDM
target model for various model stealing attacks under soft and hard label settings. Attacks targeting
EDM produce significantly lower clone accuracies compared an undefended model in most cases.
KnockoffNets
JBDA
JBDA-TR
Undef. (%)
EDM (%)
Under. (%)
EDXI (%)
Undel. (%)
EDM (%)
MNISTT
90.18(0.911)
51.34 (0.52x)
K8.4500 800)
79.04/0.79X)
89.82(0.90x)
73,8(0.73K)
Soft SoftLLabll Label
FushionMNIST
34.76 (0.38s)
15.28 (0.17%)
73,28/0
72.54(0.99)
73.27(0.80k)
74 84.17(0.81))
CIFAR-I0
85.39 (0.90K)
68.50 (0.73x)
22.05 23x)
22.01(0.25x)
21 23x)
20.96(0.22x)
CIFAR-100
53.04 (0.71k)
41.16 (0.55))
4.09
3.69 (0.05%)
4.45 (0.06x)
4.22 (0.065)
Flowers-17
69.85 (0.72x)
30.15 (0.31x)
11.76(0.12x)
11.40.
20.5910.21x)
18.75(0.19x)
Hard- Labe)
MNIST
61.20(00.22)
41.25 (0.41x)
81.530.80x1
72.6400733)
80.62(0.810)
73 1800000744)
FashionMNIST
28.60 (0.31x)
15.64 (0.17%)
70.9700.88)
66.5000.33)
70.2300775)
69.46(0.76x)
CIPAR-10
73.64 (0.78x)
54.98 (0.58x)
18.99 (0.20K)
20.08 (0.21k)
24.47 (0.200)
18.78 (0.20%)
CIFAR-100
(0.38x)
24.69 (0.33x)
4.23(0,000)
3.7500015)
5,24(0.05))
3.38008650)
FlOWCEs-17
36.25 (0.55%)
28.68 (0.30x)
11.76(0.123)
11.0%00111)
15.07(0.163)
16.34(0.17))
We use 6 rounds of data augmentation with the value of B set to 0.2 for the JBDA attack. The clone model is trained for 10 epochs in each round with the Adam optimizer with a learning rate of 0.001.
3. JBDA-TR is a variation of JBDA proposed by Juuti et al. (2019) that produces targeted misclas- sification from the clone model by using the gradient of the loss function to move towards a random target class (yr): x T Asign (V,L (C (xz; .e)))))))
EDM Defense: For the EDM defense, we use an ensemble of 5 models that are jointly trained on the accuracy and diversity objectives by using the loss function in Eqn. 3 The value of AD was selected with the constraint of having the degradation in benign accuracy of the target model be less than 0.5%. EDM also requires
auxiliary OOD) dataset for the diversity objective. We use KMNIST (Clanuwat et al. 2018) for MNIST and FashionMNISTT Tiny-Images (Torralba et al. 2008) for CIFARIÓ and CIFARTOM 1O0, and ImageNet for Flowers-17 17 as the auxiliary OOD datasets. For the DNN-based perceptual hashing we use M = 10 and m = 2. KMNIST is used as the secret classification task for MNIST and FashionMNIST. SVHN is used for the rest of the datasets. We use a simple 3-layer convolutional neural network trained on the secret classification task as the hash function fH to keep the computational overheads low.
5.2 RESULTS
Our results comparing the efficacy of various MS attacks targeting an undefended baseline and EDM targets for models trained on different classification tasks are shown in Table 3 In each case, we report the accuracy of the clone model along with the normalized clone accuracy computed with respect to the accuracy of the target model. Our results show that the clone accuracies produced by the KnockoffNets attack are much lower for EDM compared to the baseline model. With the JBDA and JBDA-TR attacks, the benefits of EDM are less pronounced as these attacks use perturbed versions of in-distribution examples to query the target, which may not produce large discontinuities in the predictions. However, JBDA and JBDA-TE -TR only work well on simpler datasets (MNIST, MNSST and do not produce good accuracies find that the discontinuous predictions produced by EDM can reduce the effectiveness of model stealing attacks for most attacks across multiple target models, offering up to a 39.7% reduction in clone accuracy compared to an undefended baseline model for existing attacks. Furthermore, our defense is also effective against adaptive attacks that are tailored for our defense (see Appendix A.I
higher dimensional datasets. Overall,
Comparison with prior work: We compare the clone accuracy obtained from our method with the adaptive misinformation (AM) defense (Kanyappa& & Qureshil 2020). AM aims to output incorrect predictions for the adversary's queries by adaptively perturbing the predictions of the model for OOD inputs. Unlike our work, AM trades off the benign accuracy for improved security. We compare the clone accuracy obtained from KnockoffNets attack between EDM and AM, with equivalent benign accuracy in Table 2 Our results show that EDM is able to outperform AM in most datasets with up to 32.16% lower clone accuracy compared to the AM de- fense. We see comparable performance with AM in case of Flowers- 17 dataset.
Table 2: Comparison of EDM with Adaptive Misinforma- tion defense
Clone Accuracy (%)
Dataset
AM
EDM
MNIST
85.5 (0.84x)
51.34 (0.52x)
FashonMNIST
27.5 (0.30k)
15.28 (0.17x)
CIFAR-10
(1.2())822)
(0.73x)
CIEAR-
(0.677)
(0.55k)
Flowers:
27.2 (0.29k
30.15 (0.31x)
7

---

Published as a conference paper at ICLR 2021
5.3 ANALYSIS
Impact of Diversity Loss on Coherence: To quantify the ef- ficacy of the diversity loss in producing dissimilar predictions, we compare the CDF of coherence values obtained for the pre- dictions of an EDM model trained with diversity loss and regular ensemble (with the same number of models) trained without diversity loss for the adversary's queries used in the KnockoffNets attack. From our results in Fig. 5 we find that the models in EDM produce much lower coherence val- ues compared to a regular ensemble. Additionally, these plots show that the models in EDM, which trained produce
Table 4: Comparing the clone ac- curacy produced by KnockoffNets by attacking a regular ensemble and a diverse ensemble.
Clone Accuracy (%)
Dutaset
Reg. Ens.
Div. Ens.
MNIST
88.01 (0.58x)
$1.34 (0.52x)
FabhioMNIST
30.11 (0.33k)
15:28 (0.17%)
CIFAR-10
84:25 (0.89)
68.50 (0.73k)
CIPAR-100
(0.67))
Flowen-17
54.78 (0.57))
(0.3x)
low coherence values on an auxiliary OOD dataset, can generalize to unseen OOD data used by the adversary to carry out model stealing.
MNIST
MishiorMNIST
CIFR-110
CIFAR-100
Flowers-17
Fraction arosa
1.0
Forehor of Data
1.0
Fraction of
1.0-
Froction DrD
1.0
Frachorn of Data
10
Diveese Ens.
Requler Ein.
0.0
0.0
0.0
0.5
0.0
0.0
0.0
0.0
0.5
Coherence
Coherence
Coherence
Coherence
Coherennce
Figure 5: Comparing the CDF of coherence values between a regular ensemble (trained without diversity loss) and Diverse Ensemble (trained with diversity loss). Diverse ensemble produces much lower coherence values on the adversary's s OOD data, indicating dissimilar predictions.
Impact of Diversity Loss on Clone Accuracy: Models trained without the diversity objective can also be used our proposal to produce discontinuous predictions. However, such models produce correlated predictions, making them easier to steal. We show the importance of using the diver- sity objective by comparing the clone accuracies obtained by attacking two ensembles- 1. Regular Ensemble (trained without diversity loss) 2. EDM (trained with diversity loss)- using the Knock- offNets attack. Our results in Table show that regular ensembles produce higher clone accuracies compared to the diverse ensemble, emphasising the importance of training with diversity loss.
6 DISCUSSION ON ADAPTIVE ATTACKS
An adversary who is aware of our defense can potentially modify the attack to improve the efficacy of MS. Fundamentally, an adversary can overcome our defense if the discontinuous predictions of EDM can be converted to a continuous function. We discuss two adaptive attack strategies and explain why such adaptive attacks are hard carry out.
1. Attacking a single model: Since the predictions of any individual model in EDM are continuous, the attacker can try to obtain the predictions of single model and use these predictions train the clone model. However, since the hash function is kept secret, it is not possible for the adversary to know which model was used service given request, making such attacks difficult execute.
2. Attacking the Ensemble: A second strategy is to somehow obtain the average predictions of all the models in the ensemble for given input. Since the average prediction is continuous, this can potentially be used to train a high-accuracy clone model. To obtain the average predictions, the adversary needs to make small perturbations to the input that cause the input to get hashed a different model. However, the use of perceptual hashing in our defense makes such perturbation- based averaging harder to perform. Perceptual hashing ensures that small perturbations to the input result in the input mapping to the same hash. For large perturbations that do map to different hash, the input no longer resembles the original input, making the averaging process meaningless. Furthermore, we note that even if the hash function was known to the adversary, such an attack would be costly, requiring an ((N) increase in the query budget to carry out. Additionally, querying the target model with a set of perturbed inputs opens up the attack to detection using methods such as PRADA (Juuti et al. 2019). which can limit the applicability of such adaptive attacks. We refer the reader Appendix A. for detailed analysis of this adaptive attack strategy.
8

---

Published as a conference paper at ICLR 2021
7 CONCLUSION
This paper proposes Ensemble of Diverse Models, a novel type of model ensemble that inherently robust to MS attacks due to the discontinuous nature of the model's predictions. The models in EDM are trained to produce dissimilar predictions on OOD data by using the diversity loss function. Our defense selects different model from the ensemble to service different inputs using an input- based hash function, producing predictions that are highly discontinuous in the input space for the adversary's 's OOD inputs. Attacks that use these discontinuous predictions train clone model end up with a bad quality clone model, reducing the efficacy of model stealing attacks. Through our empirical evaluations, we show that EDM can reduce the clone accuracy by up to 39.7%, with minimal degradation (< 0.5%) in the accuracy of the target model on benign inputs. Additionally, EDM has low computational overheads, making it a practical solution to defend against MS attacks.
8 ACKNOWLEDGEMENT
We thank Ryan Feng, Neal Mangaokar and Ousmane Dia for their valuable feedback. This work was partially supported by gift from Facebook and is partially based upon work supported by the National Science Foundation under grant numbers 1646392 and 2939445 and under DARPA award 885000. We also thank NVIDIA for the donation of the Titan V GPU that was used for this research.
REFERENCES
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep Learning for Classical Japanese Literature. arXiv e-prints, art. arXiv: 181.:18812.01718, Dec 2018.
Jacson Rodrigues Correia-Silva, Rodrigo F Berriel, Claudine Badue, Alberto F de Souza, and Thiago Oliveira-Santos. Copycat cnn: Stealing knowledge by persuading confession with random non- labeled data. In 2018 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2018.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC Confer- ence on Computer and Communications Security, pp. 1322-13233 1333, 2015.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv: 14126552, 2014.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 18Xi: 14.66066 2018
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In 2019 IEEE European Symposium on Security and Privacy (FuroS& P), pp- 512-527. IEBF, 2019.
Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adap- tive misinformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Put- tern Recognition, pp. 770-778, 2020.
Sanjay Kariyappa, Atul Prakash, and Moinuddin Qureshi. Maze: free model stealing attack gradient estimation. arXiv preprint arXiv:2005.03161, 2020.
using
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against machine learning model stealing attacks using deceptive perturbations. arXiv preprint 1rXiv:1806.000 2018.
Daniel Lowd and Christopher Meck. Adversarial learning. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pp. 641-647, 2005.
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition (CVPR'08 (06), volume 2, pp. 1447-1454. IEFF, 2006.

---

Published as a conference paper at ICLR 2021
Seong Joon Oh, Bernt Schicle, and Mario Fritz. Towards reverse-ngiineering black-box neural networks. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 121- 144. Springer, 2019.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4954 9954-4963, 2019a.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stealing attacks. In International Conference Learning Representations, 2019b.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, 2. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519, 2017.
A. Quattoni and A. Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 413 420, June 2009, doi: 101.1///CVPR..200.5006660
Nicholas Roberts, Vinay Uday Prabhu, and Matthew McAteer. Model weight theft with just noise inputs: The curious case of the petulant attacker. arXiv preprint arXiv: 191.0099,,. 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 (0(1)))198-1190, 2008
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart, Stealing machine learning models via prediction apis. In 25th {USENIX) Security Symposium (!USENIX) Security 16), pp. 601-618, 2016.
Joel A. Tropp. Just relax: convex programming methods for identifying sparse signals in noise.
IEEE Trans. Information Theory, 523))1030-1051, 2006. doi: 10.1109/11..2205584444 URL
attps://dols http///....r/1..111111100 1ttp://...11/TTIIT /ttp: 2005. 2000.8644200
Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning. In 2018
IEEE Symposium on Security and Privacy (SP), pp. 36-52. IEEE, 2018
Christoph Zauner. Implementation and benchmarking of perceptual image hash functions. 2010.
10

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 EVALUATING ADAPTIVE ATTACKS AGAINST EDM
As discussed in section 6 one possible strategy to attack EDM is to obtain the average predictions of all the models in the ensemble and train the clone model on this average prediction. To explain this adaptive attack, consider an input x for which we would like to compute the average prediction of all the models in the ensemble: y N Ei= fi(z), where {fi) are the set of all the models in EDM. EDM uses an input-based hash: (ndez(x)) = 1 + N (x)% to select the model findez that is used to service æ. If the hash function H was known, the adversary could modify the orig- inal input x to obtain a perturbed input x' = x + 5 that maps to a different hash value such that index(x")) A index(()) By querying EDM with multiple perturbed inputs {x') that produce dif- ferent values for index((1)), the adversary can obtain approximation of the average predictions y K E fEDM This approximation estimates the true average prediction well as long as the size of the perturbation is small enough to not change the predictions of the model significantly i.c f((x²) ~ fi(z). Once the average predictions are obtained, a labeled dataset {z, 5} can be constructed using these average predictions to train the clone model.
However, the hash function is unknown to the adversary, which gives rise to two challenges. First, evaluating the value of the (ndeaa(s) is not possible and second, finding small perturba- tions that would change the value of indeæ (x²)) is hard. One approach to solving this problem by averaging the predictions of the EDM over multiple inputs with large random perturbations K { such that 5 * ΣØ= fEDM((E)) Large perturbations increases the likelihood that inputs map to different indices. Averaging over a a large set of such perturbed inputs improves the estimation of the average prediction. We evaluate the efficacy of such an adaptive attack empirically. We perform an adaptive version of KnockoffNets attack by using the average predictions over K = 5 perturbed inputs to train the clone model. Note that such averaging attack requires 5-fold increase in the query budget carry out. We use combination of random affine transforms (rotation, translation, scaling, shear) and random changes to brightness, contrast, color and hue of image to obtain the perturhed inputs {=r from the original input x. Our results comparing the clone accuracy obtained from this adaptive attack com- pared to the non-adaptive attack on EDM and the single undefended model are shown in Table 5 Our results show that, while the clone accuracy improves with the adaptive attack compared to the non-adaptive attack on EDM, it still falls short of the clone accuracy obtained by attacking a single model. The improvement in clone accuracy can be attributed to the averaging of the decisions from multiple models. There are two reasons why the adaptive attack still falls short the accuracy with a single model: 1. The predictions of the model could be different for the perturbed input i.e f((a') & fz)). which gives rise to an approximation error. 2. The DNN-based perceptual hashing algorithm ensures that most of the perturbed inputs map to the original index, preventing proper averaging across the predictions of all models.
Table 5:
Evaluation of adaptive attack against EDM using multiple perturbed inputs
Dalase
Undef.
EDM
EDM (uduppiive)
MNIST
(0.914)
$1.34 (0.52s)
72.00 (0: 72x)
FashiosMNIST
34.76 (0.366))33)
15.28 (0.17k)
15,02 (0. 16x)
CIFAR-10
68.50 (0.73x)
63.89 (0.68K)
CIFAR-100
53.04 (0.71k)
41.16 (0.55a)
50.98 (0.685)
Floers-1
69.85 (0.72g
30.15 (0.31x)
54.78 (0.57%)
A.2 EFFICACY OF DNN-BASED PHASH
An ideal perceptual hash function would generate the same hash value for the original (z) and perturbed inputs (I'). To understand the efficacy of our proposed pHash algorithm, we plot the sorted histogram of index values: inder(x) = + (N,,,, generated from our hashing function for 10000 randomly perturbed images. We aggregate the histogram of sorted indices for 1000 random examples in the training set of the CIFAR-10 dataset. We compare the performance of our proposed hash function against Discrete Cosine Transform (DCT) based perceptual hash Zauner (2010) and an 1HA-1 cryptographic hash. From our results in Fig. 6 we can see that our DNN-basee pHah produces histogram that more skewed compared to the other hashing schemes. This implies that our hashing function has better transformation invariance property as it maps most of the perturbed inputs to the same index, making it suitable for perceptual hashing.
11

---

Published as a conference paper at ICLR 2021
DNN-based pHash
DCT-based pHash
Cryptographic Hash
1.0
Probability (lndex rank)
1.0
Probability (index rank)
1.0
Probabilty (index
0.8-
0.8
0.8
0.6
0.6
0.6
0.4-
0.4
0.4
0.2-
0.2
0.2
0.0
0.0
0.0
4
Index
Figure 6: Sorted histogram of index values outputs by the DNN-based pHash for perturbed inputs in CIFAR-10
Table 6: Evaluation of EDM against KnockoffNets with larger clone model. Using larger clone model yields similar clone accuracies as the original clone model.
Dataset
Undef.
EDM-Ovip.
EDM-Large
Ovig. Model
Large Medel
MNIST
90.18 (0.91%)
51.34 (0.52x)
53.10 (0.53x)
Com3
ResNee-20 20
PusticMMIST
(0.28x)
15.25
Coová
Rostee
CIRAR-10 10
$5.39 (0.90x)
68.50 (0.73x)
67.52 (0.72x)
Wres1644
Wres22:
CIFAR-1O 100
53.04 (0.71x)
41.16 (0.55k)
43.45 (0.58x)
Wres1644
Wres
Rowers-17
30.15
9:33 (0.11x)
ResNet-18 18
RostNer-32
A.3 COMPATIBILITY EDM WITH OTHER DEFENSES
Our paper proposes to use the discontinuous predictions produced by an ensemble of diverse models to defend against MS attacks. Prior works on MS defense have mainly focused on perturbing the output predictions prevent the adversary from reliably accessing the predictions of the model. Since EDM does not modify the output predictions of the model, our defense can be combined with existing perturbation based defenses (Orekondy et 2019b) Lee et al. 2018 Kariyappa & Qureshi 2020) to further improve the security of the model against MS attacks.
A.4 COMPUTATIONAL COST
Compared to an undefended baseline model, EDM requires training multiple models, increasing the training cost by factor of N. However, during inference, only a single model from the ensemble selected and evaluated, making the inference cost comparable with the baseline, The additional compute associated with evaluating the hash function is negligible compared to the cost of eval- uating the DNN used for the main classification task. For instance, the Conv3 model used in the hash function for CIFAR-10 only requires a ~ 2.5% increase in the number of flops compared to evaluating a single Wres16-4 network. EDM also requires a factor of N increase in memory costs to store the ensemble. However, in datacenter setting with multiple machines running the same inference task, the memory cost can be avoided by spreading the models of the ensemble across multiple machines, with each input being routed the selected model for inference. The negligible impact on benign accuracy and minimal computational overheads make EDM practical solution to defend against MS attacks.
A.5 KNOCKOFFNETS WITH LARGER CLONE MODEL
In the main paper, we evaluated our defense with attacks that used clone models with the same architecture as the target model. In this section, we report results for the KnockoffNets attack by using a larger model as the clone. We compare the clone accuracy obtained by using the original and the larger clone models in Table Our results show that clone accuracy obtained with larger model architecture is similar to the clone accuracy obtained with the original model architecture.
A.6 EVALUATIONS OF ATTACKS WITH LARGER QUERY BUDGET
We used an attack query budget 50K to evaluate our defense in the main paper. this section, report the results with increased query budget of 100K queries (2x). Note that, for the
12

---

Published as a conference paper at ICLR 2021
00.5
MMST (Comv3)
FasiionMMS (Coms))
CIFAR1 (Nios16-4)
CIFAR100 (Wte51544)
Defoncer Accumoo (1)
Fowers (ResNer 18)
Defonver Accumsy
Deffenee Acqumay
Acqutso
Delecgee Artirey
88.0
58.5
12
Idsx
25 30
50
76
20
20
Clone Actunny (M)
Clore Accurayy (%)
Clone Accuracy (54
Ciire Aczuuayy (S)
AlorwAssureyy (86)
Figure 7: Sorted histogram of index values outputs by the DNN-based pHash for perturbed inputs in CIFAR-10
KnockoffNets attack, in addition to the query budget, the attacker is bound by the availability of surrogate data used for the attack. Thus, for our evaluations, we stop querying the target model when we either run out of query budget or the surrogate data samples available to query the target model. For the JBDA and JBDA-TR attacks, to utilize the larger query budget, we increase the number of augmentation rounds from 6 to 10. Our results are reported in Table 7 We observe only a slight improvement in clone accuracy for KnockoftNets as the attack is limited by the availability of surrogate data (70K was the size of the largest surrogate dataset used). Even in the case of JBDA and JBDA-TR attacks, the clone accuracy only improves marginally as the improvement in clone accuracy plateaus after 6 augmentation rounds " For all the attacks, EDM continues to have significantly lower clone accuracy compared to the baseline, even with a larger query budget.
Table 7: Evaluations with a higher attack query budget of 100K queries. EDM continues to produce lower clone accuracy compared to the undefended baseline even with higher query budget.
KuockoffNets
JBDA
JBDA-TR
Dalaset
Undel. (%)
EDM (%)
Uniel. (5)
(D)
Undef. (%)
EDM (5)
MNIST
92.15 (0.93x)
$1.5 (0.52x)
(0.03/0.000)
72.46(0.73k)
91.38/0.92x)
76.09:0.76x)
FushionMNIST
35.66 (0.39s)
18.68 (0.21x)
82.82(0882)
74.66/1.823)
76.35(0.84x)
74.88(0.52x)
Solt-
CIFAR-10
87.84 (0.93x)
67.59 (0.72x)
22.9000.34x)
23.020.24x)
91 27x)
21.02/0.22x)
CIFAR-100
53.73 (0.72x)
44.60 (0.60h)
3.860085k)
4,3600.06x)
4.99(0.0%))
4.21(0.06x)
Rowers-17
69.85 (0.72x)
30.15 (0.31x)
13.76(0.155 5k)
12.8700144k
20.66(0.21)
17.2%/.19x)
A.7 EVALUATIONS WITH ADAPTIVE MISINFORMATION DEFENSE
We compare the trade-off between defender accuracy (for benign examples) and clone accuracy of the Adaptive-Misinformaiie
(AM) defense (Kariyappa & Qureshi 2020) with our proposal. AM uses an out of distribution (OOD) detector to identify the adversary 's OOD data and services such queries with incorrect predictions. By changing the value of the threshold used in the OOD detector, this defense offers a trade-off between benign accuracy and clone accuracy. In the case of EDM, one could expect similar trade-off to exist by changing the value of the hyper-parameter Ap in [qn However, empirically we found that there is only a negligible loss in defender accuracy even with large value of Åp. This is because, the Diversity Loss term associated with An has lower bound of O. Empirically, we found that by increasing the value of Åp. we can minimize the Diversity Loss term 0 with negligible loss in defender accuracy. Thus, instead of trade-off curve, we denote EDM with a single trade-off point in our plot to compare against AM. Our results in Fig. 7 show that for the same defender accuracy, EDM yields lower clone accuracy compared to AM.
"Our results are consistent with the observations made by
Juuti et al. 2019
13


Reviews:
[{"review_id": "wvJwBdRdmA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple but effective way to avoid model stealing. To the best of my knowledge, the idea is novel. The writing of this paper is quite clear and experimental results also show the effectiveness of proposed method. By introducing diversity loss, the model ensemble tends to produces discontinuous outputs which is hard to be distilled. I still have some questions:\n\n1. In your experiments, there is an assumption that the thief knows what exactly the model archs. Given this prior knowledge, what will happen if the thief use a more complex/bigger model to distill ? \n\n2. It seems  inconsistent results are observed on Flowers17 dataset, any explanation?\n\n3. There is another assumption that the attackers are constrained by attach budget (50000 trials), what will happen if more trials or unlimited trials are allowed? I think this question is important because this is often true scenario if the attacked models are of high business value.\n\n4. Have you tried your method on larger tasks such as ImageNet or other fields, for example machine translation and speech recognition?", "extra_scores": {}, "extra_text": {}}, {"review_id": "U4DPhdE0Gor", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n- The paper proposes a defense against recent flavours of model stealing attacks by exploiting the insight that the recent effective attack query out of distribution examples to the victim model.\n- The approach introduces discontinuities in the input-prediction space by (i) training an ensemble of models such that for a given OOD input, the predictions are randomized (ii) at inference time hashing an input to a particular input in the diverse ensemble.\n- Evaluation on simple datasets (MNIST, Fashion, CIFAR) show that the approach is reasonably effective at defending and moreover without a significant degradation of the utility.\n\n## Strengths\n\n**1. Well-motivated problem**\n- The defenses for model stealing are not as well-investigated as attacks. I appreciate the authors take a step towards addressing defenses; the initial results look promising.\n\n**2. Insight**\n- I like the insight used by the authors in the defense i.e., to generate discontinuities in the input-prediction space. This is well illustrated in Fig. 1.\n\n**3. Writing**\n- The paper is written well and is easy to follow.\n\n## Concerns\n\n### Major Concerns\n\n**1. $D_{in}$ vs. $D_{out}$**\n- It is a nice insight that adversarial users indeed rely on querying data from a different distribution. However, my first concern is that is also naturally true for benign users. After all, no one except the victim/defender has access to the training data distribution and all queried data (whether by adversary or benign users) is out of distribution.\n- As a result, I wonder how applicable the defense is -- would the defense intentionally mispredict in non-IID query setups? For instance, when queried with a particular dog breed unseen during training? Or querying slightly translated digit images to standard MNIST classifier?\n- More generally, the proposed approach requires drawing a strict line between in and out distribution which I find is not clearly analyzed (follow-up remarks in point 4).\n\n**2. Diversity Objective**\n- If I understand the training objective correctly (Fig. 2b, Eq. 3), each model $f_i$ in the ensemble is encouraged to generate a random prediction when queried with out-of-distribution (OOD) inputs. Moreover, in expectation, results in a uniform distribution over predictions on OOD data. For in-distribution data, the predictions ideally remain unchanged.\n- As a result, my concern is that this appears to be simple OOD detection followed by misprediction/calibration in disguise. Consequently, I wonder if one can simply leverage existing advances in OOD detection to achieve the objective.\n- Moreover the OOD implicitly performed here requires access to a corresponding dataset $D_{out}$ to model unknown OOD samples. This contrasts some existing OOD detection works (e.g., ODIN, Liang et al., ICLR '18) which shows success without explicitly modelling OOD data.\n\n**3. Hashing**\n- I am also concerned that the hashing strategy (Sec. 4.2) employed by the authors appears to be a weak link in the defense.\n- Specifically, it appears that a robust defense requires the hashing function to consistently select the same model in the ensemble in spite of small changes in the input. However, going by Fig. 6 it does not necessarily seem to be the case.\n- As a result, I wonder if an attacker can exploit the hashing function to recover whether the output is clean/poisoned -- such as by aggregating predictions over a set of transformed inputs.\n\n**4. Evaluation**\n- While the results look promising, I have two concerns here related to the evaluation.\n- First, I would have preferred if the results in Table 3 were presented as curves (with defense vs. attacker accuracy). This would have provided some insights on how the defense performs w.r.t the important $\\lambda$ hyperparameter in their objective. Additionally, it would also make the comparison with AM baseline fair, especially given that AM was evaluated on a curve and it is unclear which specific instance was used for comparison.\n- Second, which also connecting to point 1, is that the paper's experimental setting assumes a significant discrepancy between the distributions e.g., Victim's $D_{in}$ = MNIST, $D_{out}$ = KMNIST, Attacker's data = FashionMNIST. Here, the data and semantic classes are completely disjoint. I wonder how the defense performs with other choices of attacker's data e.g., EMNIST which is a bit more similar to MNIST.\n\n\n### Minor Concerns\n\n**5. Experimental settings**\n- I find missing some important experimental parameters that is not mentioned in the paper: how many images were queried by the attacker for results in Table 1? What was the value of $\\lambda$ used?\n\n**6. Subverting schemes**\n- I was also disappointed that the authors do not demonstrate that the defense is robust to some simple schemes used by an attacker to bypass the proposed defense. For instance, connecting to point (3), by aggregating predictions over a set of transformed inputs.\n\n\n### Nitpicks\n\n**7. Some nitpicks**\n- $\\lambda$ is overloaded: in Eq. 3 and also for the JBDA attacks\n- \"AM trades off benign accuracy for improved security\" -- would the proposed approach also trade this off by increasing the value of $\\lambda$?\n\n### Post-rebuttal updates\n- Thanks for the detailed response and additional evaluation.\n- Q1. Fair point -- it appears that defenses do assume attackers query from a reasonably different distribution. This defense does make this assumption explicit by including it in the training objective ($D_{out}$ in Eq. 3). But then again, it seems typical for OOD-based defenses (e.g., AM).\n- Q3. This concern also generally connects to Q6. It would be nonetheless interesting to analyze scenarios where the attacker attempts to use some auxiliary knowledge to break the defense. This is also shared by some other reviewers.\n- Q4. Thanks for presenting the curves. Assuming strictly non-overlapping OOD data does seem like a strong assumption though.\n- Nitpick editorial comment: Please serif-based fonts for text in equations e.g. $argmax(\\cdot) \\rightarrow \\text{argmax}(\\cdot), index \\rightarrow \\text{index}$\n- I am slightly increasing my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "40VEF1c43jT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles a timely problem of protecting deep neural networks from mode stealing attacks. This paper proposed an Ensemble of Diverse Model to provide diversity prediction for the adversary’s OOD query. The main contribution of this paper is the introduction of the diversity loss function on OOD data and the discontinuous prediction provided by ensembled models. The results show that EDM defense reduces the accuracy of stolen models.\n\nPros:\n1.\tThe idea of using ensemble of diverse models to create discontinuous predictions on OOD datasets is interesting.\n2.\tThe experimental evaluation is comprehensive. Five datasets and three attacks are investigated in the evaluation. Adaptive attacks using average predictions are considered in the evaluation as well.\n3.\tThis paper is well-written. Figures make the paper easy to understand.\n\nCons:\n1.\tMy major concern is the auxiliary OOD dataset used by EDM. Does the defender have access to the adversary’s OOD dataset? In the experiments, although the EDM used auxiliary OOD datasets, these auxiliary OOD datasets are more complex than the adversary’s OOD datasets. If the adversaries have access to the auxiliary OOD datasets or use other complex OOD datasets, will the defense still work?\n2.\tAs mentioned in the paper, the hashing function requires transformation invariance, so that the adversaries cannot get the average prediction. However, can the DNN-based perceptual hashing function be generalized and applied to other types of transformation? For example, to break the transformation invariance, the adversary can use different algorithms to craft noises into the input data.\n3.\tFrom Table 3, it seems that the performance of EDM on JBDA and JDBA-TR is not very well. My understanding is that since JBDA and JDBA-TR do not reply on a fixed OOD dataset, EDM trained on auxiliary OOD dataset may be well generalized to these attacks’ OOD data. \n4.\tThe query budget of the adversary is set as 50,000. A larger number of queries are used in the previous attacks. For example, in Juuti et al. 2019, 102,400 queries are used to steal models trained on the MNIST dataset. Does the small query budget used in the paper limit the performance of attacks?\n5.\tMinor comments:\na.\tWhat is CS in equation (1) and (2)? \nb.\t“A perceptual hashing algorithm is used prevent adaptive attacks”\nc.\t“Our empirical evaluations shows that”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "39Y2ORadZUr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a method to protect deep neural networks against model stealing. The propose defense trains an ensemble of classifiers using two losses one targeting accuracy and the other diversity of the ensemble. In particular, the trained classifiers are consistent on in-distribution data, but contradict each other on out-of-distribution data. The proposed defense does not affect the test accuracy of the victim model while it strongly limits the test accuracy of the clone model.\n\nDespite the concerns below, I vote for accepting the paper. The addressed issue is relevant, the proposed method is interesting. The performance in the presented experiments is convincing and the writing is overall reasonably clear.\n\nConcerns:\n1. the method hinges on keeping the hash private (\"security through obscurity\")\n2. it also hinges on the attacker not being able to query the API with in-distribution images\n3. the authors argue that the inference runtime is not impacted by the method, however, the GPU memory requirements scale linearly with the size of the ensemble, which might be costly\n\nQuestions:\n- it seems one could try to launch an adversarial attack on the hash, why should it fail?\n- how far from the in-distribution images would one have to move to start hitting the effect of the diversity loss? As a user of an API, I have some knowledge about which images are in-distribution, why is using this information impractical?\n- at the end of Sec 4.1, results on generalization to unseen out-of-distribution datasets are referred to, supposedly to be found in Sec 5.3, but I cannot find them there, where are they?\n- the hash function should map to as many members of the ensemble as uniform as possible on both the in-distribution and out-of-distribution data, how is this achieved?\n\nTypos:\n- Table 3, the number for hard-label - Flowers-17 - JBDA-TR - EDM should most probably be 16.54 not 6.54, please check\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an ensemble of diverse models as a mechanism to protect models from theft. 
The idea is quite novel. There are some concerns regarding the robustness of the hashing function (that I share), however not every paper has to be perfect, especially when it introduces a novel setup. 

AC
Decision: Accept (Poster)