Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks
Authors: Tribhuvanesh Orekondy
Bernt Schiele
Mario Fritz
Venue:  (2020)
Keywords: model functionality stealing
adversarial machine learning

Abstract: High-performance Deep Neural Networks (DNNs) are increasingly deployed in many real-world applications e.g., cloud prediction APIs. Recent advances in model functionality stealing attacks via black-box access (i.e., inputs in, predictions out) threaten the business model of such applications, which require a lot of time, money, and effort to develop. Existing defenses take a passive role against stealing attacks, such as by truncating predicted information. We find such passive defenses ineffective against DNN stealing attacks. In this paper, we propose the first defense which actively perturbs predictions targeted at poisoning the training objective of the attacker. We find our defense effective across a wide range of challenging datasets and DNN model stealing attacks, and additionally outperforms existing defenses. Our defense is the first that can withstand highly accurate model stealing attacks for tens of thousands of queries, amplifying the attacker's error rate up to a factor of 85$\times$ with minimal impact on the utility for benign users.

Published as a conference paper at ICLR 2020
PREDICTION POISONING:
TOWARDS DEFENSES
AGAINST DNN MODEL STEALING ATTACKS
Tribhuvanesh Orekondy", Bernt Schiele', Mario Fritz?
1 Max Planck Institute for Informatics
2 CISPA Helmholtz Center for Information Security
Saarland Informatics Campus, Germany (orekondy, schiele Schielee@mpi-iiee Cmpi-inf. wwg. de, fritzecispa. fritzecispa.saarland saarland
ABSTRACT
High-performance Deep Neural Networks (DNNs) are increasingly deployed many real-world applications e.g., cloud prediction APIs. Recent advances in model functionality stealing attacks via black-box access (i.e., inputs in, predic- tions out) threaten the business model of such applications, which require lot of time, money, and effort to develop. Existing defenses take a passive role against stealing attacks, such as by truncating predicted information. We find such passive defenses ineffective against DNN stealing attacks. In this paper, we propose the first defense which actively perturbs predictions targeted at poisoning the train- ing objective of the attacker. We find our defense effective across wide range challenging datasets and DNN model stealing attacks, and additionally outper- forms existing defenses. Our defense is the first that can withstand highly accurate model stealing attacks for tens of thousands of queries, amplifying the attacker's s error rate up factor of 85 with minimal impact on the utility for benign users.
INTRODUCTION
Effectiveness of state-of-the-art DNN models at variety of predictive tasks has encouraged their usage variety of real-world applications e.g., home assistants, autonomous vehicles, commercial cloud APIs. Models in such applications are valuable intellectual property of their creators, as developing them for commercial use a product of intense labour and monetary effort. Hence, vital to preemptively identify and control threats from an adversarial lens focused at such models. In this work we address model stealing, which involves an adversary attempting to counterfeit the functionality of target victim ML model by exploiting black-box access (query inputs in, posterior predictions out).
Stealing attacks dates back to Lowd & Meek (2005), who addressed reverse-engineering linear spam classification models. Recent literature predominantly focus on DNNs (specifically CNN image classifiers), and are shown to be highly effective (Tramer et al. 2016) complex models (Orekondy et al. 2019). even without knowledge of the victim's 's architecture (Papernot et al.) (007bb) nor the training data distribution. The attacks have also been shown to be highly effective replicating pay-per-query image prediction APIs, for as little as $30 (Orekondy et al. 2019).
Defending against stealing attacks however has received little attention and is lacking. Existing defense strategies aim to either detect stealing query patterns (Juuti et al. (2019). or degrade qual- ity of predicted posterior via perturbation. Since detection makes strong assumptions on the tacker's 's query distribution (c.g., small L2 distances between successive queries), our focus is on the more popular perturbation-baser defenses. A common theme among such defenses is accuracy- preserving posterior perturbation: the posterior distribution is manipulated while retaining the top-1 label. For instance, rounding decimals (Tramèr et al., 2016), revealing only high-confidence predic- tions (Orekondy al., 2019). and introducing ambiguity at the tail end of the posterior distribution (Lee al. 2018). Such strategies benefit from preserving the accuracy metric of the defender. How- ever, in line with previous works (Tramer al. 2016 Orekondy al. 2019 Lee al. 2018), find models can be effectively stolen using just the top- predicted label returned by the black-box. Specifically, in many cases observe < 1% difference between attacks that use the full range of

---

Published as a conference paper at ICLR 2020
posteriors (blue line in Fig. D train stolen models and the top-1 label (orange line) alone. In this paper, we work towards effective defenses (red line in Fig. I against DNN stealing attacks with
minimal impact to defender's accuracy.
MNIST
The main insight to our approach is that unlike a benign user, a model stealing attacker additionally uses the predictions to train a replica model. By introducing controlled perturbations to predic- tions, our approach targets poisoning the training objective (see Fig.
100
sccurocy stoles.
50-
undefended
2). Our approach allows for a utility-preserving defense, as well as trading-oft off a marginal utility cost to significantly degrade attacker's performance. As practical benefit, the defense involves single hyperparameter (perturbation utility budget) and can be used with minimal overhead to any classification model without retraining or modifications.
top-1 label
our defense
10k
30k
50k
queries
Figure 1: We find existing de- fenses (orange line) ineffective against recent attacks. Our de- fense (red line) in contrast signif- icantly mitigates the attacks.
We rigorously evaluate our approach by defending six victim mod- els, against four recent and effective DNN stealing attack strategies (Papernot et al. 2017b; Juuti et al., 2019; Orekondy et al., 2019). Our defense consistently mitigates all stealing attacks and further shows improvements over multiple baselines. In particular, we find our defenses degrades the attacker's 's query sample efficiency by 1-2 orders of magnitude. Our approach significantly reduces the at- tacker's performance (e.g., 30-53% reduction on MNIST and 13- 28% on CUB200) a marginal cost (1-2%) defender's test accu- racy. Furthermore, our approach can achieve the same level of mit- igation as baseline defenses, but by introducing significantly lesser perturbation.
Attacker's 's Loss Landscape
0
V Vub(-12)
-VwL(:
Our Perturbation Objective:
argmax W
s.t. dist(y, 9) S
Contributions.
We propose the first utility-constrained de- fense against DNN model stealing attacks; (ii) We present the first active defense which poisons the attacker's training objective by in- troducing bounded perturbations; and (iii) Through extensive exper- iments, we find our approach consistently mitigate various attacks and additionally outperform baselines.
Figure 2: We perturb posterior
predictions ÿ = + 6, with an objective of poisoning the adver- sary's 's gradient signal.
RELATED LITERATURE
Model stealing attacks (also referred to as 'extraction' or 'reverse-engineeriigg in literature aim to infer hyperparameters (Oh et al. 2018 Wang & Gong 2018)), recover exact parameters (Lowd & Meek 2005 Tramer et al. 2016 Milli et 2018). or extract the functionality (Correia-Silvv 2018 Orekondy et al. 2019) of a target black-box ML model. In some cases, the extracted model information optionally ly used to perform evasion attacks (Lowd & Meek, 20055 Nelson al. 2010 Papernot et al., 2017 The focus of our work is model functionality stealing, where the attacker's yardstick test-set accuracy of the stolen model. Initial works on stealing simple linear models
al.
(Lowd & Meek] (2005) have been recently succeeded by attacks shown to be effective on complex
CNNs (Papernot et al. 20176, Correia-Silva al., 22018] Orekondy et al. 2019) (see Appendix B
for an exhaustive list). In this work, we works towards defenses targeting the latter line of DNN
model stealing attacks.
Since ML models are often deployed in untrusted environments, a long line of work exists on guar-
anteeing certain (often orthogonal) properties to safeguard against malicious users. The properties include security (e.g., robustness towards adversarial evasion attacks (Biggio et al. [0033] Goodfel-
low et al. 2014 Madry et al., 2018) and integrity (e.g., running in untrusted environments (Tramer
& Boneh ) To prevent leakage of private attributes (e.g., identities) specific to training data in
the resulting ML model, differential privacy (DP) methods (Dwork et al. (0144) introduce random-
ization during training (Abadi et al. 2016 Papernot et al. (017a). In contrast, our defense objective
is to provide confidentiality and protect the functionality (intellectual property) of the ML model
against illicit duplication.
2

---

Published as a conference paper at ICLR 2020
Model stealing defenses are limited. Existing works (which is primarily in multiclass classification settings) aim to either detect stealing attacks (Juuti et al. 2019 Kesarwani et al. 2018 Nelson et al.) 2009 Zheng et al. 2019) perturb the posterior prediction. We focus the latter since detec- tion involves making strong assumptions on adversarial query patterns. Perturbation- predominantly randomiddd and eecuracy-preservigg (i.e., top-1 label is unchanged). Ap- proaches include revealing probabilities only of confident classes (Orekondy et al., 2211), rounding probabilities (Tramèr et al. 2016), or introducing ambiguity in posteriors (Lee et al. 2018). None of the existing defenses claim to mitigate model stealing, but rather they only marginally delay the attack by increasing the number of queries. Our work focuses on presenting an effective defense, significantly decreasing the attacker's query sample efficiency within a principled utility-constrained framework.
defenses
3 PRELIMINARIES
Model Functionality Stealing. Model stealing attacks are cast as an interaction between two parties: a vietim/defenderr V ('teacher' model) and an attacker A ('student' model). The only means of communication between the parties are via black-box queries: attacker queries inputs E X' and defender returns posterior probability distribution
E A P(ylæ) (xx where A y ž 0, 1"y y 1} is the probability simplex over K classes (we use K instead of K for notational convenience). The attack occurs in two (sometimes overlapping) phases: (i) querying: the attacker uses the black-box as an oracle labeler on a set of inputs to construct a 'transfer set' of input-prediction pairs Diransfer = {(x,, 34) }+)]= i= and (ii) training: the attacker trains model FA to minimize the empirical risk on Diransfer The end-goal of the attacker is to maximize accuracy on held-out test-set (considered the same as that of the victim for evaluation purposes).
Knowledge-limitee Attacker. In model stealing, attackers justifiably lack complete knowledge of the victim model Fv. Of specific interest are the model architecture and the input data distribution to train the victim model Py (X) that are not known to the attacker. Since prior work (Hinton et al.) 2015 Papernot et al.] |016 Orekondy et al., (019) indicates functionality largely transters across architecture choices, we now focus the query data used by the attacker. Existing attacks can be broadly categorized based on inputs (æ ~ (X)) used to query the black-box: (a) independent
distribution: (Tramer et al. 2016; Correia-Silve et al. 2018 Orekondy et al.) 2019) samples inputs from some distribution (e.g., ImageNet for images, uniform noise) independent to input data used to
train the victim model; and (b) synthetic ser: (Papernot al., 2017b) Juuti al. (209)) augment limited set of seed data by adaptively querying perturbations (e.g., using FGSM) of existing inputs. We address both attack categories in our paper.
Defense Objectives. We perturb predictions a controlled setting: ý = (x = y + s.t. ÿ. y E A The defender has two (seemingly conflicting) objectives: (i) utility: such that perturbed predictions remain useful to a benign user. We consider two utility measures: (a) Acc(FV) Diest): accuracy of defended model on test examples; and (b) dist (y. ÿ)) = y - c to measure pertur- bation. (ii) non-replicability: to reduce the test accuracy of an attacker (denoted as Acc(FA, ) who exploits the predictions to train a replica FA on Ditranseer For consistency, we evaluate both the defender's and attacker's stolen model accuracies the same set of test examples Diest
Defender's Assumptions. We closely mimic an assumption- free scenario similar to existing perturbation-based defenses. The scenario entails the knowledge-limited defender: (a) unaware whether a query is malicious or benign; (b) lacking prior knowledge of the strategy used by an at- tacker; and (c) perturbing each prediction independently (hence circumventing Sybil attacks). For added rigor, we also study attacker's countermeasures to our defense in Section 5
APPROACH: MAXIMIZING ANGULAR DEVIATION BETWEEN GRADIENTS
Targeting First-order Approximations. We identify that the attacker eventually optimizes parameters of stolen model F(::w) w) (we drop the subscript "A for readability) to mini- mize the loss on training examples ((x, g. Common to a majority of optimization algorithms is estimating the first-order approximation of the empirical loss, by computing the gradient of the loss
Motivation:
3

---

Published as a conference paper at ICLR 2020
w.r.t the model parameters E R:: u V y)
Maximizing Angular Deviation (MAD). The core idea of our approach is to perturb the posterior
probabilities which results in adversarial gradient signal that maximally deviates (see Fig. 2 from the original gradient (Eq. More formally, we add targeted noise to the posteriors which
results in gradient direction:
a - v (F(x;; w). 5)
(2)
to maximize the angular deviation between the original and the poisoned gradient signals:
max 2(1 cos L(a, u)) max | al1
(ā= a / 2, à u/ u//l
(3)
Given that the attacker model is trained to match the posterior predictions, such as by minimizing the cross-entropy loss L(y 9) -- -E Mr log yk we rewrite Equation ② as:
a - ---vwL(P(ee w); 9) V. L Un log F Fax; w) = Σ Ih V w log F ((;;w); = G²
where G G represents the Jacobian over log-likelihood predictions F(æ; w) over K classes RD. By similarly rewriting Equation (1), substituting them in Equation
w.r.t. parameters
3 and including the constraints, we arrive at our poisoning objective (Eq. 7 of our approach
which we refer to as MAD. We can optionally enforce preserving accuracy poisoned prediction via constraint 8 which will be discussed shortly.
GT y Tll
G
max 9
(= H()))
(4)
GT
yll
where
G= Vw log ((;
(G e R
(5)
s.t ÿ E A
(Simplex constraint)
(6)
dist( y, 9) S
(Utility constraint)
(7)
arg max Uk arg max yk
(For variant MAD-argmax)
(8)
&
The above presents challenge of black-box optimization problem for the defense since the defender justifiably lacks access to the attacker model (Fq. [5 Apart from addressing this challenge in the next few paragraphs, we also discuss (a) solving non-standard and non-convex constrained maximization objective; and (b) preserving accuracy of predictions via constraint
Estimating G. Since we lack access to adversary's model F, estimate the jacobian G V log Fsur (a; (Eq. 5) per input query x using a surrogate model Faur- We empirically de- termined (details in Appendix E.I choice of architecture of Fur robust to choices of adversary's architecture F. However, the initialization of Faur plays a crucial role, with best results on a fixed randomly initialized model. We conjecture this occurs due to surrogate models with high loss provide better gradient signals to guide the defender.
Heuristic Solver. Gradient-based strategies to optimize objective (Eq. 4 often leads to poor local maxima. This is in part due to the objective increasing in all directions around point (assuming G full-rank), making optimization sensitive to initialization. Consequently, we resort to heuristic to solve for ÿ. Our approach is motivated by [Hofiman (1981). who show that the maximum of convex function over a compact convex set occurs the extreme points of the set. Hence, our two-step solver: (i) searches for maximizer y" for {4) by iterating over the K extremes yı (where y==1) of the probability simplex A and (ii) then computes a perturbed posterior ý as a linear interpolation of the original posteriors y and the maximizer y": ÿ (1 a)y + az"" where is selected such that the utility constraint (Eq. is satisfied. We further elaborate on the solver and present pseudocode in Appendix C
Variant: MAD-argmax. Within our defense formulation, we encode an additional constraint (Eq. 8 to preserve the accuracy of perturbed predictions. MAD-argmax variant helps perform securacy-preserving perturbations similar to prior work. But in contrast, the perturbations are con- strained (Eq. [ and specifically introduced to maximize the MAD objective. We enforce the sccuracy-preserveny constraint in our solver by iterating over extremes of intersection of sets (q. and 8 A {y ž 0, 1" y = 1, yk yj k + j} C K
4

---

Published as a conference paper at ICLR 2020
5
EXPERIMENTAL RESULTS
5.1 EXPERIMENTAL SETUP
Victim Models and Datasets. We set up six victim models (see column "Fy' in Table D. each model trained on popular image classification dataset. All models are trained using SGD (LR 0.1) with momentum (0.5) for 30 (LeNet) or 100 epochs (VGG16), with a LR decay of 0. performed every 50 epochs. We train and evaluate each victim model on their respective train and test sets.
Attack Strategies. We hope to broadly address all DNN model stealing strategies during our defense evaluation. To achieve this, we consider attacks that vary in query data distributions (independent and syn- thetic; see Section 3 and strategies (ran- dom and adaptive). Specifically, in our experiments we use the following attack models: (i) Jacobian-based Data Aug-
Acc
FA)
jbda
]boo12
jbtop3
k.off
MINIST (LeNet)
99.4
$9.2
89.1
87.3
99.1
RashionMNIST (LaNet)
92.0
18.7
45.8
65.7
69.2
CIFAR to (VGG116
92.0
28.6
20.7
73.8
78.7
(VGGL6)
2.9
CUB200 (VGG16)
80.4
68
3.9
21.5
65.1
Caltec2556 (VGG16)
50.0
12.5
16.0
29.5
74.6
Table 1: Victim models and Accuracies, All accuracies w.r.t undefended victim model.
mentation 'JBDA" (Papernot et al. 2017b);
(ii.iii) JB-self' and 'JB-top3' (Juuti al.
2019); and (iv) Knockoff Nets 'knockoff' (Orekondy
et al. 2019); We follow the default configurations of the attacks where possible. A recap and imple- mentation details of the attack models are available in Appendix D
In all attack strategies, the adversary trains model FA to minimize the cross-entropy loss on a transfer set (Diranster = {(x, ".)) )) obtained by using the victim model Fy to pseudo-label in- puts (sampled or adaptively synthesized). By default, we use B=50K queries, which achieves reasonable performance for all attacks and additionally makes defense evaluation tractable. The size of the resulting transfer set (B=50K examples) is comparable (e.g.. 1x for CIFAR 10/100, 2.1x x for Caltech256) to size of Victim's 's training set. In line with prior work (Papernot et al. 2016 Orekondy et al. 2019), too find (Section 5.2.3) attack and defense performances are unaffected by choice of architectures, and hence use the victim architecture for the stolen model FA Due to the complex parameterization of VGG-16 (100M+), we initialize the weights from pretrained TinylmageNet or ImageNet model (except for the last FC layer, which is trained from scratch). All stolen mod- els are trained using SGD (LR=0.1) with momentum (0.5) for 30 epochs (LeNet) and 100 epochs (VGGI6). We find choices of attacker's architecture and optimization does not undermine the de- fense (discussed in Section 5.2.3)
Effectiveness of Attacks. We evaluate accuracy of resulting stolen models from the attack strate- gies as-is on the victim's test set, thereby allowing for fair head-to-head comparison with the victim model (additional details in Appendix [A] and D). The stolen model test accuracies, along with undefended victim model Fv accuracies are reported in Table We observe for all six victim models, using just 50K black-bos queries, attacks are able to significantly extract victim's 's function- ality e.g., 887 on MNIST. We find the knockoff attack to be the strongest, exhibiting reasonable performance even on complex victim models e.g., 74.6% (0.93x x Fv )) on Caltech256.
How Good are Existing Defenses? Most existing defenses in literature (Tramrr al., 2016 Orekondy et al. 2019 Lee et 2018) perform some form of information truncation on the poste- rior probabilities e.g., rounding, returning top-k labels; all strategies preserve the rank of the most confident label. We now evaluate model stealing attacks on the extreme end of information trun- cation, wherein the defender returns just the top-1 'argmax' label. This strategy illustrates a rough lower bound the strength of the attacker when using existing defenses. Specific to knockoff, we observe the attacker is minimally impacted on simpler datasets (e.g., 0.2% accuracy drop on CIFAR10; see Fig. A5 in Appendix). While this has larger impact more complex datasets involving numerous classes (e.g., a maximum of 23.4% drop observed on CUB200), the strategy also introduces significant perturbation + the posteriors. The results suggest existing defenses, which largely the top-1 label, are largely ineffective at mitigating model stealing attacks.
Defenses: Evaluation. We evaluate all defenses on non-repliaability vs. utility curve at various operating points e of the defense. We furthermore evaluate the defenses for a large query budget (50K). We use replcciiiyyyylity the accuracy of the stolen model held-out test data Diest
5

---

Published as a conference paper at ICLR 2020
FachiosMNSST
CIFAR1O
CIFAR100
CURSANI
Acc (oc(Deeenee))
Acx Altaaier)))
Acci Actacle)))
Acd Attaccio))
Acc Actakke)))
Acd Acttacca)))))
ActlAttacker))
Joda
jbseff
jotop3
Figure Attackers Our Defense. Curves obtained by varying degree of perturbation (Eq. in defense. t denotes higher numbers better and L. lower numbers are better. Non-replicability objective presented on the w-axis and utility on the y-axis.
We use two utility metrics: (a) accuracy: test-accuracy of the defended model producing perturbed predictions on Diest and (b) perturbation magnitude e: measured as L1 distance 1ly 8/11.
Defense: Baselines. We compare our approaches against three methods: (i) reverse-sigmole [ee et al. 2018): which softens the posterior distribution and introduces ambiguity among non- argmax probabilities. For this method, we evaluate non-replicability and utility metrics for the defense operating at various choices of their hyperparameter 3 E [o, 1], while keeping their dataset- specific hyperparameter fixed (MNIST: 0.2, FashionMNIST: 0.4, CIFAR10: 0.1, rest: 0.2). (ii) random noise: For controlled random-noise, we add uniform random noise 8₂ on the logit pre- 82, where
diction scores (#
( )), enforce utility by projecting 6= an ez-ball Duchi et al. 2008 and renormalize probabilities ÿ = T+r- (iii) dp-sgd: while our method and previous two baselines perturbs predictions, we also compare against introducing randomization to victim model parameters by training with the DP-SGD algorithm (Abadi al.) 2016). DP is a pop- ular technique to protect the model against training data inference attacks. This baseline allows us verify whether the same protection extends model functionality.
5.2 RESULTS
In the follow sections, we demonstrate the effectiveness of our defense rigorously evaluated across a wide range of complex datasets, attack models, defense baselines, query, and utility budgets. For readability, we first evaluate the defense against attack models, proceed comparing the defense against strong baselines and then provide analysis of the defense.
5.2. MAD DEFENSE VS. ATTACKS
Figure presents evaluation of our defenses MAD (Eq. and MAD-argmax, (Eq. against the four attack models. To successfully mitigate attacks as a defender, we want the defense curves (colored solid lines with operating points denoted by thin crosses) to away from undefended accuracies (denoted by circular discs, where c=0.0) to ideal defense performances (cyan cross, where Acc(Def.) unchanged and Acc(Att.) is chance-level).
We observe from Figure 3 that by employing an identical defense across all datasets and attacks, be greatly reduced. Across all models, find MAD provides
the effectiveness of the attacker reasonable operating points (above the diagonal), where defender achieves significantly higher test accuracies compared the attacker. For instance, on MNIST, for <1% drop defender's accuracy, our defense simultaneously reduces accuracy of the jbtop3 attacker by 52% (87.3% 887.3%-35.78% 7%) and knockoff by 29% (99.1 69.......9999 8%)). We find similar promising results even on high-dimensional complex datasets e.g., on CUB200, a 23% 155.%%-41.99. performance drop of knockoff for 2% drop defender's test performance. Our results indicate effective defenses are achievable, where the defender can trade-off marginal utility cost to drastically impede the attacker.
5.2.2 MAD DEFENSE VS. BASELINE DEFENSES
We now study how approach compares to baseline defenses, by evaluating the defenses against the knockoff attack (which resulted in the strongest attack in our experiments). From Figure we observe:

---

Published as a conference paper at ICLR 2020
MNSST
FasHonMMIST
CIFARIO
CIFAR100
CUOS009
Cahech2te
(b)
Acc(Attacker)
Acal Attacker
Nrtackrr)
Acc(Attacker]
Acd Nctaclee)
Aos(Patacker)
rudd
madh-an argimac
randhosie
dougsl
resernngyeid
ideal defiarse
Figure 4: Knockoff attack Ours Baseline Defenses (best seen magnified). Non-replicability is presented on the r-axis. On y-axis, we present two utility measures: (a) top: Utility L, distance (b) bottom: Utility Defender's accuracy. Region above the diagonal indicates instances where defender outperforms the attacker.
114. 100
80
Acc(Defender)
60-
0.00
MV
0.00
40-
argrae
100
20
0.04
rardom-soke
idal
125
Acc(Attacker)
Figure 7: Test loss. Visual-
Figure 6: Histogram Angular Devia- tions. Presented for MAD attack on CI- FAR10 with various choices of e.
ized during training. Colours and lines correspond val- ues Fig.
Figure 5: Attacker argmax. Follow-up to Figure 4p (CI- FARIO), but with attacker us- ing only the argmax label.
(i) Utility objective L1 distance (Fig. 4): Although random-noiss and reverse-sigmoid reduce attacker's 's accuracy, the strategies in most cases involves larger perturbations. In contrast, MAD and MAD-argmax provides similar non-replicability (i.e., Acc(Att.) with significantly lesser perturba- tion, especially lower magnitudes. For instance, on MNIST (first column), MAD (I,1 0.95) reduces the accuracy of the attacker to under 80% with 0.63x the perturbation as that of reverse- sigmoid and random-noise (L1 1.5).
(ii) Utility objective argmax-preserve (Fig. 40): By setting hard constraint retaining the label of the predictions, we find the accuracy-preserving defenses MAD-argmax and reverse-sigmoid successfully reduce the performance of the attacker by least 20% across all datasets. In most cases, we find MAD-argmax in addition achieves this objective by introducing lesser distortion the predictions compared reverse-sigmoid. For instance, in Fig. 4h, find MAD-argmax consistently reduce the attacker accuracy to the same amount at lesser L, distances. In reverse- sigmoid, we attribute the large L1 perturbations to shift posteriors towards a uniform distribution e.g., mean entropy of perturbed predictions is 3.02 + 0.16 (max-entropy = 3.32) at L,=1.0 for MNIST; in contrast, MAD-argmax displays mean entropy of 1.79 0.11. However, common to preeeeciyppeerringgg.o strategies is pitfall that the top-1 label is retained. In Figure 5 (see overlapping red and yellow cross- marks,, we present the results of training the attacker using only the top-1 label. In line with previous discussions, we find that the attacker is able to significantly recover the original performance of the stolen model for accuracy-preserviig defenses MAD-argmax and reverse-sigmoid.
(iii) Non-replicabilityy vs. utility trade-off (Fig. 4b): We now compare our defense MAD (blue lines) with baselines (rand-noise and dp-agd) which trade-off utility to mitigate model stealing. Our results indicate MAD offers a better defense (lower attacker accuracies for similar defender accuracies). For instance, to reduce the attacker's 's accuracy to <70. while the defender's accuracy significantly degrades using dp-sgd (39%) and rand-noise (56.4%), MAD involves a marginal decrease of 1 1%.
7

---

Published as a conference paper at ICLR 2020
CBARRC
CIARRO
1.4
100
1.2
Ac(DDoonke 7
1.0
Acc (Mefeedbe))
MAQ
0.3
60-
0.5
0.4
0.2
-0
0.2
20-
0.0
0.
70
70
40
40
50
67
70
00
90
100
40
67
00
100
Acc/Fetacke)
Acc/fittckar
ArttAetbeee))
Anc Att(derr
Figure 8: MAD Ablation experiments. Utility = (left) L1 distance (right) defender test accuracy.
Ieal
Figure 9: Subverting the Defense.
5.2.3 ANALYSIS
How much angular deviation does MAD introduce? To obtain insights on the angular deviation induced between the true and the perturbed gradient, we conduct an experiment by tracking the true gradient direction (which was unknown so far) at each training step. We simulate this by training an attacker model using online SGD (LR=0.001) over N iterations using B distinct images to query and a batch size of 1. At each step of training, the attacker queries a randomly sampled input at to the defender model and backpropogates the loss resulting from Dt. In this particular experiment, the perturbation Yr is crafted having exact knowledge of the attacker's parameters. We evaluate the angular deviation between gradients with (a) and without (u) the perturbation.
In Figure 6 we visualize a histogram of deviations: 0 = arccos [|ll||al|| where u = V y..
v-a
and a so [(w,, 9. .). We observe: (i) although our perturbation space is severely restricted (a low-dimensional probability simplex), we can introduce surprisingly high deviations (0-1159) in the high-dimennsonall parameter space of the VGG16; (ii) for values at reasonable operating points which preserves the defender's accuracy within 10% of the undefended accuracy (c.g.. e E [0.95, 0.99] for CIFAR10), that the perturbed gradient on an average leads to a slower decrease in loss function; (iii) on the extreme end, with Emax 2, the gradient direction leading to an increase on the test loss, as seen in Figure 7 (blue line). We also find the above observations reasonably transfers black-box attacker setting (see Appendix [:.4. where the perturbations are crafted without knowledge of the attacker's 's parameters. Overall, we find our approach considerably corrupts the attacker's gradient direction.
see deviations with mean 24.9º (yellow bars in Fig. 6). This indicates
average, we find the perturbations successfully flips (>-909)
Ablative Analysis. We present an ablation analysis of our approach in Figure g In this experiment, we compare our approach MAD and MAD-argmax to: (a) G I: We substitute the jacobian G (Eq. (5) with a K' x К identity matrix; and (b) =r=aand: Inner maximization term (Eq. 4 returns a random extreme of the simplex. Note that both (a) and (b) do not use the gradient information to perturb the posteriors.
From Figure we observe: (i) poor performance of =r=rand, indicating random untargeted per- turbations of the posterior probability is poor strategy; (ii) G = I, where the angular deviation maximized between the posterior probability vectors a slightly better strategy; (ii) MAD out- performs the above approaches. Consequently, we find using the gradient information (although proxy to the attacker's 's gradient signal) within our formulation (Equation 4) is crucial to providing better model stealing defenses.
Subverting the Defense. We now explore various strategies an attacker can use to circumvent the defense. To this end, we evaluate the following strategies: (a) argmax: attacker uses only the most-confident label during training; (b) arch-*; *; attacker trains other choices of architectures; (c) nquery: attacker queries each image multiple times; (d) nquery aug: same (c), but with random cropping and horizontal flipping; and (e) opt-*: attacker uses an adaptive LR optimizer e.g., ADAM (Kingma& & Ba 2014).
We present results over the subversion strategics in Figure 9 We find our defense robust to above strategies. Our results indicate that the best strategy for the attacker to cireumvent our defense
8

---

Published as a conference paper at ICLR 2020
to discard the probabilities and rely only the most confident label train the stolen model. In eecuracy-preservinn defenses (see Fig. [). this previously resulted in an adversary entirely cir- cumventing the defense (recovering up to 1.0x original performance). In contrast, we find MAD nonetheless effective in spite of the strategy, maintaining a 9% absolute accuracy reduction in attacker's stolen performance.
CONCLUSION
In this work, we were motivated by limited success of existing defenses against DNN model stealing attacks. While prior work is largely based on passive defenses focusing on information truncation, we proposed the first active defense strategy that attacks the adversary's training objective. We found our approach effective in defending variety of victim models and against various attack strategies. In particular, we find our attack can reduce the accuracy of the adversary by up to 65%, without significantly affecting defender's accuracy.
This research was partially supported by the German Research Foundation (DFG CRC 1223). We thank Paul Swoboda and David Stutz for helpful discussions.
Acknowledgement.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, 2016.
Ibrahim M Alabdulmohsin, Xin Gao, and Xiangliang Zhang. Adding robustness to support vector machines against adversarial reverse engineering. In CIKM, 2014.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. ECML PKDDD, 2013.
Varun Chandrasekaran, K Chaudhari, Irene Giacomelli, Somesh Jha, and Songbai Yan. Exploring connections between active learning and model extraction. arXiv preprint arXiv: M:X:::9905009100 2019.
Jacson Rodrigues Correia-Silva, Rodrigo Berriel, Claudine Badue, Alberto F de Souza, and Thiago Oliveira-Sane -Satoos. Copycat enn: Stealing knowledge by persuading confession with random non- labeled data. In IJCNN, 2018.
John Duchi, Shai Shalev-Shwarde Yoram Singer, and Tushar Chandra. Efficient projections onto the I b-ball for learning in high dimensions. In JCML, 2008.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 2014.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint vrXiv:1412.6577. 2014.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv:1503.02531, 2015.
Karla Leigh Hoffmann. A method for globally minimizing concave functions over convex sets. Math- ematical Programming, 20-3) 1981.
Matthew Jagiclski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High- fidelity extraction neural network models. arXiv preprint arXiv:1909.01838, 2019.
Mika Juuti, Sebastian Szyller, Alexey Dmitrenko, Samuel Marchal, and N Asokan. Prada: Protect- ing against don model stealing attacks. In Euro S&P, 2019.
Manish Kesarwani, Bhaskar Mukhoty, Vijay Arya, and Sameep Mehta. Model extraction warning in mlaas paradigm. In ACSAC, 2018.

---

Published as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against model stealing attacks using deceptive perturbations. S&P Deep Learning and Security (DLS) Workshop, 2018.
Daniel Lowd and Christopher Meek. Adversarial learning. In KDD, 2005.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt. Model reconstruction from model explanations. arXiv preprint arXiv: 1807.05188 2018.
Blaine Nelson, Marco Barreno, Fuching Jack Chi. Anthony Joseph, Benjamin IP Rubinstein, Udam Saini, Charles Sutton, JD Tygar, and Kai Xia. Misleading learners: Co-opting your spam filter. In Machine learning in cyber trust. 2009.
Blaine Nelson, Benjamin Rubinstein, Ling Huang, Anthony Joseph, Shing-hon Lau, Steven Lee, Satish Rao, Anthony Tran, and Doug Tygar. Near-optimal evasion of convex-inducing classifiers. In AISTATS, 2010.
Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black- box neural networks. In ICLR, 2018
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In CVPR, 2019.
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy, A framework for the extraction deep neural networks by leveraging public data. arXiv preprint arXiv:1905.09115. 2019.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint 1rXiv:660.002277, 2016.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In ICLR, 2017a.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. Asia CCS, 2017b.
Florian Tramer and Dan Boneh, Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In ICLR, 2019.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX Security, 2016.
Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning, In S&P, 2018.
Huadi Zheng, Qingqing Ye, Haibo Hu, Chengfang Fang, and Jie Shi. Bdpl: A boundary differen- tially private layer against machine learning model extraction attacks. In ESORICS, 2019.
10

---

Published as a conference paper at ICLR 2020
Appendix
A
OVERVIEW AND NOTATION
Victim/ DDefender
Utility Metric 2
Acc( PV, Diea))
collect
Dirain
train
Fv
+defense
F
D
test
535
Non-Replicability Metric
Utility Metric
2 Px(x)
Uh C
Acc(PA, D
y44 nod- hod
11
pertarhed
A
Dtransfer
train
FA
Adversary
Figure Overview Attack, Defense, Evaluation Metries. We consider attacker who exploits black-box access to defended model F to train a stolen model FA- this paper, we take the role the defender who intends to minimize replicability (i.c. Acc(FA, D(c)),, while maintaining utility of the predic- tions. We consider two notions of utility: (1) minimizing perturbations in predictions, measured here using Lı distance; and (2) maintaining accuracy of the defended model test Acc(F D Note that for fair head-to-head comparison, we use same held-out test set Dies evaluate accuracies of both the defended model F& and stolen model FA. Similar to all prior work, we assume Dimin Dies are drawn i..d from the same (victim) distribution Dv. Notation used in the above figure is further elaborated in Table Al
Inputs (images RCXHXW) Original, perturbed posterior predictions
y. A K
Probability simplex over K vertices
Attacker
(A(X)
Attacker's input data distribution
Dirunteer
Transfer set (= {(x y:)), where x; ~ PA(X), Y: Fv(x.))
FA
Attacker's 's (stolen) model trained on Diransler
Victin/Defender V
(x
Victim's input data distribution
Dirain
Training data (= { y1)) where Pv(X))
Fv
Undefended model trained on Ditain
F
Defended model
Diest
Test set (= {x 34)), where æ. Px()))
Table A1: Notation
B RELATED WORK: EXTENSION
A summary of existing model stealing attacks and defenses is presented in Table A2
C DETAILED ALGORITHM
We present detailed algorithm (see Algorithm 1 for our approach described in Section
The algorithm roughly follows four steps:
ii Predict (L2)): Obtains posterior probability predictions y for input æ using a victim model (w; w.
11

---

Published as a conference paper at ICLR 2020
Black-box type
Proposed Attack
Proposed Defense
Input Query Data
Adaatt?
Strategy
P:D2
AP?
2005
Random Noise
2.
Nelson di al
2019
Linear
Lahhlldd Data
Rejection
K
3.
Nelson of al. 2010
Linear
Random Noise
4.
Alakiuimeis
2014
Lincar
Random Noise
Ensembling
X
5.
Framer al 2016
Lineur, NN
Random Noise
Rounding
(013))
NN
Random Noise
7
Neserwam at 2018
Decision Tree
Detection
D
V
Chnddisskkarann
2019
Random Noise
9.
Papernot al 2017b
CNN
Synth. Data
2018
CNN
Unlabeled Data
Pai ct 20119
CNN
Unlaheled Data
2019
CNN
Unlabeled Data
Roundingg Top-k
/
12
13.
JageeSkki et
2019
CNN
Unlabeled Duta
14.
Juoti at
2019
CNN
Synth. Data
Detection
9,14
28 af
2013)
CNN
Reverse sigmoid
CNN
Targeted Pert.
9,12,14
Table A2: Existing DNN Attacks and Defenses. Complements the discussion in Section 2 ImageNet-like CNN. 't: Both. "P/D": Perturbattion/Deecttion 'AP': Accuracy preserving (1.c maintains top-1 labels of predictions). 'AC': Attacks considered.
CNNN Complex
(ii) Estimate Jacobian G (L3): We estimate a X jacobian matrix surrogate model F. By default, we use as F a randomly initialized model (more details in Appendix(E.1 EI. Each of G' represents the gradient direction (in parameter space I 2) over log likelihhod of class k.
(iii) Maximize MAD Objective (L44: We find the optimal direction y² which maximizes the MAD objective (Eq. To compute the arg max, iterative over the K extremes of the probability simplex to find y" which maximizes the objective. The extreme yı denotes a probability vector with yı 1.
(iv) Enforce Utility Constraint (L5-7): We enforce the perturbation utility constraint (Eq. 7 by considering a linear interpolation of y" and The resulting interpolation probability vector ÿ == h(a") represents the utility-consrrined perturbed prediction that returned instead of y.
Function PerturbedPredict-MAD
Input: Input data æ, model to defend Fv(;; wy), proxy attacker model F(; w)
Output: Perturbed posterior probability ÿ E s.t. dist(ū, v) 6
y Fy (x: wy)
// Obtain K-dim posteriors // Pre-compute (K x D) Jacobian
3 G = v log F(x; w)
y" = arg max (
G CG Ve
G" y G ツl
// Alternatively ext
for MAD-argmax
3 Define h(a) = (1 - a)y + az"
a" = arg max dis dist(h(a), v²) // Find optimal step-size via
bisection, or OptStep(.) for Lp norms
ÿ ; -aa))
// Perturbed probabilities
return
Function OptStep "", p): a' = max ll - 11
return
Algorithm MAD Defense. To supplement approach Section
12

---

Published as a conference paper at ICLR 2020
D ATTACK MODELS: RECAP AND IMPLEMENTATION DETAILS
Jacobian Based Data Augmentation (jbda) (Papernot et al. 2017b). The motivation of the approach is obtain a surrogate of the victim black-box classifier, with an end-goal of performing evasion attacks (Biggio et al., 2013 Goodfellow et al. 2014). We restrict discussions primarily to surrogate (the stolen model), the authors
the first part of constructing the surrogate. To obtain depend on an unlabeled 'seed' set, typically from the same distribution as that used to train the victim model. As result, the attacker assumes (mild) knowledge the input data distribution and the class-label of the victim.
The key idea behind the approach is to query perturbations of inputs, to obtain reasonable ap- proximation of the decision boundary of the victim model. The attack strategy involves performing the following steps in a repeated manner: (i) images from the substitute set (initially the seed) D is labeled by querying the victim model Fv on the substitute dataset; (iii) the substitute set is augmented using perturbations of existing images: D Pp+1 D, U {ar A++1 sgn(JF( [ (x)))
oracle labeler, (ii) the surrogate model FA is trained
E Dp), where J is the jacobian function.
We use a seed set of: 100 (MNIST and FashionMNIST), 500 (CIFARIO, R10 CUB200, Caltech256) and 1000 (CIFAR100). We use the default set of hyperparameters of Papernot al. (207bb) other respects.
Jacobian Based (self, top-k} (jbself, jbtop3) (Juuti et al. 2019 The authors generalize the above approach, by extending the manner in which the synthetic samples are produced. In jbself, the jacobian is calculated w.r.t to & nearest classes and in jb-self, w.r. the maximum posterior class predicted by FA.
Knockoff Nets (knockoff) (Orekondy et al. 2019). Knockoff is recent attack model, which demonstrated model stealing can be performed without access to seed samples. Rather, the queries to the black-hox involve natural images (which be unrelated to the training data of the victim model) sampled from a large independent data source e.g., ImageNetIK. Consequently, no knowl- edge of the input data distribution nor the class-label space of the victim model is required to perform model stealing. The paper proposes two strategies on how to sample images to query: random and adaptive. We use the random strategy in the paper, since adaptive resulted in marginal increases in an open-world setup (which we have).
As the independent data sources in our knockoff attacks, we use: EMNIST-Lettert (when stealing MNIST victim model), EMNIST (FashionMNISTT, CIFAR100 CIFARIO), CIFAR10 (CIFAR100) ImageNetlk k (CUB200, Caltech256). Overlap between query images and the training data of the victim models are purely co-incidental.
We use the code from the project's public github repository.
Evaluating Attacks. The resulting replica model FA from all the above attack strategies are eval- uated on a held-out test set. We remark that the replica model is evaluated as-is, without additional finetuning or modifications. Similar to prior work, we evaluate the accuracies of FA on the victim's 's held-out test set. Evaluating both stolen and the victim model on the same test set allows for fair head-to-headd comparison.
E SUPPLEMENTARY ANALYSIS
In this section, we present additional analysis to supplement Section 5.2.3
E. ESTIMATING G
Central to our defense estimating the jacobian matrix G = V log F(x; (Eq. 5 where F(:; (() ) is the attacker's model. However, defender with black-box attacker knowledge (where F unknown) requires determining G by instead using a surrogate model Fsur- We determine choice of Fuur empirically by studying two factors: (a) architecture of F choice of defender's 's surrogate architecture robust to varying attacker architectures (see Fig. A2); and (b) initialization of F F ini- tialization of the surrogate model parameters plays crucial role in providing better defense. We
13

---

Published as a conference paper at ICLR 2020
CIFAR10 F... VGG15
100
MNIST
FashionMNIST
CIFARIO
Acd Acc(Deeenderr
50
Acc Defeeeeerr)
60
60
50
40
40
20
0-
50
Acc(Attacker)
20
00
100
20 Acc(Attacker)
40
80
80
100
20
40
60
00
300
Acc (Attacke))
Acc (Attacker)
VGG16
rand
early
mid
late
ideal defense
Figure Influence attacker architec- ture choices on fixed surrogate.
Figure A3:
Influence of Initialization of VGG16 Surrogate Model. 'rand' = random initialization, ("carly', 'mid', "late") = (25, 50, 75)% % test accuracy of surrogate on test set.
Undefended
MAD
MNIST
0.88 + 14.41
6,47 + 12.25
FashionMNIST
0.89 + 15.76
6.65 + 14.16
CI
1.93 13.02
8.58 + 15.02
CIFAR100
2.15 + 18.82
69.26 21.4
CUBS200
4.45 + 9.66
446,93 + 23,87
Caltech256
4.93 + 21.25
815.97 + 30.3
Table A3: Run times (in ms). We report the mean and standard deviation of predictions of undefended and defended models, computed over 10K predictions.
consider four choices of initialization: ('rand', 'early', 'mid', 'late") which exhibits approximately (chance-level 25%, 50%, 75%) test accuracies respectively. We observe (see Fig. A3 that ran- domly initialized model, which far from convergence, provides better gradient signals in crafting perturbations.
E.2 RUN-TIME ANALYSIS
We present the run-times of our defended and undefended models in Table A3 The reported num- bers were summarized over 10K unique predictions performed on an Nvidia Tesla V100. We find optimization procedure Eq. 44 for all models take under second, with most 0.8s in the case of Caltech256. The primary computational bottleneck of our defense implementation is estimating matrix G R K in Eq. which currently requires performing K (i.e., number of output classes) backward passes through the surrogate model. Consequently. we find that our inference times on Caltech256 be further reduced to 0.3s + 0.04 by using efficient surrogate architecture (e.g., ResNet-34).
F ADDITIONAL PLOTS
F.1 ATTACKER EVALUATION
We present evaluation of all attacks considered in the paper undefended model in Figure A4 Furthermore, specific to the knockoff attack, we analyze how training using only the top-1 label (instead of complete posterior information) affects the attacker in Figure A5
F.2 BUDGET VS. ACCURACY
We plot the budget (i.e., number of distinct black-box attack queries to the defender) the test accuracy of the defender/attacker in Figure A6 The figure supplements Figure Tand the discussion found in Section 5.2.1 of the main paper.
14

---

Published as a conference paper at ICLR 2020
MN/ST
FashioMMNIS
CIFARID
CIFARIDU
CUBS2200
Cafteck250
1000
160
100
100
1.00
100
Nearos/Attalee))
60
80
80
80
80
201
60%
20k
43k
20k
43k
23k
40k
25k
40k
20k
43k
qucrrics
Qu2ries
qu-queri lts
quoriis
Juda
jinel
Johop3
knookeri
Figure A4: Evaluation of all attacks on undefended victim models.
MNIST
FashioMMNSST
CIFARED
CIFARLOU
CUBS200
Caltech250
1:01
100
100
2.00
100
100
Ncoracy/rtreee
80
80
80
50
60
60
00
00
60
40
40
40
40
20k
40k
20h
40k
0
20k quorics
40k
20k
40k
20k
40%
20k
40k
# que
querics
querics
querics
Posteriors
Top-1 Mbd
Figure Stolen model trained using knockoff strategy on complete posterior information (z) and only top-1 label of the posteriors (arg max, yk).
MMSS M EENISTLettt Lettess
85
Texl Actinay
MAA Undelesded
36k
48h.
Husigst
Budget
FasheoMMiSST EMNIST
D.2
83
D.92
100
Text Aucuracy
60-
Budget
Budget
Budget
Budget
0.1
8.5
0.00
CITARID CIFARII CITARIOO
100
Text Arunayy
20.
13
Eurlge
Euelge
Etsigat
Budget
Figure A6: Budget vs. Test Accuracy. Supplements Fig, 3c in the main paper.
15

---

Published as a conference paper at ICLR 2020
FachioMNIST-
CIFARIS-CIFARI00
100
300
100
Rcc(Deeender)
60-
Acc| (eefeende)))
60-
Acs(Deeendeer)
-00
randomn-ccw
MAD-argma
revene-igmml
25
25
75
Acc (Attacker)
Acc(Attacker]
Acc((Attackr))
Figure A7: Attacker argmax. Supplements Fig. 4 in the main paper.
MNIST
FashionMNIST
CIFAR10
10
1907
MNIST
FashionMNIST
CIFAR10
10P
10/
0.01
test- sattloos(attakker
20-1
test- loss (attacker)
test loss
0.1
0.5
20-1
1.0
00-2
2.0
10aa
40
Epach
Epoch
Epoch
Figure A8: Histogram of Angular Deviations (Black-box setting). Supplements Fig. 6 in the main paper. The test-loss during of the attacker model for each of the histograms (over multiple values) provided in the bottom row.
F.3 ATTACKER ARGMAX
In Figure A7 we perform the non-replicability vs. utility evaluation (complementing Fig. 5 in the main paper) under special situation: the attacker discards the probabilities and only uses the top-1 "argmax" label to train the stolen model. Relevant discussion can be found in Section 5.2.2
F,4 BLACK-BOX ANGULAR DEVIATIONS
provide the angular deviations obtained in black-box setting over the course of training the attack model. We train the attacker model using the transfer set obtained by the knockoff approach (the strongest attacker our experiments) for 50 epochs using a SGD (Ir 0.01, momentum = 0.5) and a batch size of 64. The experiment compliments our previous discussion in Section 5.2.3 of the main paper under "How much angular deviation does MAD introduce?"" As before, we estimate the angular deviations as: arceos u.o where u = WwL((w .) and [ы]]||||]
In Figure A8
L. -. We observe from Figure A8 (i) the defensive angular deviations introduced by MAD to posterior predictions transfer to a black- attacker setting, when crafting perturbations without access the adversary's model parameters; and (ii) although the setting introduces lower angular deviations at the extreme case of =22.0 (e.g.. 114.7 - 76.5 in CIFARI0), we observe the perturbation sufficient to maximize the attacker's test loss. We find significant angular deviations introduced by our approach in a black-box setting as well.
F.5 MAD ABLATION EXPERIMENTS
We present the ablation experiments covering all defender models in Figure/A Relevant discussion available in Section 5.2.3 of the main paper under "Ablative Analysis".
16

---

Published as a conference paper at ICLR 2020
1.5
12
1.-
MAD
MAD-relax
0.3
0.5
0.5
y yrarand
0.3
02
0.2
ideal
Acc(Actacker)
Acc(Attacker)
Acc Attacker)
100
100
100
(cc(eeeedeer)
80
Acc(Deeeeder)
80
(ac(DDeender)
40-
40
20
0
Attackr)))
Acc(Artacker)
Acc(Attacker)
Figure A9: MAD ablation experiments. Supplements Fig. 8 in the main paper.
17


Reviews:
[{"review_id": "r1ef7Qn-jr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method for defending against stealing attacks.\n\nPositives:\n1) The paper was very readable and clear.\n2) The proposed method is straightforward and well motivated.\n3) The authors included a good amount of experimental results. \n\n\nConcerns: \n1) You note that the random perturbation to the outputs performs poorly compared to your method, but this performance gap seems to decrease as the dataset becomes more difficult (i.e. CIFAR100). I’m concerned that this may indicate that the attackers are generally weak and this threat model may not be very serious. Overall, I’m skeptical of this threat model - the attackers require a very large number of queries, and don’t achieve great results on difficult datasets. Including results on a dataset like ImageNet would be nice. \n2) How long does this optimization procedure take? It seems possibly unreasonable for the victim to implement this defense if it significantly lengthens the time to return outputs of queries. \n3) Although this is a defense paper, it would be nice if the attacks were explained a bit more. Specifically, how are these attacks tested? You use the validation set, but does the attacker have knowledge about the class-label space of the victim? If the attacker trained with some synthetic data/other dataset, do you then freeze the feature extractor and train a linear layer to validate on the victim’s test set? It seems like this is discussed in the context of the victim in the “Attack Models” subsection, but it’s unclear what’s happening with the attacker. \n4) It would be nice to see an angular histogram plot for a model where the perturbed labels were not crafted with knowledge of this model’s parameters - i.e. transfer the proposed defense to a blackbox attacker and produce this same plot. This would motivate the defense more. \n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SygLUkwTFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at defending against model stealing attacks by perturbing the posterior prediction of a protected DNN with a balanced goal of maintaining accuracy and maximizing misleading gradient deviation. The maximizing angular deviation formulation makes sense and seemingly correct. The heuristic solver toward this objective is shown to be relatively effective in the experiments. While the theoretical novelty of the method is limited, the application in adversarial settings may be useful to advance of this research field, especially when it is relatively easy to apply by practitioners.I recommend toward acceptance of this paper even though can be convinced otherwise by better field experts.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJx6LZr0tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an effective defense against model stealing attacks. \n\nMerits:\n1) In general, this paper is well written and easy to follow.\n2) The approach is a significant supplement to existing defense against model stealing attacks. \n3) Extensive experiments. \n\nHowever, I still have concerns about the current version. \nI will possibly adjust my score based on the authors' response. \n\n1) In the model stealing setting, attacker and defender are seemingly knowledge limited. This should be clarified better in Sec. 3.  It is important to highlight that the defender has no access to F_A, thus problem (4) is a black-box optimization problem for defense. Also, it is better to have a table to summarize the notations. \n\nAdditional questions on problem formulation:\na) Problem (4) only relies on the transfer set, where $x \\sim P_A(x)$, right? \nb) For evaluation metrics, utility and non-replicability, do they have the same D^{test}? How to determine them, in particularly for F_A? \nc) One utility constraint is missing in problem (4). I noticed that it was mentioned in MAD-argmax, however, I suggest to add it to the formulation (4).\n\n2)  The details of heuristic solver are unclear. Although the authors pointed out the pseudocode in the appendix, it lacks detailed analysis. \n\n3) In Estimating G, how to select the surrogate model? Moreover, in the experiment, the authors mentioned that defense performances are unaffected by choice of architectures, and hence use the victim architecture for the stolen model. If possible, could the author provide results on different architecture choices for the stolen model as well as the surrogate model?\n\n############## Post-feedback ################\nI am satisfied with the authors' response. Thus, I would like to keep my positive comments on this paper. Although the paper is between 6 and 8, I finally decide to increase my score to 8 due to its novelty in formulation and extensive experiments. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an optimization-based defense against model stealing attacks.  A criticism of the paper is that the method is computationally expensive, and was not demonstrated on more complex problems like ImageNet.  While this criticism is valid, other reviewers seem less concerned by this because the SOTA in this area is currently focused on smaller problems.  After considering the rebuttal, there is enough reviewer support for this paper to be accepted.
Decision: Accept (Poster)