Defending against Data-Free Model Extraction by  Distributionally Robust Defensive Training
Authors: Zhenyi Wang
Li Shen
Tongliang Liu
Tiehang Duan
Yanjun Zhu
Donglin Zhan
David Doermann
Mingchen Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: Data-Free Model Extraction; Defense

Abstract: Data-Free Model Extraction (DFME) aims to clone a black-box model without knowing its original training data distribution, making it much easier for attackers to steal commercial models. Defense against DFME faces several challenges: (i) effectiveness; (ii) efficiency; (iii) no prior on the attacker's query data distribution and strategy. However, existing defense methods: (1) are highly computation and memory inefficient; or (2) need strong assumptions about attack data distribution; or (3) can only delay the attack or prove a model theft after the model stealing has happened. In this work, we propose a Memory and Computation efficient defense approach, named MeCo, to prevent DFME from happening while maintaining the model utility simultaneously by distributionally robust defensive training on the target victim model. Specifically, we randomize the input so that it: (1) causes a mismatch of the knowledge distillation loss for attackers; (2) disturbs the zeroth-order gradient estimation; (3) changes the label prediction for the attack query data. Therefore, the attacker can only extract misleading information from the black-box model. Extensive experiments on defending against both decision-based and score-based DFME demonstrate that MeCo can significantly reduce the effectiveness of existing DFME methods and substantially improve running efficiency.

Defending against Data-Free Model Extraction by Distributionally Robust Defensive Training
Zhenyi Wang', Li Shen', Tongliang Liu², Tiehang Duan', Yanjun Zhu Donglin Zhan David Docrmann", Mingehen Gao University of Maryland, College Park, USA 2JD Explore Academy, China The University Sydney, Australia WWest Virginia University, USA Northeastern University, USA CColumbia University, USA
"University at Buffalo, USA
zwang1690umm edu; mathshenli@@mail. com; tongliang.-iueeydney edu;
ti tichang. duan@gmail, com; ya. www.auoothhhssteer... edu;
dz2478@columbia. edu; {doermann, mgao@))buuffllo.ed.
Abstract
Data-Free Model Extraction (DFME) aims to clone black-box model without knowing its original training data distribution, making it much easier for attackers steal commercial models. Defense against DFME faces several challenges: (i) effectiveness; (ii) efficiency; (iii) no prior on the attacker's query data distribution and strategy. However, existing defense methods: (1) highly computation and memory inefficient; or (2) need strong assumptions about attack data distribution; (3) can only delay the attack or prove model theft after the model stealing has happened. In this work, we propose Memory and Computation efficient defense approach, named McCo, prevent DFME from happening while maintaining the model utility simultaneously by distributionally robust defensive training on the target victim model. Specifically, we randomize the input so that it: (1) causes a mismatch of the knowledge distillation loss for attackers; (2) disturbs the zeroth- order gradient estimation; (3) changes the label prediction for the attack query data. Therefore, the attacker can only extract misleading information from the black-box model. Extensive experiments on defending against both decision-based and score- based DFME demonstrate that MeCo can significantly reduce the effectiveness of existing DFME methods and substantially improve running efficiency.
1 Introduction
Model extraction attack aims to replicate the functionality of a public API with only query access. Most model extraction methods focus data-based model extraction, i.e., attacker can access small subset of the in-distribution training data 41 or a relevant surrogate dataset (36] of the target model. Beyond data-hased model extraction (DBME), recent promising results with data-free model extraction (DFME) show that the attacker can clone model with performance close to that of the target black-box model even without prior knowledge of the distribution of the proprietary training data. Those DFME techniques make it much easier for attackers to steal the model without collecting and annotating relevant training data. DFME can be further categorized into score- baaed DFME [1] where the target model provides softmax probability outputs to the users; and decision-based 54 47,, where the target model only provides the top-1 label. Thus, the model owners
DFME
CCorresponding author: Zhenyi Wang. Li Shen and Mingchen Gao
"The problem setting name, DFME, coincides with the method name in 51 We use DFME denote the problem setting of model extraction without accessing the in-distribution training data, specific method.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

face a more critical problem how to prevent those black-box pre-trained models from stealing in the data-free setting while simultaneously guaranteeing a high-quality inference service? The main challenges of defending against DFME are: (1) effectiveness requirement: the defender should substantially reduce the clone model accuracy; (2) efficiency requirement: the defense procedure should be memory and computation efficient; (3) lack of knowledge on the attackers: the attack query data distribution and attack strategy are unknown to the defender.
Existing work on Model Extraction (ME) defense primarily focuses on data-based ME attacks. We categorize the existing ME defense methods into pre-attack, delay-attack, and post-attack defenses. Pre-attack defenses aim to prevent ME attack from happening. They either perturb the output probabilities [38 33 or integrate multiple models (22] or detect attack queries from benign queries
[23] Delay-attack defenses aim to delay the ME attack instead of preventing it [TO]. Post-attack
defenses neither prevent nor delay the ME; instead, they aim to prove a model theft 17 32 Our
work falls under the pre-aftack defense category since our goal is to prevent ME from happening.
However, existing pre-attack defense methods have limitations when applied to DFME: (1) output perturbatton--aase methods perform optimization during deployment is computationally and memory expensive; (2) ensemble-based methods are memory inefficient since they need to store multiple models; (3) detection- based methods require strong assumptions about the query data distribution; the
attacker can easily make the attack queries indistinguishable from benign ones [59] and circumvent
the detection [28] rendering the defense ineffective; (4) some defense methods [33 23 22 require
the knowledge of the attack query data. However, this prior knowledge is unknown to the defender.
To address existing defense methods' limitations, we propose new and orthogonal class of defense method, named Memory and Computation efficient defense (MeCo), through a randomized defense strategy. Specifically, MeCo adds data-dependent random perturbation to the query input. MeCo can effectively defend against DFME for several reasons. For score-based DFME: (i) it leads to a mismatch of the knowledge distillation loss for attackers; (ii) existing DFME methods heavily rely on zeroth-order gradient estimation. MeCo can disturb the zeroth-order gradient estimation so that the attacker can only obtain its inaccurate estimation. For decision-based DFME, MeCo can change the label prediction of attack query data. The attacker can only learn from incorrectly labeled data.
MeCo would reduce the target model utility on benign queries without additional mechanisms. Maintaining the target model utility needs to : (i) minimally sacrifice the classification accuracy on the test set, and (ii) minimize the perturbation magnitude of the output class probability. To maintain the target model utility on benign queries, we propose a principled distributionally robust optimization (DRO) framework to train the target model and perturbation generator. Our DRO framework simulates the worst-case in-distribution (ID) test data from the training data (accessible for the defender). Then, we apply the random perturbation to the simulated test data and train the target model and perturbation generator on the simulated test data to ensure the worst-case generalization on the ID test data. Compared to existing works, MeCo has numerous advantages, including (I) McCo is substantially more computation and memory efficient without complex optimization and storing multiple models during deployment; (2) it avoids detecting attack queries from benign ones; (3) does not need the knowledge of the attack query data distribution. More importantly, even attackers know our defense strategy and adopt an adaptive attack strategy, MeCo is still effective since stealing a random function further increases the difficulty of ME thanks to the randomness introduced in MeCo. Extensive experiments compared various defense methods show that MeCo significantly reduces the accuracy of the clone model across different query budgets.
In summary, our main contributions are three-fold:
We propose a novel principled defensive training framework that substantially improves the memory and computation efficiency during deployment defend against DFME attacks.
We propose distributionally robust optimization (DRO) method to randomly perturb the inputs to defend against DFME effectively while maintaining the model utility simultaneously.
Extensive experiments on defending against both score-based and decision-based DFME show the effectiveness of MeCo, reducing the clone model accuracy by up to 35% while maintaining the target model utility. Further, MeCo can also effectively defend against data-based ME and boost its performance. More importantly, MeCo achieves substantially more computation and memory efficiency than cxisting methods, c.g., 17x 172x speed up.

---

2 Related Work
Data-Free Model Extraction (DFME). Model extraction (ME) attack aims to extract and clone the functionality of the public API with only query access; representative works include [31 9 50 [3
33
16 40 60 27.. Recently, ME has been extended to the data-free free setting, named DFME, which cannot access original training data. ZSDB3KD [54] and DFMS-HL HL [47] focus on the decision-based setting, ie., only the hard label (top-1 label) is predicted for the attacker. MAZE and DFME [21 are score-basee methods, i.e., soft label (softmax output) is delivered to the attacker.
36
7
63
Model Extraction Defense. We categorize the model extraction defense methods into three cat- egories: pre-attack defenses, delay-attack defenses, and post-attack defenses, according to when the defense happens during an attack. (i) Pre-attack defenses aim to prevent the ME attack from happening. There are three classes of methods: (1) output probabilities perturbation
methods 126 38 33: Prediction poisoning (P-poison) (38 and GRAD [K perform a complex optimization during deployment; (2) beeembleebssed methods: EDM [22 integrates multiple models for diverse predictions. (3) detection-based methods: Adaptive misinformation (23), PRADA [1] and VarDetect 39] detect the attack queries from the benign queries. However, those methods have limitations when applied to DFME: (1) they [38 [33 significantly increase the computation and memory cost;
(2) they 23 19 39 have a high risk of incorrectly classifying attack or benign queries. The attacker making the defense ineffective. (3) GRAD (3], Adaptive misinformation (33, and EDM 22 need to know the prior knowledge and distribution of attack query distribution, which is unknown to defenders in the data-free setting. Our method falls under this pre-attack defense category and addresses their limitations from a novel and orthogonal perspective. (ii) Delay-attack defenses aim to delay the ME attack instead of preventing the attack. Proof of work (PoW) [0] delays model stealing by significantly increasing the computation cost of query access for model extraction, e.g., solving a puzzle. Ours fundamentally different from PoW two aspects: (1) PoW could not prevent model stealing if the users spend more time, computation cost, money, etc. By contrast, our method is to prevent model stealing instead of delaying model stealing: (2) PoW needs multiple teachers to evaluate the privacy leakage of queries 42 Our method only requires a single teacher; thus, ours is substantially more memory and computation efficient than PoW. (iii) Post-attack defenses aim to prove model theft after pre-trained model has been stolen, e.g., through watermark- [2 methods 17 49), proof-of-learnngg [88] and dataset inference 32. However, the post-attack defenses only perform verification of model theft but cannot prevent the model from being stolen. This requires a model owner to obtain access to the stolen model. If the stolen model is not used as API, the defender cannot verify whether the model has been stolen.
can easily evade the detection 28
Distributionally Robust Optimization (DRO) DRO is flexible and powerful optimization frame- work to make decisions under uncertainty 43 (5), where robustness is an important factor 62] 56]
DRO constructs a set of probability distributions, known as an ambiguity set, and then minimizes the worst-case performance within the ambiguity set, thus guaranteeing the model performance. There have been various machine learning applications of DRO, such as dealing with group-shift [45]. subpopulation shift [6]), and long-tailed learning [57). To the best of our knowledge, our work is the first principled method with DRO for DFME defense.
3 Problem Setup and Preliminaries
3.1 Problem Setup
Data-Free Model Extraction (Attacker). In DFMF, the attacker sends a query input x to the target victim model T parameterized with 0r and receives a prediction P x)) T(æ; 0r). For the score-based setting [1). the target model delivers the soft label (output class probabilities) to the attacker. In the decision-based setting [54], the target model only delivers the hard label (the top-1 class) prediction to the attacker. Following 554] 21 511, we assume the following attacker knowledge: (1) data-free: the attacker cannot access the original training data distribution of the target victim model. The attacker typically employs synthetic out-of nut-of-distribution (OOD) data to query the target model an attempt to steal it. (2) black-box: the attacker does not know the architecture and model parameters of the target model. Given test dataset Diest associated with the black-box model, the attacker aims to train a compact clone model C' with parameters Oc that maximize the testing accuracy.
3

---

Defense against DFME (Defender). Following [8 22 we assume the defender does not know: (1) whether a query malicious or benign: (2) the attack strategies adopted by an attacker; (3) the model architecture used by an attacker. The goals of the defender are three-fold: (1) effective: minimize the test accuracy that the attacker can achieve; (2) utility preserving: maintain the accuracy on benign inputs and minimize the perturbation magnitude of the output probability; (3) efficient: the defense procedure should be memory and computation efficient.
3.2 Preliminaries
Knowledge Distillation (KD). Existing DFME methods build on top of KD [S] 14 44 58 24].
Assume we have a pre-trained teacher (target) model I with parameters Qr. and student (clone) model C with parameters Ic. Suppose the ouiput probabilities of teacher and student for the input æ are T(x; 0r) and Ca:; Oc), respectively. The training objective of KD is as the following:
((r. = Lc( (T 0r), C(x;; Oc):
(1)
where Le( y) is the cross-entrope loss, KL is the KL divergence between two probability distribu- tions, and a is the weighting factor. In this work, we focus on defending against score-based and decision-based DFME methods. Due to space limitations. we give brief description of ccore-based DFME while placing the details of decision-ber -based DFME in Appendix X
Score-based DFME The representative ccore-based DFME works are 51 21 We briefly describe how they work. The attacker has a pseudo data generator G parameterized by Ag with random vector as input. It generates pseudo data by the query æ the target and clone model; they output class probabilities according to the generated pseudo data, i.c., yr Tæ: 0r) and yc C(x; Oc). The attacker jointly optimizes the clone model C and generator below:
G(z; 0g), where & (0, I). Then, the attacker sends
Lc KL(yrllyc),
Lg -KL(yrllgc)
(2)
Since the target network black-box, to backpropagate into the generator network weights, they apply zeroth-order gradient [30] on the outputs of the generator, i.e.,
aLo aLa ox
aCa
1 Cola + 6pm)) Lo(x)
Vag Lo
where
(3)
d0G
Ox 00g²
dx
m
where μ4 is a random direction; m is the number of queries; ò is a small step size. The generator G and the clone model C alternatively update their parameters by minimizing the loss Le and Lg. The attacker can obtain a clone model C(ar; 0c) after a certain number of training iterations.
4 Methodology
To defend against DFME, we propose a DRO defensive training strategy. We present the defense method in Section 4.1
We then discuss the defensive training details and deployment algorithm in We illustrate how the proposed defense can defend against DFME in Section 4.3
Section
4.1 Distributionally Robust Defensive Training
The core techniques of the score-based DFME are two-fold: (1) using KD loss to match the target and clone model outputs; (2) zeroth-order gradient estimation. On the other hand, decision-based DFME mainly relies on the label prediction of the target model. We propose a random perturbation technique to make the attacker estimate misleading information. However, adding random noise to the query input strongly restricts the flexibility of random perturbation since it is difficult to maintain the model utility while effectively defending against DFME simultancously. Furthermore, the added randomness to the input should be input-dependent since different inputs have different sensitivities to the neural network decision boundary. We thus propose a flexible random perturbation generator to learn dete-dependent random perturbation that can adaptively generate different perturbations for different inputs while maintaining the performance on benign input. First, we denote the data-dependent random perturbation generator as (ar, é) parameterized by parameters w with query and random noise e as input. The perturbation generator is two-block ResNet with filter 64. only accounts for tiny proportion of the target model. We explain intuition in Figure]
Our method builds

---

Megata query
Reative query
Benign query:
Query
Semuraatin
Pertabbe dan
query:
Pertubbed query
Dac boundary
w/l perturbation
(a) Data-dependent perturbation
(b) Shift of data points
(c) Change of class probability with our defense method
Figure I: Illustration of our defense method. (a) The perturbation generator takes the input image and random Gaussian noise as imput to generate data-dependent perturbation, which is added to the original data to disturb model extraction. (b) According to [54] the attack queries are closer to the decision boundary than the benign (positive and negative) queries. All queries with added perturbation become causing some attack queries to invert labels while leaving labels benign queries unaffected.
closer the decision boundary,
(c) The perturbation generator generates large perturbation on attack queries so that the output class probabilities are perturbed more significantly (top-1 label may change). In contrast, the outputs on benign queries are only perturbed slightly due the DRO defensive training.
O Attack
O Attack query
Perturbed Attack query Perturbed Berigg query
query
Perturbed query Perturbed Burign quary
Berign query
A
Benign query
Berágn query
Be quary data
DRO medial stlliy
(a)
(b)
Figure Illustration of our proposed defense mechanisms. (a) Using query inputs without any input perturbation results in favorable model utility but could not defend against model extraction attack. (b) The red region represents the distribution of benign query data. Applying input perturbation to every input yiclds excellent defense performance but compromises model utility. (c) The red region illustrates the distribution of benign query data. Distributionally robust optimization (DRO) ensures model utility by minimizing loss the worst-case perturbed training data (simulation of test data), leading to significantly smaller perturbation magnitudes on benign inputs compared to training without DRO. Beyond the distribution of benign query data distribution (attack queries), DRO isn't employed, leading to arbitrary perturbation magnitudes on those inputs and strong defense against DFME.
on the intuition 54 that the attack query is closer to the decision boundary, while the benign query is farther away from the decision boundary in the DFME setting. We add random perturbation to the query input so that the attack query closer to or crosses the decision boundary. Their output probabilities from the target model will be perturbed more significantly in the score-based setting. Their labels are more likely to be flipped in the decision-basee setting. contrast, benign queries are far from the decision boundary, thus not influencing benign queries much.
Simply adding the proposed data-dependent random perturbation to the query inputs would reduce the model utility. We propose Distributionally Robust Optimization (DRO) framework to train the target model and perturbation generator to maintain the target model utility, i.e., (i) minimally sacrifice the classification accuracy on the test set, and (ii) mini- mize the perturbation magnitude of the output class probabilities. addition, as depicted in Figure 2, we conducted comparative analysis of three scenarios: one without random input perturbation, another with random input perturbation, and the third involving DRO defensive training. Our DRO defensive train- ing exhibits dual capability in effectively defending against DFME while maintaining the model utility.
DRO
APYAMEES AETARISS BAR OUR OUR BUERT DO ONTa CROTEUTTIO
Training dats
Simulsted testing data
Figure 3: The proposed DRO framework perturbs original training data distribution to simulate worst- test data distributions considering the high uncertainty of data distribution represented by blue and orange color on right figure.
Since the test data distribution is unknown during deployment, we propose flexible framework with DRO optimize under uncertainty (3). DRO constructs an ambiguity set probability distributions and optimizes the worst-case performance within the ambiguity set, thus guarantceing
5

---

the performance, Our framework takes that the underlying probability distribution of test data is unknown and lies in an ambiguity set of probability distributions around the training data distribution M4-- The proposed DRO framework perturbs the training data distribution to simulate the test data distribution, illustrated in Figure [ It optimizes the worst-case performance in the ambiguity set to guarantee the model utility after adding random perturbation. We formulate the proposed DRO framework in the probability measure (distributions or densities) space:
min sup EL C0mpp(00)) x + www (x) E), Dr, pEP
(4)
s.t. P - {u: (L(())))) S }3
(5)
Ex [T + hww (ax E)) T(>)| ST.
(6)
where the inner sup optimization in Eq. (4) is to compute and simulate the worst-case test data distribution (denoted as π) around the training data distribution (denoted no). I is defined as the probability distribution that achieves uuP Eg x + (2, (,.,,)))))))))))))) c). - P in Eq. (5) denotes the ambiguity set of probability measures for the test data distribution to characterize its uncertainty. One common choice to define P is through KL divergence. (M]] denotes the KL divergence between probability measure (Ao and μ.. ,3 is a constant to characterize the closeness between μoo and μe to ensure the worst-case test data distribution π does not deviate from the raw training data distribution (40 much. Eq. (6) ensures that model utility (output class probabilities, i.e., T(x)) does not change much after adding random perturbation, i.e., T(x + hww (2z, e)). The probability perturbation magnitude is measured by 1 norm, i.c., || 111. It is crucial to guarantee target model output class probabilities after query perturbation close to that of without query perturbation since benign users need these informative probabilities to derive helpful knowledge
33 is a constant.
To solve the above optimization problem, we convert Eq. optimization problem by Lagrange duality (6) (detailed derivations are put in Appendix I1.1):
6 into the following unconstrained
min sup[Em--p L(Or, (2, e). y) (_(م|])))) + -Em=-lll(( (z) T(=)l (7)
By:
The constant y controls the regularization magnitude for model utility. The KL-divergence KL (loo) is handled by Wasserstein gradient flow (WGF) (presented in the following sections). therefore for
simplicity, the regularization weight for it is set to 1.0 throughout the paper. We name Eq. 7 as Defense DRO. The optimization in Eq. is still challenging to solve the inner sup optimization is over probability measure space, which is an infinite- ciintteddimensionall function space.
Solution to the Defense DRO. To make the solution of the Defense DRO (Eq. 7 tractable, we reformulate it from a new continuous dynamics perspective. To achieve this goal, we first define the energy functional F1)) as follows:
F(n) = V(++ + KT(()+=))
(8)
where V(n)= = -E L(OT, x + (ar, e), v). By defining such energy functional F(x), the Eq. (7 can be equivalently formulated by the following gradient flow system Eq. p 10)
0,14 = Ww, F(u)) == div (4
8F on
(m:):
(9)
d0T
or Enwna C(T ; + la v).v) +yEE..I. + - (10)
dt
where Eq. (9) solves the inner sup problem in Eq. 77) with WGF in Wasserstein space (Details
provided in Appendix [1.2p and Eq. (TO) solves the outer minimization problem in Eq. for
parameter update with gradient flow in Euclidean space. Below, we propose a method for efficiently solving the Eq. (9. We view each training data one particle and arrive at the following test data distribution simulation solution (solution to Eq. ( (Details are provided in Appendix T1.2):
==N
Σ k(xi, x/) U(x// aU (iae/,00) 0r) + +V V k(r), x[)]
(11)
where oi denotes the tth training data x' perturbed at time t; a is the data transformation rate. (xx 0r) = - L(Or x..). k( is the Gaussian kernel, i.c. 2,) = exp(- [a::
(m==),))) =,)²), The
first term in the RHS of Eq. (TT) moves the training data towards the worst-case simulated test data distribution by raising the energy functional. The data distribution change is controlled by the sum of gradients from the mini-batch of the training data, which are weighted by the kernel function k(x; (())) smoothing the gradients of training data. The second term (kernel gradient) acts as a repelling force that keeps the transformed data from concentrating on a single data point, thus diversifying the simulated test data.

---

4.2 End-to-end Defensive Training for DFME
We summarize the defensive training algorithm in Algorithm 1 Line 3-4 simulates the worst-case test data distribution. Line 5-8 adds dete-dependent random perturbation to the simulated test data and trains the target model and perturbation generator on the simulated test data.
MeCo Deployment. During testing, we perform similar perturbation for each query. Given any query input x. Vp To, (x: + how (2e 6)): e ((0, I). Then, return the results Yp the user.
Algorithm McCo Training.
1: REQUIRE; Target model 7 with parameters 0r; data-dependent perturbation generator hw with parameters
w; target model learning rate yj; Q is the number test data simulation steps; Der is the training dataset. 2: for k: === to M do [M is the number of training iterations)
3: randomly sample new mini-batch data (æ,, y) from Dar
4: perturb mini-batch data be (æQ. y) Q steps Eq. II
5: add random perturbation to ( y) be (aq + ha (#q, v)
6: calculate the loss g(Or, w) = ( œQ + (=o, e), + + tww - T(=)))
7: train the target model by 0 0f
train the perturbation generator by w ++1 w (Vllgg(rr,)) end
9:
4.3 Why Can MeCo Defend against DFME
Below, we give explanations on why Algorithm could intrinsically defend against both score-based and decision-based DFME from the perspective: (1) mismatch the KD loss for the attacker; (2)
disturb the zeroth-order gradient estimation; (3) change the label prediction on attack query.
Mismatch the KD learning objective. The KL-divergence loss Eq. ① is critical to match the proba- bility output between the target and clone model. By adding data-dependent random perturbation, we can encourage the mismatch of the KD loss between the target model and clone model, i.c.,
(T + hw e): 0r), C(æ; 0c))
(12)
The mismatch would mislead the attacker to learn in the wrong direction since they think that T(æ + (æt, <); 0r) is the desired output for x but should be T(x, 0r), illustrated in Figure
Disturb the zeroth-order gradient estimation. By adding data-dependent random perturbation, the zeroth-order gradient estimation of score-based DFME (Le in Eq. (3) becomes:
Oee
aCa dx
Lo(x + ō4z + vt) Lo(x + vf)
(13)
m
8
where ii and vi are two data-dependers perturbations correspond to the two model inputs 8pp, and æ, respectively. Compared with Eq. (3). Eq. (1) would make inaccurate gradient estimation by changing the gradient estimation direction due to the inconsistent random perturbation added to different inputs. Consequently, this would lead the gradient of the pseudo data generator in the wrong direction, i.e., Vo. in Eq. (3) would be inaccurate. Then, the generator would not generate informative samples. The uninformative pseudo samples would cause the clone model Eq. (2) to leam incorrect information.
Therefore, by jointly (i) mismatching the KD loss for the attacker; (ii) disturbing the zeroth-order gradient estimation, MeCo can effectively defend against DFME, In summary, MeCo has many advantages compared to existing methods since it does not need to: (1) solve a complex optimization during testing; (2) store multiple models for ensemble; (3) detect the attack query from the benign query: (4) know the attack query data distribution. Due to the space limitations, we additionally show why MeCo can defend against decision- DFME methods by changing the label prediction on attack query in Appendix
Theoretical Analysis In this section, we delve into the theoretical analysis of our proposed method. We assess the effectiveness of our approach by analyzing it from the perspective of gradient-bias injection 22 As depicted in Figure our method progressively introduces perturbations into the gradients calculated by the attacker. This process increases the optimization error that accumulates during the model-stcaling process, rendering it more challenging for the attacker to successfully
7

---

(a)
(b)
Figure 4: Theoretical explanations illustration: (a) We showcase the optimization trajectory employing the ground truth loss function that the attacker should idcally adopt (it remains inaccessible to the attacker owing the unknown input perturbation generator). The cloned model converges toward the globally optimal stolen model, effectively emulating the target victim model. (b) Contrastingly, we depict the optimization trajectory utilizing a noisy and imprecise loss function, which is the actual choice gradient bias due to the stochastic input perturbation generator causes the cloned model to deviate from the optimal stolen model, leading to ineffective model theft.
attacker. The introduction of
extract the target model. For a more comprehensive understanding, we have provided detailed theoretical analysis in the Appendix due space constraints.
5
Experiments
5.1 Experimental Setup
Datasets. We perform experiments on four standard datasets used in DFME literature, including MNIST, CIFAR10, CIFAR100
Table 1: Clone model accuracy after applying defense meth- ods on CIFAR-10 and CIFAR-100 with ResNet34-8x as the target model, which provides soft label
25 and MinilmageNet [52 (100 classes).
CIFARIO Clone Model Architecture
Defense
Baselines. We compare SOTA DFME and defense baselines. Attack Baselines: (1) decision-based DFME methods: DFMS-HL [7]. We do not compare to ZSDB3KD [51)
ResNeit8-8X
MobileNetV2
DenseNet121
undefended J
87.36 :: 0.78%
75.23 :: 1.50%
73.89 4: 1.29%
RandP J
1.37%
70.56
2.23%
70.03
2.345
P-poison
1.735
1.36%
GRAD J
70.33 =: 1.68866
65.82
1.67%
00.06
1.57%
McCo
51.68
1.96%
46.53
2.09%
61.38
2.41%
since requires much larger number of queries and is very slow. (2) score-based DFME methods: MAZE 21 and DFME 51 Defense Baselines: we compare to: (I) Undefended: the target model without using any defense strategy; (2) Random Perturb (RandP) [38]: randomly perturb the output probabilities. (3) P-poison 38]; (4) GRAD
undefended
0.733%
1.67%
20.00
1.79%
RandP J
28.76 =: 2.38%
22.03
1.50%%
18.79
1.3855
MAZE
P-poison
26.81 + 2.1956
20.89
1.8%
17,08
2.28%
GRAD
McCo
21,89 =: 2.07%
18.75
2.11%
17.95
4645
Attack
Defense
Clone Model Architecture
ResNetiS-SX
MobileNetV2
DenseNet121
undefended.
58.72
2.82%
28. + 1.97%
27.28 2.08%
RandP 1
41.69 0:9:::20 2.91%
22 2.19%
23.61
70%
DEME
P-poison |
38.72 4: 3.06%
20.87 4: 2.61%
21.89 1: 2.90%
[33] gradient redirection defense. We a large 4 perturbation budget equal to 1.0 for those defense baselines in the experiments
GRAD
39.07 2.72%
20.71 1: 2,80%
22.08
2.78%
1.05%
1.36%
MAZE
to generate strong defense. That is. yyy 1l1 < 1.0; where y and ÿ are the unmodified/modifie output probabilities, respectively. We put more baseline details in Appendix I0.
Implementation Details. The random perturbation generator is two-block ResNet structure with a filter size of 64, which small network compared to the backbone (only accounts for tiny proportion of the backbone). For decision-based DFME methods, following [47]. we use query budget 10M for CIFAR100 and 8M for CIFAR- For score-based DFME methods,
following [51], we set the number of queries to be 2M for MNIST, 20M for CIFAR10. 10, and 200M for CIFARI00, respectively. We perform each experiment for runs with a mean and standard deviation of results. We provide more implementation details in Appendix 7
5.2 Results of Defense against DFME
Clone model accuracy. For score-based DFME setting (soft label), we show the results on CIFAR10 and CIFAR100 in Table [ For decision-based DFME setting (hard label), we show the results in

---

Table 2: Clone model accuracy after applying different defense methods on CIFAR- 10 and CIFAR-100 with ResNeet4-8x the target model, which only provides hard label
CIFARIO Clone Model Architocture
CIFARIOO Close Model Architecture
Attack
Defense
R-SNer18-8X
MohileNetV2
De
ResNer18-8X
MobileNetV2
De
undefended
81.67 = 1.90%
79.28 = 1 187%
68.87 1: 2.08%
72.57 = 1.28%
02.71 :: 1 1.68%
63.58 = 1.79%
DRMS-HL
RandP
P-paison
84.02
81.06
2.31%
187%
78.71
79.02
1.96%
1 1.9%%
68.05
68.16 4 2 25%
72.43
71.83
1.45%
L.32%
02.06
1.79%
1 1.2
63,16 1.75%
62. 02.73-
1.91%
GRAD
84.28
1.985%
78.83
= 1.91%
68.11 1.90%
71.80
1.37%
82.60
+ 1.71%%
62.57
1.80%
McCo L
76.86
2.09%
71.22
1.87%
62.33
2.01%
59.30
1.70%
55.32
1.68%
56.80 1.86%
Table 3: Target model utility (test accuracy and 1 norm of the output probabilities perturbation magnitude)
Method
MNIST
CIFARIO
CIFARI00
ACC
1000
ACC1
norm
ACC1
1 norm
undefended
08.01
0.16%
0.0
94.91
0.37%
76.71
1.25%
RandP
98.52 + 0.19%
1.0
53.98
0.28%%
1.0
76.23 + 1.999%
1.0
P-poison
98.87
0.35%
1.0
94.38
0.61%
1.0
75.42 1.2%
1.0
GRAD
0.31%
94.65
75.60
1.4%%
McCo
98.63 2: 0.28%
0.0126
91.17 2: 0.56%
0.099
75.30 4: 0.08%
0.312
Table 2 Due to the space limitations, we present the results on MinilmageNet and MNIST in Table and 7 Appendix. & indicates the accuracy the lower, the better; 1 indicates the accuracy the higher, the better. For 1/AAFFRR/I in Table I we use ResNet34 [3] as the target model. The clone model architecture includes, ResNet- 18, MobileNetV2 (4). DenseNet [15]. We further change the target model architecture as GoogLeNet 48 for CIFARIO in Table 2 in Appendix.
The results show MeCo significantly reduces the effectiveness of existing DFME methods by up to 35% and is substantially more effective than the compared methods since (1) RandP lacks data- dependent information; it maintains the utility for almost all the query data, which is unnecessary for attack queries since the attacker can still learn useful information. (2) P-poison needs a random initialized surrogate attacker model, which acts as an adversary model. (3) GRAD needs to know the attack query set train the surrogate model. Those surrogates have large gaps compared the DFME attacker model. Those methods thus perform poorly since the attack query data distribution and model are unknown the defender. While McCo does not need a surrogate model.
Target model utility. We evaluate the target model utility by (1) target model test accuracy after adopting the defense strategy; (2) l norm between the target model output probabilities with and without input perturbation averaged on the test dataset, i.e., Eg-D,... ...........1 [[ h (x.e)) c)) - -T())] 1- The results (see Table 3 (MNIST with LeNet, CIFAR10 10 and CIFARI00 with ResNet34-8x) 8x)) indicate that MeCo maintains the target model utility with a slight trade-off of benign accuracy but with much better preservation of output probabilities in terms of I norm. The baselines have large perturbations because the baseline defense methods perturb all the query data with the same magnitude (4 perturbation budget of 1.0); this unnecessary since the in-distribution benign query does not need such significant perturbation. In contrast, McCo explicitly optimizes the 4 norm of perturbation with DRO on the simulated test data distribution so that the perturbation magnitude is much smaller.
5.3 Application on Data-Based ME (DBME) Attack
We apply McCo to defend against traditional DBME methods with Knockoff Nets 37 and Jacobian- Based Dataset Augmentation (JBDA) [41]. We present the results in Appendix 9.1 McCo can significantly outperform existing methods and further reduce the clone model accuracy. When the distribution of attack query data is closer to the training data distribution of the target model, the effectiveness of MeCo defense weakens. This is due the fact that MeCo applies smaller perturbations to the data distribution similar to the training data of the target model as a result of defensive training.
5.4 Adaptive Attacks
We further analyze the robustness of MeCo against the attacker's countermeasures. Namely, we consider the situations where attackers know about our defense and have tailored the attacks to our defense method by adding data-dependent random perturbation to the query inputs to learn an additional random perturbation generator. In Table 4 MeCo is still effective since the random perturbation added by the defender and attacker are different due to randomness. There is still a mismatch in the KD loss and zeroth-order gradient estimation for the attacker. Interestingly, the performance of the clone model becomes even worse after the attacker performs adaptive attack.

---

We believe this is because after the attacker adds the random perturbation, they will need to learn random function. increasing the difficulty of model extraction. In addition, for score-based DFME. we also compare to the adaptive attack method where attackers use only hard label, not probability outputs. We show the results Table 5 MeCo is still very effective since the pseudo samples are nearer to the decision boundary, and perturbing the input would easily change the model outputs on those samples. Consequently, the attacker still cannot learn useful information.
following 38 33
5.5
Ablation Study
Effect of DRO. We evaluate how much improvement DRO can bring the model urility preservation in Table 12 in Ap- pendix. We can observe that with DRO, our method significantly improves the model utility by 5.3% 5.6% compared to the one without DRO training.
Table 4: Clone model accuracy after applying adaptive attack CIFARIO with ResNet34-8x the target model
Attuck
Defense
Clone Model Architecture
ResNet18-8X
MobileNetV2
DenseNeti21
undefended
87.36 0.78%
76.23 1.3%%
73.89 + 1.25%
DFME
McCo
McCo, aduptive
51.68 4: 1.96%
25.79 =: 0.81 %
20.18 + 1.17%
16.53
2.09%
22.32 = 1.22%
61.38 2: 2.11%
undefended
45.17 0.73%
6.6%%
20.03 1.79%
Effect of different query budgets for attacker. To evaluate the effect of dif- ferent query budgets on defense perfor- mance, we evaluate the clone model ac- curacy with varying defense methods on CIFAR10 in Figure Appendix. MeCo substantially outperforms various base- lines with varying query budgets.
MAZE
McCo adaptive 1.
McCo
23
19.82 2.03%
2..0%
18.75
18.27 + 2.23%
2.%1%
17.08 :: 2.28%
16.31
1.76%
Table 5: Clone model accuracy that attacker only uses hard label instead of output prohabilities CIFAR10
Attack
Defense
Clone Model Architecture
ResNetiS-SX
MobileNerV2
DenseNe:121
undefended
0.78%
+: 1.58%
73.89 +: 29%
DEME
McCo, hard label J
McCo, soft label
41.23 1: 0.58 %
51.68
1.96%
38.67 =: 0.97%
46.53 #: 2.00%
61. 38 + 2-1%%
42.31 1: 1.78%
We evaluate the hyprrarraeeter sensitivity for y and Q in Table I0 and Table Min Appendix. We observe that model utility increases as Ty increases with a trade-off of a decrease defense performance. Results show that Q = 2 performs the best. With the increase of Q, the simulated test data may be harder to learn; thus, the benign accuracy slightly decreases.
Hyperparameter Sensitivity.
Test time speed and memory comparisons. We compare the running time of best-performing methods in Table T3 Appendix. MeCo achieves 17 ~ 172x speed up on CIFAR10 and CIFAR100, This is because P-poison and GRAD solve computationally expensive optimization during testing. In contrast, MeCo does not need this optimization. We provide the results in Table [4] in Appendix for memory consumption evaluation. MeCo is very memory efficient compared baselines.
Training efficiency. We compare our training efficiency to baselines in Table 15 in Appendix. MeCo increases the training cost by 1.3x. However, MeCo substantially improves the test time running efficiency. We believe this slightly additional computation cost is worth it.
Conclusion
In this paper, explore the defense against DFME. We propose memory and computation efficient (MeCo) defense method through distributionally robust defensive training by adding a data-dependent random perturbation generator to perturb the input data so that the attacker cannot steal useful information from the black-box model. At the same time, MeCo maintains the target model utility. Extensive experiments demonstrate MeCo's effectiivnnss, computation, and memory efficiency.
Limitations and Broader Impacts. Our proposed defensive training helps build safe and trustworthy AI systems. This would be beneficial for protecting current large-scal pre-trained models used in the public APIs since they spend lots of money and time to tume and deploy those valuable large-scale models. However, sharing models and insights is key driver of progress in the Al research field, and if the community becomes more protective of their models, it could slow down advancements.
Acknowledgement This research was partly supported by NSF through grants IIS- and IIS 2239537. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318 18, DP220102121, LP220100527. LP220200949, and IC19010003 We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the A5000 used for this research.
10

---

References
[1] Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In 27th USENIX Security Symposium (USENIX Security 18), pages 1615-1631, Baltimore. MD, August 2018. USENIX Association.
[2] Ahmad Ajallocian and Scbastian U Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008. 0005), 2020.
[3] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savare. Gradient flows: In metric spaces and in the space of probability measures. (Lectures in Mathematics. ETH), 2008.
[4] Buse Gul Atli, Sebastian Szyller. Mika Juuti, Samuel Marchal, and N. Asokan. Extraction of complex dan models: Real threat boogeyman? Onn Shehory, Eitan Farchi, and Guy Barash, editors, Engineering Dependable and Secure Machine Learning Systems, pages 42-57. Springer International Publishing, 2020.
[5] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems, 2014.
[6] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[7] Varun Chandrasekaran, Kamalika Chaudhuri, Trene Giacomelli, Somesh Jha, and Songbai Yan. Exploring connections between active learning and model extraction. In 29th USENIX Security Symposium (USENIX Security 20), pages 1329-1326. USENIX Association, August 2020.
[8] Sinho Chewi, Thibaut Le Gouic, Chen Lu, Tyler Maunu, and Philippe Rigollet. Svgd as kernelized wasserstein gradient flow of the chi-squared divergence. In Advances in Neural Information Processing Systems, 2020.
[9] Jacson Rodrigues Morreia-Silvar Rodrigo Berriel, Claudine Badue, Alberto de Souza, and Thiago Sliviia-Saatoss Copycat CNN: Stealing knowledge by persuading confession with random non-labeled data. In 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, jul 2018.
[10] Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen La, and Nicolas Papernot. Increasing the cost of model extraction with calihrated proof of work. In International Conference on Learning Representations, 2022.
[11] Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial extreme value analysis. In International Conference Learning Representations, 2022.
[12] Junfeng Guo, Yiming Lixu Wang. Shu- T Xia, Heng Huang, Cong Liu, and Bo Domain watermark: Effective and harmless dataset copyright protection is closed at hand. In Thirty-seventh Conference on Newral Information Processing Systems, 2023.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[14] Geolfrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
[15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2261-2269, 2017.
[16] Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks. In USENIX Security 2020, 2020.
[17] Hengrui Jia, Christopher A. Choquette-Chov Varun Chandrasckaran, and Nicolas Papernot. Entangled watermarks as defense against model extraction. In 30th USENIX Security Symposium (USENIX Security 21), pages 1937-1954, USENIX Association, August 2021.
[18] Hengrui Jia, Mohammad Yaghini. Christopher A. Choquette-Choo Natalie Dullerud, Anvith Thudi, Varun Chandrasckaran, and Nicolas Papernot. Proof-of leloming- Definitions and practice. In Proceedings of the 42nd IEEE Symposium on Security and Privacy, 2021.
[19] Mika Juuti, Sebastian Szyller. Samuel Marchal, and N. Asokan. Prada: Protecting against dnn model stealing attacks. In 4th IEEE European Symposium Security and Privacy, 2019.
11

---

[20] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradiest methods under the polyak-lojasiewizz condition. In Machine Learning and Knowledge Discovery in Databasex: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16, pages 195-811. Springer, 2016.
[21] Sanjay Kariyappa, Atul Prakash, and Moinuddin K Qureshi. Maze: Data-free model stealing attack using zeroth-order gradient estimation. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[22] Sanjay Kariyappa, Atul Prakash, and Moinuddin K Qureshi. Protecting [dnn]s from theft using an ensemble of diverse models. In International Conference on Learning Representations, 2021.
[23] Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adaptive misinformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrn Recognition, 2020.
[24] Jangho Kim, ScongUk Park, and Nojun Kwak. Paraphrasing complex network: network compression via factor transfer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2765-2774, 2018.
[25] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
[26] Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against machine learning model stealing attacks using deceptive perturhations, 2018
[27] Guanlin Li, Guowen Xu, Shangwei Guo, Han Qiu, Jiwei Li, and Tianwei Zhang. Extracting robust models with uncertain examples. In The Eleventh International Conference Learning Representations, 2023.
[28] Huiying Li. Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y. Zhao. Blacklight: Scalable defense for neural networks against query-based black-box attacks. In 2022 Conference 31th USENIX security symposium (USENIX Security 22). 2022.
[29] Qiang Liu. Stein variational gradient descent as gradient flow. Advances in Neural Information Processing Systems, 2017.
[30] Sijia Liu, Pin-Yo Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 2020.
[31] Daniel Lowd and Christopher Meek. Adversarial learning. In KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, page 641-647. Association for Computing Machinery, 2005.
[32] Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In International Conference on Learning Representattons, 2021.
[33] Mantas Mazeika, Bo Li, and David Forsyth. How to steer your adversary: Targeted and efficient model stealing defenses with gradient redirection. 2022.
[34] Erich Miersemann. Calculus of Variations, Lecture Notes. Leipzig University, 2012.
[35] Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario Fritz. Towards reverse-engineeriig black-box neural networks. International Conference on Learning Representations, 2018
[36] Trihhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: models. In IEEF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Stealing functionality of black-box
[37] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4949-4958, 2019.
[38] Tribhuvanesh Orekondy, Bernt Schiele. and Mario Fritz. Prediction poisoning: Towards defenses against dan model stealing attacks. In International Conference on Learning Representations, 2020.
[39] Soham Pal, Yash Gupta, Aditya Kanade, and Shirish Shevade. Stateful detection of model extraction attacks, 2021.
[40] Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy, Activethief: Model extraction using active learning and unannotated public data, In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, 2020.
12

---

[41] Nicolas Papernot, Patrick McDaniel. Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. ACM Asia Conference on Computer and Communications Security, 2016.
[42] Nicolas Papemot, Shuang Song. Iya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with PATE, In International Conference on Learning Representations 2018
[43] Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. 2019.
[44] Adriana Romero. Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. 2014.
[45] Shiori Sagawa, Pang Wei Koh. Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. International Conference on Learning Representations. 2020.
[46] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.
[47] Sunandini Sanyal, Sravanti Addepalli, and Venkatesh Babu. Towards data-free model stcaling in a hard label setting. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15263-15272, 2022.
[48] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Frhan, Vincent Vanhoucke, and Andrew Rahinovich. Going deeper with convolutions. In 2015 IEEF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
[49] Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N. Asokan. Dawn: Dynamic adversarial water- marking of neural networks. In Proceedings of the ACM Multimedia, 2021.
[50] Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter. and Thomas Ristenpart. Stealing machine learning models via prediction apis. In Proceedings of USENIX Security 2016, 2016.
[51] Jean- Baptiste Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papernot. Data-free free model extraction. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[52] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot lcarning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[53] Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning, In In the 39th IEEE Symposium on Security and Privacy, 2018.
[54] Zi Wang. Zero-shot knowledge distillation from a decision-basse black-box model. In International Conference on Machine Learning, pages 19675-10688 PMLR, 2021.
[55] Yihan Wu, Aleksandar Bojchevski, and Heng Huang. Adversarial weight perturbation improves generaliza- tion in graph neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 1041510025, 2023.
[56] Yihan Wa, Hongyang Zhang, and Heng Huang. Retrievalguard: Provably robust 1 1-nearest neighbor image retrieval. In International Conference on Machine Learning, pages 24266.24279. PMLR, 2022.
[57] Ziyu Xu, Chen Dan, Justin Khim, and Pradeep Ravikumar. Class-weighted classification: Trade-offs and robust approaches. Proceedings of the International Conference on Machine Learning, 2020.
[58] Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[59] Zhenrui Yue, Zhankui He, Huimin Zeng, and Julian McAuley. Black-box attacks sequential recom- menders via data-free model extraction. In Proceedings of the 15th ACM Conference on Recommender Systems, RecSys '21. Association for Computing Machinery, 2021.
[60] Santiago Zanella- BegellaBeeguein, Shruti Tople, Andrew Paverd, and Boris Köpf. Grey-box extraction natural language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12278 12286. PMLR, 18 24 Jul 2021.
13

---

[61] Runtian Zhai, Chen Dan, J. Zico Kolter, and Pradeep Ravikumar, Doro: Distributional and outlier robust optimization. Proceedings the International Conference on Machine Learning, 2021.
[62] Hongyang Zhang. Yihan Wu, and Heng Huang. How many data are needed for robust learning? arXiv preprint arX1v:2202.11592. 2022.
[63] Xinyi Zhang, Chengfang Fang, and Jie Shi. Thief, beware of what get you there: Towards understanding model extraction attack, 2021.
14


Reviews:
[{"review_id": "8yUq5LGukl", "reviewer": "Reviewer_VEKG", "summary": "This paper proposes a defense against data-free model extraction attack (DFME). The basic idea is to add randomized data-dependent perturbations to the input query. It proposes a new training method that considers the perturbation generator to mitigate the risks of dropping the benign accuracy. Namely, it will trian the model and the perturbation generator so that the model can behave properly on the benign queries.\n", "questions": "How does the method tell the differences between normal and malicious queries?\n\nIs there a theoretical guarantee of the proposed method?\n", "limitations": "The paper discussed the limitations and broader impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is interesting to add random perturbations to the test data for defending against the DFME attacks.\n\nThe paper is well organized, with all items discussed.\n", "weaknesses": "From the very high level, I do not see how this works. The method tries to add perturbations to the test data. This will obviously decrease the benign accuracy. But I am not able to identify what makes the training method work. In DFME attacks, the queries are just normal queries, which can be similar to the ones used in training or testing. I am not sure how the method can work without identifying different types of queries.\n\nOne argument may be the query in the attack will be different from the ones used in training. Does that mean that the defense assumes the attack query out of distribution? I think out-of-distribution attack queries will surely degrade the attack performance. \n\nIs there a theoretical guarantee of the proposed method?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback.\n\n**Q 1.1**: Adding perturbation to test data decreases benign accuracy.\n\n**A 1.1**: We explain why our method maintains benign accuracy by considering three cases: (We invite you to refer to **Figure 3 for visual depiction in the \"rebuttal.pdf\" within the global response**)\n\n* In the first scenario, target model undergoes standard training on the target dataset. Although model utility remains intact, its vulnerability to model theft is heightened due to the lack of defense mechanisms.\n\n* Second, random input perturbations are introduced to query data without our proposed defensive training. As a result, target victim model gains protection against theft attempts, but this comes at the cost of compromised model utility.\n\n* To tackle these challenges and achieve the goals of preserving model utility and establishing robust defenses against model extraction attacks, we introduce our distributionally robust optimization approach (outlined in Equations 4-6 in the main text). This procedure aims to uphold model utility by optimizing worst-case generalization on perturbed test data, achieved by simulating testing data from the training data. This approach applies the common assumption that benign queries share the same distribution as training data, a prevalent assumption in existing model extraction defenses [5].\n\n**Q 1.2**: DFME queries can be similar to the ones in training or testing. How does the method tell the differences between normal and malicious queries?\n\n**A 1.2**: \n* First, we would like to clarify that DFME attacks commonly operate under the assumption that the attacker lacks knowledge of the training data distribution employed by the target model. In other words, the attacker operates in a \"data-free\" manner, being able to employ **flexible data distributions that differ from the training data distribution** to interact with the target model [3,4].\nFurthermore, in light of the insights and findings presented in PoW [1], the data utilized for attack queries is synthetic and out-of-distribution (OOD) since these data are generated synthetically using a data generator. Consequently, they possess a **different distribution** compared to the data encountered during training or testing. \n\n* Additionally, we leverage the insights in [2], which observe the fact that attack queries tend to reside closer to decision boundary, while normal queries are positioned further away. Consequently, the output probabilities associated with attack queries experience more significant perturbations upon the application of input perturbation. Conversely, owing to normal queries distance from the decision boundary, the output probabilities of normal queries undergo comparatively milder perturbations.\n\n* We employ distributionally robust optimization, as Equations 4-6 in the main text, to ensure the preservation of model utility on the test data through training a data-dependent perturbation generator, denoted as $h_{\\omega}(x, \\epsilon)$. When $x$ corresponds to a benign query, one that aligns with the distribution of training data, the perturbation magnitude is inherently restrained due to the defensive training of the perturbation generator. This restraint arises from the training and simulation of these data points derived from training data, thereby guaranteeing model utility. In contrast, when $x$ pertains to an attack query, the perturbation generator generates entirely distinct random perturbations. This is because attack queries are nearer to decision boundary,  substantiated by [2]. As a result, the prediction class probabilities are perturbed more significantly with input perturbation.\n\n**Q 1.3**: defense assumes attack query OOD?\n\n**A 1.3**: Thanks for your questions.\nAs in earlier clarifications,  DFME attack query data comprises synthetic out-of-distribution (OOD) data. This is consistently substantiated by a range of existing studies [1,2,3,4]. Due to the attacker's ability to leverage diverse and appropriate data distributions for querying the target model, the exact distribution of these query data remains unknown to defender. Our proposed method perturbs these data and effectively ensures the preservation of model utility and attaining an excellent defense performance. The principles are  illustrated in **Q 1.1** and **A 1.1**.\n\nFurthermore, our method **does not rely on the assumption that attack query is OOD** and can be viewed as random smoothing: $T_{\\theta_T}^{S}(x) = \\int T_{\\theta_T}(x + h_{\\omega}(x, \\epsilon)) d\\epsilon$, where $\\epsilon \\sim N(0, I)$. Compared to learning a single function $T_{\\theta_T}(x)$, learning the smoothed function $T_{\\theta_T}^{S}(x)$ necessitates a larger number of queries for attacker due to the requirement of calculating the integral. By necessitating a larger query budget, our approach makes it considerably harder for attackers to successfully clone the model.\n\n**Q 2**: I think OOD attack queries will surely degrade the attack performance.\n\n**A**:  We would like to clarify that successful model extraction **does not necessarily require in-distribution query data**. As evidenced by existing DFME  attacks [3, 4], they employ out-of-distribution (OOD) data for querying the target model. Despite the utilization of such OOD data, these attacks still closely replicated the functionality of target model without degrading attack performance.\n\n**Q3**: theoretical guarantee\n\n**A**:  We invite you to refer to **global response** for theoretical analysis. We provide **Figure 1 in rebuttal.pdf in the global response** for visualization. \n\nReference:\n\n[1] Increasing the Cost of Model Extraction with Calibrated Proof of Work. ICLR 2022\n\n[2] Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model. ICML 2021\n\n[3] Data-Free Model Extraction. CVPR 2021\n\n[4] Towards Data-Free Model Stealing in a Hard Label Setting. CVPR 2022\n\n[5] Protecting DNNs from Theft using an Ensemble of Diverse Models. ICLR 2021\n"}, {"review_id": "nXLHZ64ZwQ", "reviewer": "Reviewer_v48m", "summary": "In this paper, authors propose a novel principled defensive training framework that substantially improves the memory and computation efficiency during deployment to defend against DFME attacks and a distributionally robust optimization method to randomly perturb the inputs to defend against DFME effectively while maintaining the model utility simultaneously.", "questions": "See weakness", "limitations": "This method does not adequately evaluate the negative effects of the target model.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I agree that this work would be beneficial for protecting current large-scale pre-trained models used in the public APIs.\n\n2. The experiment in this article is sufficient and the description is clear. And the method proposed by the author is novel to a certain extent.\n\n3. The author's description of the existing research background is clear, and the analysis of the problems existing in the field is satisfactory.", "weaknesses": "1. In my opinion, the method proposed by the author has defects in protecting the performance of the original model, that is, it does not fully take into account the impact on the performance of the target model, which can also be seen from Table 3. After all, the weak accuracy improvement of these models in practical applications is expensive.\n2. As stated by the author, the methodology of this article lacks a theoretical explanation. The intuitive explanation of the mechanism of action of the model is not deep enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation for your valuable suggestions.\n\n\n**Q1**: In my opinion, the method proposed by the author has defects in protecting the performance of the original model, that is, it does not fully take into account the impact on the performance of the target model, which can also be seen from Table 3. After all, the weak accuracy improvement of these models in practical applications is expensive.\n\n**A**:  We appreciate your comments and your acknowledgment of the challenges involved in defending against model extraction attacks. In practice, developing effective defenses necessitates striking a balance between benign accuracy and defense performance.\n\n  * In our research, we thoroughly evaluated various defense baselines, including GRAD, P-poison, and Random defense, etc. It is important to note that most compared baselines sacrifice benign accuracy to enhance defense performance, our proposed defense method strives to achieve a more optimal trade-off.\n\n* As demonstrated in Table 3, our defense methods significantly reduce output probability perturbations when compared to other defense approaches. This improvement highlights the effectiveness of our method in mitigating the risk of model extraction attacks with only slightly compromising the model's performance on benign inputs.  This is beneficial for benign users in practical applications, while the compared defense methods need to significantly modify the output probabilities.\n\n* Regarding the slight reduction in benign accuracy, it is essential to emphasize that this decrease is minimal and does not significantly impact the overall model performance. In most cases, our proposed method only slightly reduces the benign accuracy, e.g., 0.28% on Minist, 0.74% on CIFAR10, and 1.35% on CIFAR100. Despite this marginal drop in accuracy, our method remains competitive with the baseline defense methods. Additionally, our method is significantly more efficient and effective in defending against model extraction attacks as shown in other tables. \n\n * Overall, our work aims to strike a balance between robustness against model extraction attacks and maintaining competitive accuracy levels.\n\n\n\n**Q2**: As stated by the author, the methodology of this article lacks a theoretical explanation. The intuitive explanation of the mechanism of action of the model is not deep enough.\n\n**A**: Thank you for pointing out this. We provided a comprehensive theoretical foundation for our proposed methods, with the intention of illuminating the fundamental principles and mechanisms that underpin the effectiveness of our approach. These theoretical elucidations are further illustrated in the **global response**.  Furthermore, we also included figure visualizations to complement our theoretical explanations, as depicted in **Figure 1 in the 'rebuttal.pdf' within the global response**.\n\n\nOur method can also be analyzed from another perspective. The introduction of random input perturbation allows our method to act as a form of random smoothing, defined by the function: $T_{\\theta_T}^{S}(x) = \\int T_{\\theta_T}(x + h_{\\omega}(x, \\epsilon)) d\\epsilon$, where $\\epsilon \\sim N(0, I)$. This smoothing technique enhances our method's resilience against model extraction attacks. It is important to note that compared to learning a single function $T_{\\theta_T}(x)$, the process of learning the smoothed function $T_{\\theta_T}^{S}(x)$ necessitates a larger number of queries for the attacker due to the computational complexity of calculating the integral. \nBy necessitating a larger query budget, our approach makes it considerably harder and more resource-intensive for attackers to successfully clone the model. As a result, our defense method offers a robust and effective barrier against model replication.\n\n"}, {"review_id": "4hltrevLJm", "reviewer": "Reviewer_e67e", "summary": "* __Problem Statement__: The paper proposes a defense against data-free model extraction attacks (where the attacker black-box queries a victim image classifier, s.t queries are then used to train a clone model)\n* __Approach__: The proposed defense \"DRO\" returns class probabilities as a result of the defender *perturbing* the input image. The objective of the perturbation is to flip the top-1 predicted class of attacker's queries (characterized by lying close to decision boundaries), but while preserving utility (measured by L1 distance to unperturbed probabilities).\n* __Experimental findings__: The approach is evaluated on 4x standard datasets, compared against 3x recent baseline approaches and in addition provides more studies (e.g., ablation). Results indicate that the proposed approach outperforms the baselines (in terms of both accuracy and utility).", "questions": "1. Hyperpameter choices\n    1. How were the defense hyperparameters determined? I'm concerned that if they are a result of hyperparam searches (e.g., over a sweep of attacks), it might not generalize to novel attacks.\n    2. Hyperparameter choices: Are the defense hyperparameters fixed for all choices of datasets, clone model architectures, and attacks? \n2. DRO optimization objective vs. Adversarial training: Can the authors clarify on how the min-max optimization objective (Eq. 4) is different from adversarial training? After all, in both cases, the objective is to find parameters $\\theta$ s.t predictions are robust to perturbed inputs. \n3. (Suggestion) Sec 5.4 Adaptive attacker: Please elaborate on how the attacker \"adapts\". It is unclear what the attacker does from L341.", "limitations": "Limitations are adequately discussed in Sec. 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. __Originality__: 4/5. The approach is original and well-motivated. Although some ideas have been explored before (i.e., perturbing outputs, changing gradient direction), the proposed approach explores a novel objective (learnt perturbation of inputs to introduce a targeted shift in prediction) and proposes a reasonable way to achieve it.\n2. __Quality__: 4/5. The paper is high-quality. The insights motivating the approach are sound and the experimental section is exhaustive.\n3. __Clarity__: 3/5. The approach is somewhat clear (certain technical choices/notation are unclear; more under questions).\n4. __Significance__: 3/5. The approach is somewhat significant -- it clearly improves upon existing defenses, however against a very specific class of model extract attacks. Specifically, \"data-free\" attacks that have shown to be successful largely in extreme settings (such as with attacker querying the victim millions of times).", "weaknesses": "1. __DRO optimization setup - unclear__: I found the intuition for the defense objective (Fig 2, L148-207) clear. However, it is unclear how the objective formulation (Eq. 4-6) achieves this. Based on Fig. 2 and L183-190, it appears that the requirement is to perturb images s.t the label is flipped (i.e., maximize CE loss wrt parameters $\\omega$), however the outer maximization in Eq. 6 seems to do the opposite? Furthermore, in Eq. 6, since the utility is upper-bounded $||\\cdots||_1 \\le \\tau$, it appears that the perturbation generator $h_\\omega$ can simply learn an identity function?\n2. __Evaluation curves missing__: The paper although extensively evaluates the defense, it appears that it is primarily for a specific defense effectiveness-utility operating point (specific value of $\\gamma$ and $Q$). This contrasts prior works which evaluates a curve of attacker's vs. defender's classification accuracies (e.g., Fig. 4 GRAD, Fig. 4 PredPoison). Without such a curve, it's somewhat unclear to determine the overall performance of the attack.\n3. __Perturbation budget $\\le$ 1.0__: The L_1 perturbation budget $\\tau$ in the defense chosen appears to be a fixed to 1 (L222, L298) seems unreasonably high. After all, this allows the defender to always flip the top-1 class-label. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your insightful comments.\n\n**Q1.1**: DRO optimization setup: Based on Fig. 2 and L183-190, it appears that the requirement is to perturb images s.t the label is flipped (i.e., maximize CE loss wrt parameters )? \n\n**A1.1**:  Eq 4-6 is to **maintain the model utility on benign inputs**.  Given that perturbations applied to benign inputs can diminish model utility by altering probability outputs, the rationale behind this Distributionally Robust Optimization (DRO) formulation is to ensure the robust generalization of the model on perturbed test data. Consequently, the objective of this optimization is to minimize the loss function across the simulated test set from the training data.\n\n**Q1.2**: DRO optimization setup: in Eq. 6, since the utility is upper-bounded, it appears that the perturbation generator can simply learn an identity function?\n\n**A1.2**:  Eq.6 is a constraint (not optimization objective) for the optimization problem in Eq. 4.  Learning an identity function is only  *a feasible solution of the optimization, but is not the optimal solution*. This is due to the interdependence between the parameters of the perturbation generator and parameters of the target model.\n\n**Q2**: evaluate a curve of attacker's vs. defender's classification accuracies.\n\n**A**:   Thanks for your question, please refer to **Figure 2 in rebuttal.pdf in the global response** for the evaluation curves.   \n\n**Q3**:  The L_1 perturbation budget appears to be fixed to 1 seems unreasonably high.\n\n**A**:  Thanks for pointing out this. \n**Set a perturbation budget equal to 1 is intended to create  stronger baseline defense methods for comparisons**. A larger perturbation budget can **strengthen the defense of the compared baseline methods** by allowing more flexibility to adjust the prediction logits. To further illustrate the impact of varying perturbation budgets on defense methods, we have included additional results with smaller perturbation budgets of 0.2 and 0.5 in the following table. The results show that the defense performance of compared baseline defense methods appears weaker when the defender's perturbation budget is constrained. In such cases, the defender can only perturb the output probabilities using a smaller budget, which poses a challenge to existing defense mechanisms.\n\n*Perturbation Budget of 0.2*\n\n|Attack| Defense|ResNet18-8X|MobileNetV2|DenseNet121|\n|--------|----------|------------|-------------|-------------|\n||Undefended|87.36%±0.78%|75.23%±1.53%|73.89%±1.29%|\n|DFME|RandP|86.32%±1.05%|74.67%±2.02%|72.16%±2.05%|\n||P-poison|83.17%±1.66%|73.32%±1.45%|71.68%±1.51%|\n||GRAD|84.26%±1.72%|71.82%±1.67%|71.23%±1.68%|\n||MeCo|**51.68%±1.96%**|**46.53%±2.09%**|**61.38%±2.41%**|\n\n*Perturbation Budget of 0.5*\n\n|Attack|Defense|ResNet18-8X|MobileNetV2|DenseNet121|\n|--------|----------|------------|-------------|-------------|\n||Undefended|87.36%±0.78%|75.23%±1.53%|73.89%±1.29%|\n|DFME|RandP|85.53%±1.65%|74.62%±2.37%|71.56%±2.79%|\n||P-poison|81.19%±1.86%|69.71%±1.56%|70.93%±1.58%|\n||GRAD|82.89%±1.82%|68.30%±1.75%|71.32%±1.73%|\n||MeCo|**51.68%±1.96%**|**46.53%±2.09%**|**61.38%±2.41%**|\n\n**Q4**: How were the defense hyperparameters determined? Are the defense hyperparameters fixed for all choices of datasets, clone model architectures, and attacks?\n\n**A**:  Thank you for your questions. In the data-free model extraction setting, where the attack query data distribution is unknown, we use a surrogate dataset to represent the attack query data and then choose the hyperparameters that achieve the worst performance on the validation dataset of target dataset. For example, we adopt CIFAR100 dataset as a surrogate query dataset to query the target victim model trained on CIFAR10. Our next step involves defensive training using our proposed method on CIFAR10 training dataset. To evaluate the effectiveness of our defense, we simulate the model stealing process by calculating the loss function of the attacker. This is done by comparing the outputs of the cloned model and the target model on CIFAR100 dataset.\nWe choose the hyperparameters that lead to the worst performance on the validation set of CIFAR10 with the extracted model. \n\nThese selected hyperparameters should be tuned for each target dataset, i.e., dataset-dependent. After tuning for each dataset,  then the hyperparameters remain fixed across clone model architectures, and various attacks.\n\n**Q5**: DRO optimization vs. Adversarial training\n\n**A**: Thanks for your question. We'd like to clarify the distinction between adversarial training and distributionally robust optimization (DRO). \nAdversarial training focuses on optimizing the model's performance against worst-case perturbations for **each individual training data point**. The goal is to enhance the model's robustness to small adversarial perturbations. However, this approach will lead to a drop in performance on the test data, as the model becomes *too conservative*.\n\nOn the other hand, DRO aims to optimize the model's performance by considering the worst-case performance over **a set of possible data distributions**. Instead of focusing solely on individual data points, DRO accounts for *uncertainties in the data and generalizes well to different distributions*. As a result, DRO tends to improve performance on the test data, striking a balance between standard empirical optimization and adversarial training.\n\n**Q6**:  Adaptive attacker: Please elaborate on how the attacker \"adapts\"\n\n**A**: The adaptive strategy involves introducing data-dependent perturbations to the clone model's input to simulate the defense mechanism. This approach aims to bridge the gap between the defense used during testing and the training of the clone model for attacker. By incorporating the defense mechanism into the clone model's input, attacker seeks to better adapt to defender's protection strategy. This allows attacker to improve the effectiveness of their attacks.\n\n\n"}, {"review_id": "ZIqbyQuRQN", "reviewer": "Reviewer_xYzi", "summary": "This paper presents a new method to defend against depth function preserving model extraction (DFME) attacks. The method proposed by the authors, called MeCo, adds data-dependent random perturbations to the input data, making it difficult for attackers to extract useful information from the black-box model. The authors claim that MeCo effectively reduces the accuracy of the cloned model while maintaining the utility of the target model. The approach was evaluated on several data sets and compared to existing defense strategies, showing excellent performance in terms of effectiveness, computational efficiency, and memory usage.", "questions": "Can you provide more details on the theoretical basis of the proposed approach?\nHow does the proposed method perform under different attack scenarios?", "limitations": "The authors acknowledge the lack of theoretical analysis as a limitation of the current work and take it as a direction for future work. They also mention the potential benefits of their proposed defense training for protecting large-scale pre-trained models used in public apis. However, a more detailed discussion of the possible negative social impacts of the proposed approach would be useful.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The quality of the work is evident in the extensive experiments conducted and the clear presentation of results. \n2.The main advantage of the paper is its novel approach to defend against DFME attacks. The proposed method, MeCo, is both effective and efficient, which is demonstrated by experimental results. However, these advantages may be affected by a lack of theoretical support.", "weaknesses": "1.The paper only conducted experiments in cifar10, cifar100 and mnist data sets, but the lack of relevant experiments on large data sets led to some doubts about the extensibility of the model.\n2.The model may need more testing and proof in generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your constructive feedback.\n\n**Q1**:  lack of relevant experiments on large data sets led to some doubts about the extensibility of the model. \n\n**A**: Thanks for your suggestions. As requested, we included the results obtained on the MiniImageNet [1] dataset with 100 classes, which is a subset of the ImageNet dataset. The results are presented in Table 8 of the Appendix in our submission.  For convenience, we also present the results on MiniImageNet  in the following table.\n\nIn addition, we took into consideration the dataset evaluated in the current state-of-the-art model extraction defense methods, as referenced in [2]. As a result, we extended our experiments to include the CUB200 dataset [3] with 200 image classes, which, as mentioned in [2], represents the largest dataset utilized for model extraction defense. The detailed outcomes of our experiments on the CUB200 dataset are provided in the following tables.  We hope that these additional results further contribute to the comprehensive evaluation of our proposed model and its defense against model extraction techniques.\n\n*Clone Model Accuracy with Different Defense Methods on **MiniImageNet**  with Different Clone Model Architecture*\n\n| Attack | Defense  | ResNet18-8X | MobileNetV2 | DenseNet121 |\n|--------|----------|------------|-------------|-------------|\n|        | Undefended | 35.89% ± 3.97% | 28.71% ± 3.25% | 25.05% ± 3.68% |\n| DFME   | RandP | 30.76% ± 4.09% | 22.06% ± 3.83% | 20.23% ± 3.97% |\n|  (Soft Label)   | P-poison | 29.36% ± 4.23% | 21.83% ± 3.77% | 20.01% ± 3.89% |\n| | GRAD | 29.87% ± 3.76% | 21.65% ± 3.75% | 19.82% ± 3.77% |\n|        | MeCo | **23.29% ± 3.83%** | **17.83% ± 3.67%** | **16.73% ± 3.88%** |\n|        |          |            |             |             |\n|        | Undefended | 46.72% ± 4.86% | 40.35% ± 4.97% | 38.71% ± 3.85% |\n| DFMS-HL | RandP | 45.09% ± 4.93% | 39.51% ± 4.83% | 38.08% ± 3.95% |\n| (Hard Label) | P-poison | 45.16% ± 5.03% | 39.06% ± 4.72% | 37.78% ± 4.26% |\n|        | GRAD | 45.32% ± 5.21% | 39.17% ± 4.85% | 37.85% ± 4.32% |\n|        | MeCo | **39.23% ± 4.83%** | **35.81% ± 4.69%** | **32.30% ± 4.56%** |\n\n*Clone Model Accuracy with Different Defense Methods on **CUB200**  with Different Clone Model Architecture*\n\n| Attack | Defense  | ResNet18-8X | MobileNetV2 | DenseNet121 |\n|--------|----------|------------|-------------|-------------|\n|        | Undefended | 49.75% ± 3.82% | 32.83% ± 3.53% | 29.08% ± 3.92% |\n| DFME   | RandP | 38.89% ± 4.16% | 26.85% ± 3.96% | 23.81% ± 4.05% |\n|        | P-poison | 36.45% ± 4.38% | 25.31% ± 3.91% | 22.09% ± 4.03% |\n| (Soft Label) | GRAD | 31.70% ± 3.61% | 24.78% ± 3.89% | 21.91% ± 3.86% |\n|        | MeCo | **20.78% ± 4.67%** | **18.65% ± 4.90%** | **15.86% ± 4.51%** |\n|        |          |            |             |             |\n|        | Undefended | 58.89% ± 5.03% | 49.35% ± 5.18% | 46.71% ± 4.37% |\n| DFMS-HL | RandP | 55.28% ± 5.39% | 46.67% ± 5.65% | 45.19% ± 4.32% |\n| (Hard Label) | P-poison | 52.71% ± 5.36% | 45.28% ± 5.31% | 43.76% ± 4.91% |\n|        | GRAD | 51.43% ± 5.73% | 46.51% ± 5.50% | 42.39% ± 4.82% |\n|        | MeCo | **43.31% ± 5.16%** | **38.27% ± 5.03%** | **33.42% ± 4.95%** |\n\nReference:\n\n[1] Matching Networks for One Shot Learning.  NeurIPS 2016.\n\n[2] How to Steer Your Adversary: Targeted and Efficient Model Stealing Defenses with Gradient Redirection. ICML 2022.\n\n[3] Caltech-UCSD Birds 200. 2010\n\n**Q2**: The model may need more testing and proof in generalization.\n\n**A**: Thank you for your suggestions. In response to your concerns, we made significant enhancements to our work based on your feedback. Specifically, we have included additional experimental results and in-depth theoretical explanations for the proposed methods.\n\nIn response to your first question **Q1**, we have incorporated supplementary experimental results that shed further light on the performance and generalization of our approach. These results can be found in **Q1**.\n\nFurthermore, we have provided a comprehensive theoretical basis for our proposed methods. Our aim was to elucidate the underlying principles and mechanisms that support the efficacy of our approach. These theoretical explanations are also elaborated upon in the global response. We believe that these additional elements substantially strengthen the quality and comprehensiveness of our work.\n\n**Q3**: Can you provide more details on the theoretical basis of the proposed approach? How does the proposed method perform under different attack scenarios?\n\n**A**:  Thank you for your comments, we provided a comprehensive theoretical basis for our proposed methods. Our aim was to elucidate the underlying principles and mechanisms that support the efficacy of our approach. These theoretical explanations are also elaborated upon **in the global response**.  Furthermore, we supplemented our theoretical explanations with visualization, which can be found in Figure 1 within the **'rebuttal.pdf' document as part of the global response**.\n\nFurthermore, our method can also be viewed as random smoothing: $T_{\\theta_T}^{S}(x) = \\int T_{\\theta_T}(x + h_{\\omega}(x, \\epsilon)) d\\epsilon$, where $\\epsilon \\sim N(0, I)$. Compared to learning a single function $T_{\\theta_T}(x)$, learning the smoothed function $T_{\\theta_T}^{S}(x)$ necessitates a larger number of queries for attacker due to the requirement of calculating the integral. By necessitating a larger query budget, our approach makes it considerably harder for attackers to successfully clone the model.\n\n**Q4**:  Negative Social Impact\n\n**A**:   Sharing models and insights is a key driver of progress in the AI research field, and if the community becomes more protective of their models, it could slow down advancements.\nStricter model extraction defenses could create economic barriers for startups that rely on reverse engineering or analyzing existing models as part of their business strategy.\n\n\n "}]

Author rebuttal:
# Global Response (Theoretical Analysis)
Given the random perturbation applied to the query input, the loss function for extracting the victim model becomes noisy. Attacker has to employ the following noisy loss function to extract the target model.

$\mathcal{L}_{C}^{\Delta}(\theta_C) := KL(T(x + h^{\omega}(x, \epsilon); \theta_T), C(x;\theta_C))$

where $T(x, \theta_T)$ is the target victim model with parameters $\theta_T$, $C(x;\theta_C)$ is the clone model with parameters $\theta_C$ and  $h_{\omega}(x, \epsilon)$ is the data-dependent perturbation. $KL$ denotes the KL divergence

The ground truth loss function without input perturbation (which remains inaccessible to the attackers due to their lack of knowledge regarding the amount of input perturbation) is illustrated below.

$\mathcal{L}_{C}(\theta_C) :=   KL(T(x; \theta_T), C(x;\theta_C))$

Ideally, attacker should optimize the following loss. 

$\mathcal{L_C}^{*} = \min_{\theta_C} \mathcal{L}_{C}(\theta_C)$

 Due to the noisy loss function, attacker loss gradient becomes noisy and biased, shown as the following:

 **Attacker Loss Gradient Modeling**: 

$G_t(\theta_C) = \nabla_{\theta_C} \mathcal{L_C}^{\Delta}(\theta_C) = \nabla_{\theta_C} \mathcal{L}_{C}(\theta_C) + B_t(\theta_C) + N_t(\theta_C)$

where $\nabla_{\theta_C} \mathcal{L}_{C}^{\Delta}(\theta_C)$ is the actual gradient adopted by the attacker

$\nabla_{\theta_C} \mathcal{L}_{C}(\theta_C)$ is the ground truth loss gradient without input perturbation, which is inaccessible to attacker

$B_t(\theta_C)$ is the gradient bias due to the noisy loss function introduced by perturbation generator $h_w$

$N_t(\theta_C)$ is the random variable introduced by the randomness in data samples. For analysis convenience, the expectation of the noise is assumed to be zero, i.e. $\mathbb{E} N_t(\theta_C) = 0$

where attacker updates the clone model as $\theta_C^{t+1} = \theta_C^t - \alpha G_t$; $\alpha$ is learning rate

**Assumption 1**: We assume Polyak- Łojasiewicz (PL)
condition on attacker loss function $\mathcal{L}_{C}(\theta_C)$

$|\nabla_{\theta_C} \mathcal{L_C}(\theta_C)|^2 \geq 2\omega (\mathcal{L_C}(\theta_C))$

**Assumption 2**
We assume the following smoothness condition for the loss function  $\mathcal{L}_{C}(\theta_C)$ by following [2]:

$\mathcal{L}{C}(\theta_C^2) \leq \mathcal{L}{C}(\theta_C^1) + \langle \nabla_{\theta_C} \mathcal{L}_{C}(\theta_C^1), \theta_C^2 - \theta_C^1 \rangle + \frac{A}{2} |\theta_C^2 - \theta_C^1|^2$

**Assumption 3** Assume bounded gradient bias and randomness 

$\mathbb{E}||N_t(\theta_C)||^2 \leq D ||\mathcal{L}_{C}(\theta_C) + B_t(\theta_C)||^2 + \rho^2$

$||B_t(\theta_C)||^2 \leq d||\nabla_{\theta_C} \mathcal{L}_{C}(\theta_C)||^2 + \tau^2$ ($0 \leq d<1$)

where $\omega, D, d, \rho, \tau$ are constants

**Theorem 1**. Assume the attacker loss $\mathcal{L}_{C}(\theta_C)$ function satisfy assumption 2 and 3. $\alpha \leq \frac{1}{(D+1)A}$. Then, the attacker loss function satisfies the following inequality:

$\mathbb{E}[\mathcal{L_C}(\theta_C^{t+1})] \leq \mathcal{L_C}(\theta_C^{t}) + \frac{\alpha}{2}(d-1)|\nabla_{\theta_C} \mathcal{L}_{C}(\theta_C)|^2 + \frac{\alpha}{2}\tau^2 + \frac{\alpha^2 A \rho^2}{2}$

**Proof**: By assumption 2 and 3. We set $\theta_C^{1} = \theta_C^{t}$ and $\theta_C^{2} = \theta_C^{t+1}$ with $\theta_C^{t+1} = \theta_C^t - \alpha G_t$

$\mathbb{E}[\mathcal{L_C}(\theta_C^{t+1})] \leq \mathcal{L_C}(\theta_C^{t}) -\alpha \langle \nabla_{\theta_C} \mathcal{L}_{C}(\theta_C), \mathbb{E}(G_t) \rangle + \frac{\alpha^2A}{2} \left(\mathbb{E}|G_t - \mathbb{E}G_t|^2 + \mathbb{E}|\mathbb{E}G_t|^2\right)$

$= \mathcal{L_C}(\theta_C^{t}) -\alpha \langle \nabla{\theta_C} \mathcal{L_C}(\theta_C), \mathbb{E}(G_t) \rangle + \frac{\alpha^2A}{2} \left(\mathbb{E}|N_t|^2 + \mathbb{E}| \nabla_{\theta_C} \mathcal{L_C}(\theta_C) + B_t|^2\right)$

$\leq \mathcal{L_C}(\theta_C^{t}) -\alpha \langle \nabla{\theta_C} \mathcal{L_C}(\theta_C), \mathbb{E}(G_t) \rangle + \frac{\alpha^2A}{2} \left((D+1)\mathbb{E}|  \nabla_{\theta_C} \mathcal{L_C}(\theta_C) + B_t|^2 + \rho^2\right)$
 
$\mathbb{E}[\mathcal{L_C}(\theta_C^{t+1})] \leq \mathcal{L_C}(\theta_C^{t}) + \frac{\alpha}{2} \left(-2 \langle \nabla_{\theta_C} \mathcal{L_C}(\theta_C), \mathbb{E}(G_t) \rangle + |\nabla{\theta_C}\mathcal{L}_{C}(\theta_C) + B_t|^2 \right) + \frac{A\alpha^2\rho^2}{2}$

$= \mathcal{L_C}(\theta_C^{t}) + \frac{\alpha}{2} (-|\nabla{\theta_C} \mathcal{L}_{C}(\theta_C)|^2 + |B_t|^2) + \frac{A\alpha^2\rho^2}{2}$

$\leq \mathcal{L_C}(\theta_C^{t}) + \frac{\alpha}{2}(d-1) |\nabla{\theta_C} \mathcal{L}_{C}(\theta_C)|^2 + \frac{A\alpha^2\rho^2}{2}$

**Theorem 2** With assumption 3, convergence error of attacker loss function can be estimated as the following:

$L_C^{T}  \leq (1 - \alpha \omega(1-d))^T L_C^{0} + \frac{\tau^2 + A\alpha \rho^2}{\omega(1-d)}$

**Proof**:  We define $L_C^t = \mathcal{L_C}(\theta_C^t) - \mathcal{L}_{C}^{*}$. Then, we apply assumption 1 to theorem 1 and got following:

$L_C^{t+1} = (1 - \alpha \omega(1-d)) L_C^{t} + \frac{\alpha}{2}\tau^2 + \frac{\alpha^2A\rho^2}{2}$

We set $\kappa = \frac{\tau^2 + A\alpha \rho^2}{\omega(1-d)}$.
 Above equation can be rearranged as following :

$L_C^{T} - \kappa \leq (1 - \alpha \omega(1-d))^T(L_C^{0} - \kappa)$

Therefore, $L_C^{T}  \leq (1 - \alpha \omega(1-d))^T L_C^{0} + \frac{\tau^2 + A\alpha \rho^2}{\omega(1-d)}$

**Remark** From the above analysis, it becomes evident that accumulation of gradient estimation errors leads to a deviation of the final estimation error $L_C^{T} := \mathcal{L_C}(\theta_C^T) - \mathcal{L_C}^{*}$ from the ground truth. 
The first term in the above inequality $(1 - \alpha \omega(1-d))^T L_C^{0} \rightarrow 0$.
This deviation occurs due to the increase in $\frac{\tau^2 + A\alpha \rho^2}{\omega(1-d)}$ caused by higher gradient bias $\tau$ and gradient randomness $\rho$.  Consequently, extracted model by attacker has a larger optimization error.


























Meta-review: The paper points out three problems with data-free model extraction defense methods, namely inefficiency, strong assumptions, and untimely. In order to solve the above problems, the author proposes a defense method called MeCo, which can prevent the above model extraction attacks by achieving efficient distributed robust training while maintaining the practicality of the model. In a rebuttal, the author's response provides a detailed theoretical analysis as well as a hyperparameter analysis, convincing the reviewers to update their assessment. Due to providing more insights and theoretical explanations in the rebuttal, the AC suggested accepting this paper. The AC had a thorough discussion with the SAC and the SAC agreed on the decision. Detailed theoretical explanations about the paper and related experiments in different attack scenarios need to be further supplemented. Authors are encouraged to consider the reviewers' detailed comments and clearly present the guiding theory in the final version.
Decision: Accept (poster)