Title: Defending against Data-Free Model Extraction by  Distributionally Robust Defensive Training
Venue: NeurIPS 2023
Accepted: yes
Keywords: Data-Free Model Extraction; Defense
Abstract: Data-Free Model Extraction (DFME) aims to clone a black-box model without knowing its original training data distribution, making it much easier for attackers to steal commercial models. Defense against DFME faces several challenges: (i) effectiveness; (ii) efficiency; (iii) no prior on the attacker's query data distribution and strategy. However, existing defense methods: (1) are highly computation and memory inefficient; or (2) need strong assumptions about attack data distribution; or (3) can only delay the attack or prove a model theft after the model stealing has happened. In this work, we propose a Memory and Computation efficient defense approach, named MeCo, to prevent DFME from happening while maintaining the model utility simultaneously by distributionally robust defensive training on the target victim model. Specifically, we randomize the input so that it: (1) causes a mismatch of the knowledge distillation loss for attackers; (2) disturbs the zeroth-order gradient estimation; (3) changes the label prediction for the attack query data. Therefore, the attacker can only extract misleading information from the black-box model. Extensive experiments on defending against both decision-based and score-based DFME demonstrate that MeCo can significantly reduce the effectiveness of existing DFME methods and substantially improve running efficiency.

Defending against Data-Free Model Extraction by Distributionally Robust Defensive Training

Abstract

Data-Free Model Extraction (DFME) aims to clone a black-box model without knowing its original training data distribution, making it much easier for attackers to steal commercial models. Defense against DFME faces several challenges: (i) effectiveness; (ii) efficiency; (iii) no prior on the attacker's query data distribution and strategy. However, existing defense methods: (1) are highly computation and memory inefficient; or (2) need strong assumptions about attack data distribution; or (3) can only delay the attack or prove a model theft after the model stealing has happened. In this work, we propose a Memory and Computation efficient defense approach, named MeCo, to prevent DFME from happening while maintaining the model utility simultaneously by distributionally robust defensive training on the target victim model. Specifically, we randomize the input so that it: (1) causes a mismatch of the knowledge distillation loss for attackers; (2) disturbs the zerothorder gradient estimation; (3) changes the label prediction for the attack query data. Therefore, the attacker can only extract misleading information from the black-box model. Extensive experiments on defending against both decision-based and scorebased DFME demonstrate that MeCo can significantly reduce the effectiveness of existing DFME methods and substantially improve running efficiency.

1 Introduction

Model extraction attack aims to replicate the functionality of a public API with only query access. Most model extraction methods focus on data-based model extraction , i.e., an attacker can access a small subset of the in-distribution training data [41], or a relevant surrogate dataset [36] of the target model. Beyond data-based model extraction (DBME), recent promising results with data-free model extraction (DFME) 2 show that the attacker can clone a model with performance close to that of the target black-box model even without prior knowledge of the distribution of the proprietary training data. Those DFME techniques make it much easier for attackers to steal the model without collecting and annotating relevant training data. DFME can be further categorized into score-based DFME [21, 51], where the target model provides softmax probability outputs to the users; and decision-based DFME [54, 47], where the target model only provides the top-1 label. Thus, the model owners

face a more critical problem - how to prevent those black-box pre-trained models from stealing in the data-free setting while simultaneously guaranteeing a high-quality inference service? The main challenges of defending against DFME are: (1) effectiveness requirement: the defender should substantially reduce the clone model accuracy; (2) efficiency requirement: the defense procedure should be memory and computation efficient; (3) lack of knowledge on the attackers: the attack query data distribution and attack strategy are unknown to the defender.

Existing work on Model Extraction (ME) defense primarily focuses on data-based ME attacks. We categorize the existing ME defense methods into pre-attack, delay-attack, and post-attack defenses. Pre-attack defenses aim to prevent ME attack from happening. They either perturb the output probabilities [38, 33] or integrate multiple models [22] or detect attack queries from benign queries [23]. Delay-attack defenses aim to delay the ME attack instead of preventing it [10]. Post-attack defenses neither prevent nor delay the ME; instead, they aim to prove a model theft [17, 18, 32]. Our work falls under the pre-attack defense category since our goal is to prevent ME from happening. However, existing pre-attack defense methods have limitations when applied to DFME: (1) output perturbation-based methods perform optimization during deployment is computationally and memory expensive; (2) ensemble-based methods are memory inefficient since they need to store multiple models; (3) detection-based methods require strong assumptions about the query data distribution; the attacker can easily make the attack queries indistinguishable from benign ones [59] and circumvent the detection [28], rendering the defense ineffective; (4) some defense methods [33, 23, 22] require the knowledge of the attack query data. However, this prior knowledge is unknown to the defender.

To address existing defense methods' limitations, we propose a new and orthogonal class of defense method, named Memory and Computation efficient defense (MeCo), through a randomized defense strategy. Specifically, MeCo adds data-dependent random perturbation to the query input. MeCo can effectively defend against DFME for several reasons. For score-based DFME: (i) it leads to a mismatch of the knowledge distillation loss for attackers; (ii) existing DFME methods heavily rely on zeroth-order gradient estimation. MeCo can disturb the zeroth-order gradient estimation so that the attacker can only obtain its inaccurate estimation. For decision-based DFME, MeCo can change the label prediction of attack query data. The attacker can only learn from incorrectly labeled data.

MeCo would reduce the target model utility on benign queries without additional mechanisms. Maintaining the target model utility needs to : (i) minimally sacrifice the classification accuracy on the test set, and (ii) minimize the perturbation magnitude of the output class probability. To maintain the target model utility on benign queries, we propose a principled distributionally robust optimization (DRO) framework to train the target model and perturbation generator. Our DRO framework simulates the worst-case in-distribution (ID) test data from the training data (accessible for the defender). Then, we apply the random perturbation to the simulated test data and train the target model and perturbation generator on the simulated test data to ensure the worst-case generalization on the ID test data. Compared to existing works, MeCo has numerous advantages, including (1) MeCo is substantially more computation and memory efficient without complex optimization and storing multiple models during deployment; (2) it avoids detecting attack queries from benign ones; (3) does not need the knowledge of the attack query data distribution. More importantly, even if attackers know our defense strategy and adopt an adaptive attack strategy, MeCo is still effective since stealing a random function further increases the difficulty of ME thanks to the randomness introduced in MeCo. Extensive experiments compared to various defense methods show that MeCo significantly reduces the accuracy of the clone model across different query budgets.

In summary, our main contributions are three-fold:

· Wepropose a novel principled defensive training framework that substantially improves the memory and computation efficiency during deployment to defend against DFME attacks.

· We propose a distributionally robust optimization (DRO) method to randomly perturb the inputs to defend against DFME effectively while maintaining the model utility simultaneously.

· Extensive experiments on defending against both score-based and decision-based DFME show the effectiveness of MeCo, reducing the clone model accuracy by up to 35% , while maintaining the target model utility. Further, MeCo can also effectively defend against data-based ME and boost its performance. More importantly, MeCo achieves substantially more computation and memory efficiency than existing methods, e.g., 17 × ∼ 172 × speed up.

2 Related Work

Data-Free Model Extraction (DFME). Model extraction (ME) attack aims to extract and clone the functionality of the public API with only query access; representative works include [31, 9, 50, 53, 35, 36, 7, 63, 16, 40, 4, 60, 27]. Recently, ME has been extended to the data-free setting, named DFME, which cannot access original training data. ZSDB3KD [54] and DFMS-HL [47] focus on the decision-based setting, i.e., only the hard label (top-1 label) is predicted for the attacker. MAZE and DFME [21, 51] are score-based methods, i.e., soft label (softmax output) is delivered to the attacker.

Model Extraction Defense. We categorize the model extraction defense methods into three categories: pre-attack defenses, delay-attack defenses, and post-attack defenses, according to when the defense happens during an attack. (i) Pre-attack defenses aim to prevent the ME attack from happening. There are three classes of methods: (1) output probabilities perturbation-based methods [26, 38, 33]: Prediction poisoning (P-poison) [38] and GRAD [33] perform a complex optimization during deployment; (2) ensemble-based methods: EDM [22] integrates multiple models for diverse predictions. (3) detection-based methods: Adaptive misinformation [23], PRADA [19] and VarDetect [39] detect the attack queries from the benign queries. However, those methods have limitations when applied to DFME: (1) they [38, 33] significantly increase the computation and memory cost; (2) they [23, 19, 39] have a high risk of incorrectly classifying attack or benign queries. The attacker can easily evade the detection [28, 11], making the defense ineffective. (3) GRAD [33], Adaptive misinformation [23], and EDM [22] need to know the prior knowledge and distribution of attack query distribution, which is unknown to defenders in the data-free setting. Our method falls under this pre-attack defense category and addresses their limitations from a novel and orthogonal perspective. (ii) Delay-attack defenses aim to delay the ME attack instead of preventing the attack. Proof of work (PoW) [10] delays model stealing by significantly increasing the computation cost of query access for model extraction, e.g., solving a puzzle. Ours is fundamentally different from PoW in two aspects: (1) PoW could not prevent model stealing if the users spend more time, computation cost, money, etc. By contrast, our method is to prevent model stealing instead of delaying model stealing; (2) PoW needs multiple teachers to evaluate the privacy leakage of queries [42]. Our method only requires a single teacher; thus, ours is substantially more memory and computation efficient than PoW. (iii) Post-attack defenses aim to prove a model theft after a pre-trained model has been stolen, e.g., through watermark-based [1, 12] methods [17, 49], proof-of-learning [18] and dataset inference [32]. However, the post-attack defenses only perform verification of model theft but cannot prevent the model from being stolen. This requires a model owner to obtain access to the stolen model. If the stolen model is not used as API, the defender cannot verify whether the model has been stolen.

Distributionally Robust Optimization (DRO) DRO is a flexible and powerful optimization framework to make decisions under uncertainty [43, 55], where robustness is an important factor [62, 56]. DRO constructs a set of probability distributions, known as an ambiguity set, and then minimizes the worst-case performance within the ambiguity set, thus guaranteeing the model performance. There have been various machine learning applications of DRO, such as dealing with group-shift [45], subpopulation shift [61], and long-tailed learning [57]. To the best of our knowledge, our work is the first principled method with DRO for DFME defense.

3 Problem Setup and Preliminaries

3.1 Problem Setup

Data-Free Model Extraction (Attacker). In DFME, the attacker sends a query input x to the target victim model T parameterized with θ T and receives a prediction P ( y | x ) = T ( x ; θ T ) . For the score-based setting [21, 51], the target model delivers the soft label (output class probabilities) to the attacker. In the decision-based setting [54], the target model only delivers the hard label (the top-1 class) prediction to the attacker. Following [54, 21, 51], we assume the following attacker knowledge: (1) data-free : the attacker cannot access the original training data distribution of the target victim model. The attacker typically employs synthetic out-of-distribution (OOD) data to query the target model in an attempt to steal it. (2) black-box : the attacker does not know the architecture and model parameters of the target model. Given a test dataset D test associated with the black-box model, the attacker aims to train a compact clone model C with parameters θ C that maximize the testing accuracy.

Defense against DFME (Defender). Following [38, 22], we assume the defender does not know: (1) whether a query is malicious or benign; (2) the attack strategies adopted by an attacker; (3) the model architecture used by an attacker. The goals of the defender are three-fold: (1) effective: minimize the test accuracy that the attacker can achieve; (2) utility preserving: maintain the accuracy on benign inputs and minimize the perturbation magnitude of the output probability; (3) efficient: the defense procedure should be memory and computation efficient.

3.2 Preliminaries

Knowledge Distillation (KD). Existing DFME methods build on top of KD [5, 14, 44, 58, 24]. Assume we have a pre-trained teacher (target) model T with parameters θ T , and a student (clone) model C with parameters θ C . Suppose the output probabilities of teacher and student for the input x are T ( x ; θ T ) and C ( x ; θ C ) , respectively. The training objective of KD is as the following:

where L c ( x , y ) is the cross-entropy loss, KL is the KL divergence between two probability distributions, and α is the weighting factor. In this work, we focus on defending against score-based and decision-based DFME methods. Due to space limitations, we give a brief description of score-based DFME while placing the details of decision-based DFME in Appendix 8.

Score-based DFME. The representative score-based DFME works are [51, 21]. We briefly describe how they work. The attacker has a pseudo data generator G parameterized by θ G with random vector z as input. It generates pseudo data by x = G ( z ; θ G ) , where z ∼ N (0 , I ) . Then, the attacker sends the query x to the target and clone model; they output class probabilities according to the generated pseudo data, i.e., y T = T ( x ; θ T ) and y C = C ( x ; θ C ) . The attacker jointly optimizes the clone model C and generator G as below:

Since the target network is black-box, to backpropagate into the generator network weights, they apply zeroth-order gradient [30] on the outputs of the generator, i.e.,

where µ i is a random direction; m is the number of queries; δ is a small step size. The generator G and the clone model C alternatively update their parameters by minimizing the loss L C and L G . The attacker can obtain a clone model C ( x ; θ C ) after a certain number of training iterations.

4 Methodology

To defend against DFME, we propose a DRO defensive training strategy. We present the defense method in Section 4.1. We then discuss the defensive training details and deployment algorithm in Section 4.2. We illustrate how the proposed defense can defend against DFME in Section 4.3.

4.1 Distributionally Robust Defensive Training

The core techniques of the score-based DFME are two-fold: (1) using KD loss to match the target and clone model outputs; (2) zeroth-order gradient estimation. On the other hand, decision-based DFME mainly relies on the label prediction of the target model. We propose a random perturbation technique to make the attacker estimate misleading information. However, adding random noise to the query input strongly restricts the flexibility of random perturbation since it is difficult to maintain the model utility while effectively defending against DFME simultaneously. Furthermore, the added randomness to the input should be input-dependent since different inputs have different sensitivities to the neural network decision boundary. We thus propose a flexible random perturbation generator to learn a data-dependent random perturbation that can adaptively generate different perturbations for different inputs while maintaining the performance on benign input. First, we denote the data-dependent random perturbation generator as h ω ( x , ϵ ) parameterized by parameters ω with query x and random noise ϵ as input. The perturbation generator is a two-block ResNet with filter 64. It only accounts for a tiny proportion of the target model. We explain our intuition in Figure 1. Our method builds

Perturbation

generator

Query

z

∼

N

(0,

I

)

Perturbed query

(a) Data-dependent perturbation

Positive query

Attack query

Perturbation

Decision boundary

Negative query

Perturbed data

(b) Shift of data points

Benign query:

Attack query:

w/o perturbation

w/ perturbation

(c) Change of class probability with

our defense method

(a)

(b)

(c)

o

Attack query

Attack query

Perturbed Attack query

Attackquery

PerturbedAttackquery

△

Benign query

△

Benignquery

PerturbedBenignquery

△

Benign query

PerturbedBenignquery

Benignquery data

DROguarantees

modelutility

on the intuition [54] that the attack query is closer to the decision boundary, while the benign query is farther away from the decision boundary in the DFME setting. We add random perturbation to the query input so that the attack query is closer to or crosses the decision boundary. Their output probabilities from the target model will be perturbed more significantly in the score-based setting. Their labels are more likely to be flipped in the decision-based setting. In contrast, benign queries are far from the decision boundary, thus not influencing benign queries much.

Simply adding the proposed data-dependent random perturbation to the query inputs would reduce the model utility. We propose a Distributionally Robust Optimization (DRO) framework to train the target model and perturbation generator to maintain the target model utility, i.e., (i) minimally sacrifice the classification accuracy on the test set, and (ii) minimize the perturbation magnitude of the output class probabilities. In addition, as depicted in Figure 2, we conducted a comparative analysis of three scenarios: one without random input perturbation, another with random input perturbation, and the third involving DRO defensive training. Our DRO defensive training exhibits a dual capability in effectively defending against DFME while maintaining the model utility.

Training data

Simulated testing data

DRO

0.5

0.4

0.3

0.2

0.1

0'0

simulated test data distribution 1

simulated test data distribution 2

training data distribution

0.5

0.4

0.3

0.2

0.1

0.0

training data distribution

Since the test data distribution is unknown during deployment, we propose a flexible framework with DRO to optimize under uncertainty [43]. DRO constructs an ambiguity set of probability distributions and optimizes the worst-case performance within the ambiguity set, thus guaranteeing

the performance. Our framework takes that the underlying probability distribution of test data µ is unknown and lies in an ambiguity set of probability distributions around the training data distribution µ 0 . The proposed DRO framework perturbs the training data distribution to simulate the test data distribution, illustrated in Figure 3. It optimizes the worst-case performance in the ambiguity set to guarantee the model utility after adding random perturbation. We formulate the proposed DRO framework in the probability measure (distributions or densities) space :

where the inner sup optimization in Eq. (4) is to compute and simulate the worst-case test data distribution (denoted as π ) around the training data distribution (denoted as µ 0 ). π is defined as the probability distribution that achieves sup µ ∈P E x ∼ µ L ( θ T , x + h ω ( x , ϵ ) , y ) . P in Eq. (5) denotes the ambiguity set of probability measures for the test data distribution to characterize its uncertainty. One common choice to define P is through KL divergence. KL ( µ || µ 0 ) denotes the KL divergence between probability measure µ 0 and µ . β is a constant to characterize the closeness between µ 0 and µ to ensure the worst-case test data distribution π does not deviate from the raw training data distribution µ 0 much. Eq. (6) ensures that model utility (output class probabilities, i.e., T ( x ) ) does not change much after adding random perturbation, i.e., T ( x + h ω ( x , ϵ )) . The probability perturbation magnitude is measured by l 1 norm, i.e., || · || 1 . It is crucial to guarantee target model output class probabilities after query perturbation close to that of without query perturbation since benign users need these informative probabilities to derive helpful knowledge [33]. τ is a constant.

To solve the above optimization problem, we convert Eq. (4-6) into the following unconstrained optimization problem by Lagrange duality [6] (detailed derivations are put in Appendix 11.1):

The constant γ controls the regularization magnitude for model utility. The KL-divergence KL ( µ || µ 0 ) is handled by Wasserstein gradient flow (WGF) (presented in the following sections), therefore for simplicity, the regularization weight for it is set to 1.0 throughout the paper. We name Eq. (7) as Defense DRO . The optimization in Eq. (7) is still challenging to solve as the inner sup optimization is over probability measure space, which is an infinite-dimensional function space .

Solution to the Defense DRO. To make the solution of the Defense DRO (Eq. (7)) tractable, we reformulate it from a new continuous dynamics perspective. To achieve this goal, we first define the energy functional F ( µ ) as follows:

where V ( µ ) = -E x ∼ µ L ( θ T , x + h ω ( x , ϵ ) , y ) . By defining such energy functional F ( µ ) , the Eq. (7) can be equivalently formulated by the following gradient flow system Eq. (9, 10):

where Eq. (9) solves the inner sup problem in Eq. (7) with WGF in Wasserstein space (Details provided in Appendix 11.2) and Eq. (10) solves the outer minimization problem in Eq. (7) for parameter update with gradient flow in Euclidean space. Below, we propose a method for efficiently solving the Eq. (9). We view each training data as one particle and arrive at the following test data distribution simulation solution (solution to Eq. (9)) (Details are provided in Appendix 11.2):

i th i where x t denotes the i training data x perturbed at time t ; α is the data transformation rate. U ( x , θ T ) = -L ( θ T , x ,y ) . k ( x i , x j ) is the Gaussian kernel, i.e., k ( x i , x j ) = exp( -( x i -x j ) 2 2 σ 2 ) . The first term in the RHS of Eq. (11) moves the training data towards the worst-case simulated test data distribution by raising the energy functional. The data distribution change is controlled by the sum of gradients from the mini-batch of the training data, which are weighted by the kernel function k ( x i t , x j t ) , smoothing the gradients of training data. The second term (kernel gradient) acts as a repelling force that keeps the transformed data from concentrating on a single data point, thus diversifying the simulated test data.

4.2 End-to-end Defensive Training for DFME

We summarize the defensive training algorithm in Algorithm 1. Line 3-4 simulates the worst-case test data distribution. Line 5-8 adds data-dependent random perturbation to the simulated test data and trains the target model and perturbation generator on the simulated test data.

MeCo Deployment. During testing, we perform similar perturbation for each query. Given any query input x , y p = T θ T ( x + h ω ( x , ϵ )) ; ϵ ∼ N (0 , I ) . Then, return the results y p to the user.

4.3 Why Can MeCo Defend against DFME

Below, we give explanations on why Algorithm 1 could intrinsically defend against both score-based and decision-based DFME from the perspective: (1) mismatch the KD loss for the attacker; (2) disturb the zeroth-order gradient estimation; (3) change the label prediction on attack query.

Mismatch the KD learning objective. The KL-divergence loss Eq. (1) is critical to match the probability output between the target and clone model. By adding data-dependent random perturbation, we can encourage the mismatch of the KD loss between the target model and clone model, i.e.,

The mismatch would mislead the attacker to learn in the wrong direction since they think that T ( x + h ω ( x , ϵ ); θ T ) is the desired output for x but should be T ( x , θ T ) , as illustrated in Figure 1.

Disturb the zeroth-order gradient estimation. By adding data-dependent random perturbation, the zeroth-order gradient estimation of score-based DFME ( ∂ L G ∂ x in Eq. (3)) becomes:

where v 1 i and v 2 i are two data-dependent perturbations correspond to the two model inputs x + δ µ i and x , respectively. Compared with Eq. (3), Eq. (13) would make inaccurate gradient estimation by changing the gradient estimation direction due to the inconsistent random perturbation added to different inputs. Consequently, this would lead the gradient of the pseudo data generator in the wrong direction, i.e., ∇ θ G L G in Eq. (3) would be inaccurate. Then, the generator would not generate informative samples. The uninformative pseudo samples would cause the clone model Eq. (2) to learn incorrect information.

Therefore, by jointly (i) mismatching the KD loss for the attacker; (ii) disturbing the zeroth-order gradient estimation, MeCo can effectively defend against DFME. In summary, MeCo has many advantages compared to existing methods since it does not need to : (1) solve a complex optimization during testing; (2) store multiple models for ensemble; (3) detect the attack query from the benign query; (4) know the attack query data distribution. Due to the space limitations, we additionally show why MeCo can defend against decision-based DFME methods by changing the label prediction on attack query in Appendix 8.

Theoretical Analysis In this section, we delve into the theoretical analysis of our proposed method. We assess the effectiveness of our approach by analyzing it from the perspective of gradient-bias injection [2]. As depicted in Figure 4, our method progressively introduces perturbations into the gradients calculated by the attacker. This process increases the optimization error that accumulates during the model-stealing process, rendering it more challenging for the attacker to successfully

(a)

(b)

3

-2

L -

0

2

Z-

0

1

extract the target model. For a more comprehensive understanding, we have provided detailed theoretical analysis in the Appendix due to space constraints.

5 Experiments

5.1 Experimental Setup

Datasets. We perform experiments on four standard datasets used in DFME literature, including MNIST, CIFAR10, CIFAR100 [25] and MiniImageNet [52] (100 classes).

Baselines. We compare SOTA DFME and defense baselines. Attack Baselines : (1) decision-based DFME methods: DFMS-HL [47]. We do not compare to ZSDB3KD [51] since it requires a much larger number of queries and is very slow. (2) score-based DFME methods: MAZE [21] and DFME [51]. Defense Baselines : we compare to: (1) Undefended : the target model without using any defense strategy; (2) Random Perturb ( RandP ) [38]: randomly perturb the output probabilities. (3) P-poison [38]; (4) GRAD [33]: gradient redirection defense. We set a large l 1 perturbation budget equal to 1.0 for those defense baselines in the experiments

to generate strong defense. That is, || y -ˆ y || 1 ≤ 1 . 0 ; where y and ˆ y are the unmodified/modified output probabilities, respectively. We put more baseline details in Appendix 10.

Implementation Details. The random perturbation generator is a two-block ResNet structure with a filter size of 64, which is a small network compared to the backbone (only accounts for a tiny proportion of the backbone). For decision-based DFME methods, following [47], we use a query budget of 10 M for CIFAR100 and 8 M for CIFAR-10. For score-based DFME methods, following [51], we set the number of queries to be 2 M for MNIST, 20 M for CIFAR10, and 200 M for CIFAR100, respectively. We perform each experiment for 5 runs with a mean and standard deviation of results. We provide more implementation details in Appendix 7.

5.2 Results of Defense against DFME

Clone model accuracy. For score-based DFME setting (soft label), we show the results on CIFAR10 and CIFAR100 in Table 1. For decision-based DFME setting (hard label), we show the results in

Table 2. Due to the space limitations, we present the results on MiniImageNet and MNIST in Table 8 and 7 in Appendix. ↓ indicates the accuracy the lower, the better; ↑ indicates the accuracy the higher, the better. For CIFAR10/CIFAR100 in Table 1, we use ResNet34 [13] as the target model. The clone model architecture includes, ResNet-18, MobileNetV2 [46], DenseNet121 [15]. We further change the target model architecture as GoogLeNet [48] for CIFAR10 in Table 9 in Appendix.

The results show MeCo significantly reduces the effectiveness of existing DFME methods by up to 35% and is substantially more effective than the compared methods since (1) RandP lacks datadependent information; it maintains the utility for almost all the query data, which is unnecessary for attack queries since the attacker can still learn useful information. (2) P-poison needs a random initialized surrogate attacker model, which acts as an adversary model. (3) GRAD needs to know the attack query set to train the surrogate model. Those surrogates have large gaps compared to the DFME attacker model. Those methods thus perform poorly since the attack query data distribution and model are unknown to the defender. While MeCo does not need a surrogate model.

Target model utility. We evaluate the target model utility by (1) target model test accuracy after adopting the defense strategy; (2) l 1 norm between the target model output probabilities with and without input perturbation averaged on the test dataset, i.e., E x ∼D test || T ( x + h ω ( x , ϵ )) -T ( x ) || 1 . The results (see Table 3 (MNIST with LeNet, CIFAR10 and CIFAR100 with ResNet34-8x)) indicate that MeCo maintains the target model utility with a slight trade-off of benign accuracy but with much better preservation of output probabilities in terms of l 1 norm. The baselines have large perturbations because the baseline defense methods perturb all the query data with the same magnitude ( l 1 perturbation budget of 1.0); this is unnecessary since the in-distribution benign query does not need such significant perturbation. In contrast, MeCo explicitly optimizes the l 1 norm of perturbation with DRO on the simulated test data distribution so that the perturbation magnitude is much smaller.

5.3 Application on Data-Based ME (DBME) Attack

We apply MeCo to defend against traditional DBME methods with Knockoff Nets [37] and JacobianBased Dataset Augmentation (JBDA) [41]. We present the results in Appendix 9.1. MeCo can significantly outperform existing methods and further reduce the clone model accuracy. When the distribution of attack query data is closer to the training data distribution of the target model, the effectiveness of MeCo defense weakens. This is due to the fact that MeCo applies smaller perturbations to the data distribution similar to the training data of the target model as a result of defensive training.

5.4 Adaptive Attacks

We further analyze the robustness of MeCo against the attacker's countermeasures. Namely, we consider the situations where attackers know about our defense and have tailored the attacks to our defense method by adding data-dependent random perturbation to the query inputs to learn an additional random perturbation generator. In Table 4, MeCo is still effective since the random perturbation added by the defender and attacker are different due to randomness. There is still a mismatch in the KD loss and zeroth-order gradient estimation for the attacker. Interestingly, the performance of the clone model becomes even worse after the attacker performs an adaptive attack.

We believe this is because after the attacker adds the random perturbation, they will need to learn a random function, increasing the difficulty of model extraction. In addition, for score-based DFME, following [38, 33], we also compare to the adaptive attack method where attackers use only hard label, not probability outputs. We show the results in Table 5. MeCo is still very effective since the pseudo samples are nearer to the decision boundary, and perturbing the input would easily change the model outputs on those samples. Consequently, the attacker still cannot learn useful information.

5.5 Ablation Study

Effect of DRO. We evaluate how much improvement DRO can bring to the model utility preservation in Table 12 in Appendix. We can observe that with DRO, our method significantly improves the model utility by 5 . 3% -5 . 6% compared to the one without DRO training.

Effect of different query budgets for attacker. To evaluate the effect of different query budgets on defense performance, we evaluate the clone model accuracy with varying defense methods on CIFAR10 in Figure 5 in Appendix. MeCo substantially outperforms various baselines with varying query budgets.

Hyperparameter Sensitivity. We evaluate the hyperparameter sensitivity for γ and Q in Table 10 and Table 11 in Appendix. We observe that model utility increases as γ increases with a trade-off of a decrease in defense performance. Results show that Q = 2 performs the best. With the increase of Q , the simulated test data may be harder to learn; thus, the benign accuracy slightly decreases.

Test time speed and memory comparisons. We compare the running time of best-performing methods in Table 13 in Appendix. MeCo achieves 17 ∼ 172 × speed up on CIFAR10 and CIFAR100. This is because P-poison and GRAD solve computationally expensive optimization during testing. In contrast, MeCo does not need this optimization. We provide the results in Table 14 in Appendix for memory consumption evaluation. MeCo is very memory efficient compared to baselines.

Training efficiency. We compare our training efficiency to baselines in Table 15 in Appendix. MeCo increases the training cost by 1 . 3 × . However, MeCo substantially improves the test time running efficiency. We believe this slightly additional computation cost is worth it.

6 Conclusion

In this paper, we explore the defense against DFME. We propose a memory and computation efficient (MeCo) defense method through distributionally robust defensive training by adding a data-dependent random perturbation generator to perturb the input data so that the attacker cannot steal useful information from the black-box model. At the same time, MeCo maintains the target model utility. Extensive experiments demonstrate MeCo's effectiveness, computation, and memory efficiency.

Limitations and Broader Impacts . Our proposed defensive training helps build safe and trustworthy AI systems. This would be beneficial for protecting current large-scale pre-trained models used in the public APIs since they spend lots of money and time to tune and deploy those valuable large-scale models. However, sharing models and insights is a key driver of progress in the AI research field, and if the community becomes more protective of their models, it could slow down advancements.

Acknowledgement This research was partly supported by NSF through grants IIS-1910492 and IIS-2239537. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the A5000 used for this research.

References

Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In 27th USENIX Security Symposium (USENIX Security 18) , pages 1615-1631, Baltimore, MD, August 2018. USENIX Association.

Ahmad Ajalloeian and Sebastian U Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051 , 2020.

Luigi Ambrosio, Nicola Gigli, and Giuseppe Savare. Gradient flows: In metric spaces and in the space of probability measures. (Lectures in Mathematics. ETH) , 2008.

Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, and N. Asokan. Extraction of complex dnn models: Real threat or boogeyman? In Onn Shehory, Eitan Farchi, and Guy Barash, editors, Engineering Dependable and Secure Machine Learning Systems , pages 42-57. Springer International Publishing, 2020.

Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems , 2014.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan. Exploring connections between active learning and model extraction. In 29th USENIX Security Symposium (USENIX Security 20) , pages 1309-1326. USENIX Association, August 2020.

Sinho Chewi, Thibaut Le Gouic, Chen Lu, Tyler Maunu, and Philippe Rigollet. Svgd as a kernelized wasserstein gradient flow of the chi-squared divergence. In Advances in Neural Information Processing Systems , 2020.

Jacson Rodrigues Correia-Silva, Rodrigo F. Berriel, Claudine Badue, Alberto F. de Souza, and Thiago Oliveira-Santos. Copycat CNN: Stealing knowledge by persuading confession with random non-labeled data. In 2018 International Joint Conference on Neural Networks (IJCNN) . IEEE, jul 2018.

Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, and Nicolas Papernot. Increasing the cost of model extraction with calibrated proof of work. In International Conference on Learning Representations , 2022.

Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial extreme value analysis. In International Conference on Learning Representations , 2022.

Junfeng Guo, Yiming Li, Lixu Wang, Shu-Tao Xia, Heng Huang, Cong Liu, and Bo Li. Domain watermark: Effective and harmless dataset copyright protection is closed at hand. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2261-2269, 2017.

Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks. In USENIX Security 2020 , 2020.

Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, and Nicolas Papernot. Entangled watermarks as a defense against model extraction. In 30th USENIX Security Symposium (USENIX Security 21) , pages 1937-1954. USENIX Association, August 2021.

Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In Proceedings of the 42nd IEEE Symposium on Security and Privacy , 2021.

Mika Juuti, Sebastian Szyller, Samuel Marchal, and N. Asokan. Prada: Protecting against dnn model stealing attacks. In 4th IEEE European Symposium on Security and Privacy , 2019.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16 , pages 795-811. Springer, 2016.

Sanjay Kariyappa, Atul Prakash, and Moinuddin K Qureshi. Maze: Data-free model stealing attack using zeroth-order gradient estimation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Sanjay Kariyappa, Atul Prakash, and Moinuddin K Qureshi. Protecting {dnn}s from theft using an ensemble of diverse models. In International Conference on Learning Representations , 2021.

Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adaptive misinformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: network compression via factor transfer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages 2765-2774, 2018.

Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.

Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against machine learning model stealing attacks using deceptive perturbations, 2018.

Guanlin Li, Guowen Xu, Shangwei Guo, Han Qiu, Jiwei Li, and Tianwei Zhang. Extracting robust models with uncertain examples. In The Eleventh International Conference on Learning Representations , 2023.

Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y. Zhao. Blacklight: Scalable defense for neural networks against query-based black-box attacks. In 2022 Conference 31th USENIX security symposium (USENIX Security 22) , 2022.

Qiang Liu. Stein variational gradient descent as gradient flow. Advances in Neural Information Processing Systems , 2017.

Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine , 2020.

Daniel Lowd and Christopher Meek. Adversarial learning. In KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining , page 641-647. Association for Computing Machinery, 2005.

Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In International Conference on Learning Representations , 2021.

Mantas Mazeika, Bo Li, and David Forsyth. How to steer your adversary: Targeted and efficient model stealing defenses with gradient redirection. 2022.

Erich Miersemann. Calculus of Variations, Lecture Notes . Leipzig University, 2012.

Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. International Conference on Learning Representations , 2018.

Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4949-4958, 2019.

Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stealing attacks. In International Conference on Learning Representations , 2020.

Soham Pal, Yash Gupta, Aditya Kanade, and Shirish Shevade. Stateful detection of model extraction attacks, 2021.

Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. Activethief: Model extraction using active learning and unannotated public data. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, 2020.

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In ACM Asia Conference on Computer and Communications Security , 2016.

Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with PATE. In International Conference on Learning Representations , 2018.

Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. 2019.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. 2014.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. International Conference on Learning Representations , 2020.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4510-4520, 2018.

Sunandini Sanyal, Sravanti Addepalli, and R. Venkatesh Babu. Towards data-free model stealing in a hard label setting. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15263-15272, 2022.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1-9, 2015.

Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N. Asokan. Dawn: Dynamic adversarial watermarking of neural networks. In Proceedings of the ACM Multimedia , 2021.

Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In Proceedings of USENIX Security 2016 , 2016.

Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papernot. Data-free model extraction. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning. In In the 39th IEEE Symposium on Security and Privacy , 2018.

Zi Wang. Zero-shot knowledge distillation from a decision-based black-box model. In International Conference on Machine Learning , pages 10675-10685. PMLR, 2021.

Yihan Wu, Aleksandar Bojchevski, and Heng Huang. Adversarial weight perturbation improves generalization in graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10417-10425, 2023.

Yihan Wu, Hongyang Zhang, and Heng Huang. Retrievalguard: Provably robust 1-nearest neighbor image retrieval. In International Conference on Machine Learning , pages 24266-24279. PMLR, 2022.

Ziyu Xu, Chen Dan, Justin Khim, and Pradeep Ravikumar. Class-weighted classification: Trade-offs and robust approaches. Proceedings of the International Conference on Machine Learning , 2020.

Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Zhenrui Yue, Zhankui He, Huimin Zeng, and Julian McAuley. Black-box attacks on sequential recommenders via data-free model extraction. In Proceedings of the 15th ACM Conference on Recommender Systems , RecSys '21. Association for Computing Machinery, 2021.

Santiago Zanella-Beguelin, Shruti Tople, Andrew Paverd, and Boris Köpf. Grey-box extraction of natural language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 12278-12286. PMLR, 18-24 Jul 2021.

Runtian Zhai, Chen Dan, J. Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. Proceedings of the International Conference on Machine Learning , 2021.

Hongyang Zhang, Yihan Wu, and Heng Huang. How many data are needed for robust learning? arXiv preprint arXiv:2202.11592 , 2022.

Xinyi Zhang, Chengfang Fang, and Jie Shi. Thief, beware of what get you there: Towards understanding model extraction attack, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method to defend against depth function preserving model extraction (DFME) attacks. The method proposed by the authors, called MeCo, adds data-dependent random perturbations to the input data, making it difficult for attackers to extract useful information from the black-box model. The authors claim that MeCo effectively reduces the accuracy of the cloned model while maintaining the utility of the target model. The approach was evaluated on several data sets and compared to existing defense strategies, showing excellent performance in terms of effectiveness, computational efficiency, and memory usage.\n\nSTRENGTHS:\n1.The quality of the work is evident in the extensive experiments conducted and the clear presentation of results. \n2.The main advantage of the paper is its novel approach to defend against DFME attacks. The proposed method, MeCo, is both effective and efficient, which is demonstrated by experimental results. However, these advantages may be affected by a lack of theoretical support.\n\nWEAKNESSES:\n1.The paper only conducted experiments in cifar10, cifar100 and mnist data sets, but the lack of relevant experiments on large data sets led to some doubts about the extensibility of the model.\n2.The model may need more testing and proof in generalization.\n\nQUESTIONS:\nCan you provide more details on the theoretical basis of the proposed approach?\nHow does the proposed method perform under different attack scenarios?\n\nLIMITATIONS:\nThe authors acknowledge the lack of theoretical analysis as a limitation of the current work and take it as a direction for future work. They also mention the potential benefits of their proposed defense training for protecting large-scale pre-trained models used in public apis. However, a more detailed discussion of the possible negative social impacts of the proposed approach would be useful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* __Problem Statement__: The paper proposes a defense against data-free model extraction attacks (where the attacker black-box queries a victim image classifier, s.t queries are then used to train a clone model)\n* __Approach__: The proposed defense \"DRO\" returns class probabilities as a result of the defender *perturbing* the input image. The objective of the perturbation is to flip the top-1 predicted class of attacker's queries (characterized by lying close to decision boundaries), but while preserving utility (measured by L1 distance to unperturbed probabilities).\n* __Experimental findings__: The approach is evaluated on 4x standard datasets, compared against 3x recent baseline approaches and in addition provides more studies (e.g., ablation). Results indicate that the proposed approach outperforms the baselines (in terms of both accuracy and utility).\n\nSTRENGTHS:\n1. __Originality__: 4/5. The approach is original and well-motivated. Although some ideas have been explored before (i.e., perturbing outputs, changing gradient direction), the proposed approach explores a novel objective (learnt perturbation of inputs to introduce a targeted shift in prediction) and proposes a reasonable way to achieve it.\n2. __Quality__: 4/5. The paper is high-quality. The insights motivating the approach are sound and the experimental section is exhaustive.\n3. __Clarity__: 3/5. The approach is somewhat clear (certain technical choices/notation are unclear; more under questions).\n4. __Significance__: 3/5. The approach is somewhat significant -- it clearly improves upon existing defenses, however against a very specific class of model extract attacks. Specifically, \"data-free\" attacks that have shown to be successful largely in extreme settings (such as with attacker querying the victim millions of times).\n\nWEAKNESSES:\n1. __DRO optimization setup - unclear__: I found the intuition for the defense objective (Fig 2, L148-207) clear. However, it is unclear how the objective formulation (Eq. 4-6) achieves this. Based on Fig. 2 and L183-190, it appears that the requirement is to perturb images s.t the label is flipped (i.e., maximize CE loss wrt parameters $\\omega$), however the outer maximization in Eq. 6 seems to do the opposite? Furthermore, in Eq. 6, since the utility is upper-bounded $||\\cdots||_1 \\le \\tau$, it appears that the perturbation generator $h_\\omega$ can simply learn an identity function?\n2. __Evaluation curves missing__: The paper although extensively evaluates the defense, it appears that it is primarily for a specific defense effectiveness-utility operating point (specific value of $\\gamma$ and $Q$). This contrasts prior works which evaluates a curve of attacker's vs. defender's classification accuracies (e.g., Fig. 4 GRAD, Fig. 4 PredPoison). Without such a curve, it's somewhat unclear to determine the overall performance of the attack.\n3. __Perturbation budget $\\le$ 1.0__: The L_1 perturbation budget $\\tau$ in the defense chosen appears to be a fixed to 1 (L222, L298) seems unreasonably high. After all, this allows the defender to always flip the top-1 class-label.\n\nQUESTIONS:\n1. Hyperpameter choices\n    1. How were the defense hyperparameters determined? I'm concerned that if they are a result of hyperparam searches (e.g., over a sweep of attacks), it might not generalize to novel attacks.\n    2. Hyperparameter choices: Are the defense hyperparameters fixed for all choices of datasets, clone model architectures, and attacks? \n2. DRO optimization objective vs. Adversarial training: Can the authors clarify on how the min-max optimization objective (Eq. 4) is different from adversarial training? After all, in both cases, the objective is to find parameters $\\theta$ s.t predictions are robust to perturbed inputs. \n3. (Suggestion) Sec 5.4 Adaptive attacker: Please elaborate on how the attacker \"adapts\". It is unclear what the attacker does from L341.\n\nLIMITATIONS:\nLimitations are adequately discussed in Sec. 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
In this paper, authors propose a novel principled defensive training framework that substantially improves the memory and computation efficiency during deployment to defend against DFME attacks and a distributionally robust optimization method to randomly perturb the inputs to defend against DFME effectively while maintaining the model utility simultaneously.

STRENGTHS:
1. I agree that this work would be beneficial for protecting current large-scale pre-trained models used in the public APIs.

2. The experiment in this article is sufficient and the description is clear. And the method proposed by the author is novel to a certain extent.

3. The author's description of the existing research background is clear, and the analysis of the problems existing in the field is satisfactory.

WEAKNESSES:
1. In my opinion, the method proposed by the author has defects in protecting the performance of the original model, that is, it does not fully take into account the impact on the performance of the target model, which can also be seen from Table 3. After all, the weak accuracy improvement of these models in practical applications is expensive.
2. As stated by the author, the methodology of this article lacks a theoretical explanation. The intuitive explanation of the mechanism of action of the model is not deep enough.

QUESTIONS:
See weakness

LIMITATIONS:
This method does not adequately evaluate the negative effects of the target model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes a defense against data-free model extraction attack (DFME). The basic idea is to add randomized data-dependent perturbations to the input query. It proposes a new training method that considers the perturbation generator to mitigate the risks of dropping the benign accuracy. Namely, it will trian the model and the perturbation generator so that the model can behave properly on the benign queries.

STRENGTHS:
It is interesting to add random perturbations to the test data for defending against the DFME attacks.

The paper is well organized, with all items discussed.

WEAKNESSES:
From the very high level, I do not see how this works. The method tries to add perturbations to the test data. This will obviously decrease the benign accuracy. But I am not able to identify what makes the training method work. In DFME attacks, the queries are just normal queries, which can be similar to the ones used in training or testing. I am not sure how the method can work without identifying different types of queries.

One argument may be the query in the attack will be different from the ones used in training. Does that mean that the defense assumes the attack query out of distribution? I think out-of-distribution attack queries will surely degrade the attack performance. 

Is there a theoretical guarantee of the proposed method?

QUESTIONS:
How does the method tell the differences between normal and malicious queries?

Is there a theoretical guarantee of the proposed method?

LIMITATIONS:
The paper discussed the limitations and broader impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07