SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenyi Wang $^{1}$ , Li Shen $^{2}$ , Tongliang Liu $^{3}$ , Tiehang Duan $^{4}$
Yanjun Zhu $^{5}$ , Donglin Zhan $^{6}$ , David Doermann $^{7}$ , Mingchen Gao $^{7}$
<sup>1</sup>University of Maryland, College Park, USA <sup>2</sup>JD Explore Academy, China
<sup>3</sup>The University of Sydney, Australia <sup>4</sup>West Virginia University, USA
$^{5}$ Northeastern University, USA  $^{6}$ Columbia University, USA
<sup>7</sup>University at Buffalo, USA
zwang169@umd.edu; mathshenli@gmail.com; tongliang.liu@sydney.edu;
tiehang.duan@gmail.com; ya.zhu@northeastern.edu;
dz2478@columbia.edu; {doermann, mgao8}@buffalo.edu
Data-Free Model Extraction (DFME) aims to clone a black-box model without knowing its original training data distribution, making it much easier for attackers to steal commercial models. Defense against DFME faces several challenges: (i) effectiveness; (ii) efficiency; (iii) no prior on the attacker's query data distribution and strategy. However, existing defense methods: (1) are highly computation and memory inefficient; or (2) need strong assumptions about attack data distribution; or (3) can only delay the attack or prove a model theft after the model stealing has happened. In this work, we propose a Memory and Computation efficient defense approach, named MeCo, to prevent DFME from happening while maintaining the model utility simultaneously by distributionally robust defensive training on the target victim model. Specifically, we randomize the input so that it: (1) causes a mismatch of the knowledge distillation loss for attackers; (2) disturbs the zeroth-order gradient estimation; (3) changes the label prediction for the attack query data. Therefore, the attacker can only extract misleading information from the black-box model. Extensive experiments on defending against both decision-based and score-based DFME demonstrate that MeCo can significantly reduce the effectiveness of existing DFME methods and substantially improve running efficiency.
Model extraction attack aims to replicate the functionality of a public API with only query access. Most model extraction methods focus on data-based model extraction, i.e., an attacker can access a small subset of the in-distribution training data [41], or a relevant surrogate dataset [36] of the target model. Beyond data-based model extraction (DBME), recent promising results with data-free model extraction (DFME) show that the attacker can clone a model with performance close to that of the target black-box model even without prior knowledge of the distribution of the proprietary training data. Those DFME techniques make it much easier for attackers to steal the model without collecting and annotating relevant training data. DFME can be further categorized into score-based DFME [21, 51], where the target model provides softmax probability outputs to the users; and decision-based DFME [54, 47], where the target model only provides the top-1 label. Thus, the model owners
$^{1}$ Corresponding author: Zhenyi Wang, Li Shen and Mingchen Gao
The problem setting name, DFME, coincides with the method name in [51]. We use DFME to denote the problem setting of model extraction without accessing the in-distribution training data, not a specific method.
face a more critical problem - how to prevent those black-box pre-trained models from stealing in the data-free setting while simultaneously guaranteeing a high-quality inference service? The main challenges of defending against DFME are: (1) effectiveness requirement: the defender should substantially reduce the clone model accuracy; (2) efficiency requirement: the defense procedure should be memory and computation efficient; (3) lack of knowledge on the attackers: the attack query data distribution and attack strategy are unknown to the defender.
Existing work on Model Extraction (ME) defense primarily focuses on data-based ME attacks. We categorize the existing ME defense methods into pre-attack, delay-attack, and post-attack defenses. Pre-attack defenses aim to prevent ME attack from happening. They either perturb the output probabilities [38, 33] or integrate multiple models [22] or detect attack queries from benign queries [23]. Delay-attack defenses aim to delay the ME attack instead of preventing it [10]. Post-attack defenses neither prevent nor delay the ME; instead, they aim to prove a model theft [17, 18, 32]. Our work falls under the pre-attack defense category since our goal is to prevent ME from happening. However, existing pre-attack defense methods have limitations when applied to DFME: (1) output perturbation-based methods perform optimization during deployment is computationally and memory expensive; (2) ensemble-based methods are memory inefficient since they need to store multiple models; (3) detection-based methods require strong assumptions about the query data distribution; the attacker can easily make the attack queries indistinguishable from benign ones [59] and circumvent the detection [28], rendering the defense ineffective; (4) some defense methods [33, 23, 22] require the knowledge of the attack query data. However, this prior knowledge is unknown to the defender.
To address existing defense methods' limitations, we propose a new and orthogonal class of defense method, named Memory and Computation efficient defense (MeCo), through a randomized defense strategy. Specifically, MeCo adds data-dependent random perturbation to the query input. MeCo can effectively defend against DFME for several reasons. For score-based DFME: (i) it leads to a mismatch of the knowledge distillation loss for attackers; (ii) existing DFME methods heavily rely on zeroth-order gradient estimation. MeCo can disturb the zeroth-order gradient estimation so that the attacker can only obtain its inaccurate estimation. For decision-based DFME, MeCo can change the label prediction of attack query data. The attacker can only learn from incorrectly labeled data.
MeCo would reduce the target model utility on benign queries without additional mechanisms. Maintaining the target model utility needs to: (i) minimally sacrifice the classification accuracy on the test set, and (ii) minimize the perturbation magnitude of the output class probability. To maintain the target model utility on benign queries, we propose a principled distributionally robust optimization (DRO) framework to train the target model and perturbation generator. Our DRO framework simulates the worst-case in-distribution (ID) test data from the training data (accessible for the defender). Then, we apply the random perturbation to the simulated test data and train the target model and perturbation generator on the simulated test data to ensure the worst-case generalization on the ID test data. Compared to existing works, MeCo has numerous advantages, including (1) MeCo is substantially more computation and memory efficient without complex optimization and storing multiple models during deployment; (2) it avoids detecting attack queries from benign ones; (3) does not need the knowledge of the attack query data distribution. More importantly, even if attackers know our defense strategy and adopt an adaptive attack strategy, MeCo is still effective since stealing a random function further increases the difficulty of ME thanks to the randomness introduced in MeCo. Extensive experiments compared to various defense methods show that MeCo significantly reduces the accuracy of the clone model across different query budgets.
In summary, our main contributions are three-fold:
- We propose a novel principled defensive training framework that substantially improves the memory and computation efficiency during deployment to defend against DFME attacks.
- We propose a distributionally robust optimization (DRO) method to randomly perturb the inputs to defend against DFME effectively while maintaining the model utility simultaneously.
- Extensive experiments on defending against both score-based and decision-based DFME show the effectiveness of MeCo, reducing the clone model accuracy by up to  $35\%$ , while maintaining the target model utility. Further, MeCo can also effectively defend against data-based ME and boost its performance. More importantly, MeCo achieves substantially more computation and memory efficiency than existing methods, e.g.,  $17 \times \sim 172 \times$  speed up.
Data-Free Model Extraction (DFME). Model extraction (ME) attack aims to extract and clone the functionality of the public API with only query access; representative works include [31, 9, 50, 53, 35, 36, 7, 63, 16, 40, 4, 60, 27]. Recently, ME has been extended to the data-free setting, named DFME, which cannot access original training data. ZSDB3KD [54] and DFMS-HL [47] focus on the decision-based setting, i.e., only the hard label (top-1 label) is predicted for the attacker. MAZE and DFME [21, 51] are score-based methods, i.e., soft label (softmax output) is delivered to the attacker.
Model Extraction Defense. We categorize the model extraction defense methods into three categories: pre-attack defenses, delay-attack defenses, and post-attack defenses, according to when the defense happens during an attack. (i) Pre-attack defenses aim to prevent the ME attack from happening. There are three classes of methods: (1) output probabilities perturbation-based methods [26, 38, 33]: Prediction poisoning (P-poison) [38] and GRAD [33] perform a complex optimization during deployment; (2) ensemble-based methods: EDM [22] integrates multiple models for diverse predictions. (3) detection-based methods: Adaptive misinformation [23], PRADA [19] and VarDetect [39] detect the attack queries from the benign queries. However, those methods have limitations when applied to DFME: (1) they [38, 33] significantly increase the computation and memory cost; (2) they [23, 19, 39] have a high risk of incorrectly classifying attack or benign queries. The attacker can easily evade the detection [28, 11], making the defense ineffective. (3) GRAD [33], Adaptive misinformation [23], and EDM [22] need to know the prior knowledge and distribution of attack query distribution, which is unknown to defenders in the data-free setting. Our method falls under this pre-attack defense category and addresses their limitations from a novel and orthogonal perspective. (ii) Delay-attack defenses aim to delay the ME attack instead of preventing the attack. Proof of work (PoW) [10] delays model stealing by significantly increasing the computation cost of query access for model extraction, e.g., solving a puzzle. Ours is fundamentally different from PoW in two aspects: (1) PoW could not prevent model stealing if the users spend more time, computation cost, money, etc. By contrast, our method is to prevent model stealing instead of delaying model stealing; (2) PoW needs multiple teachers to evaluate the privacy leakage of queries [42]. Our method only requires a single teacher; thus, ours is substantially more memory and computation efficient than PoW. (iii) Post-attack defenses aim to prove a model theft after a pre-trained model has been stolen, e.g., through watermark-based [1, 12] methods [17, 49], proof-of-learning [18] and dataset inference [32]. However, the post-attack defenses only perform verification of model theft but cannot prevent the model from being stolen. This requires a model owner to obtain access to the stolen model. If the stolen model is not used as API, the defender cannot verify whether the model has been stolen.
Distributionally Robust Optimization (DRO) DRO is a flexible and powerful optimization framework to make decisions under uncertainty [43, 55], where robustness is an important factor [62, 56]. DRO constructs a set of probability distributions, known as an ambiguity set, and then minimizes the worst-case performance within the ambiguity set, thus guaranteeing the model performance. There have been various machine learning applications of DRO, such as dealing with group-shift [45], subpopulation shift [61], and long-tailed learning [57]. To the best of our knowledge, our work is the first principled method with DRO for DFME defense.
Data-Free Model Extraction (Attacker). In DFME, the attacker sends a query input  $\mathbf{x}$  to the target victim model  $T$  parameterized with  $\theta_T$  and receives a prediction  $P(\mathbf{y}|\mathbf{x}) = T(\mathbf{x};\theta_T)$ . For the score-based setting [21, 51], the target model delivers the soft label (output class probabilities) to the attacker. In the decision-based setting [54], the target model only delivers the hard label (the top-1 class) prediction to the attacker. Following [54, 21, 51], we assume the following attacker knowledge: (1) data-free: the attacker cannot access the original training data distribution of the target victim model. The attacker typically employs synthetic out-of-distribution (OOD) data to query the target model in an attempt to steal it. (2) black-box: the attacker does not know the architecture and model parameters of the target model. Given a test dataset  $\mathcal{D}_{test}$  associated with the black-box model, the attacker aims to train a compact clone model  $C$  with parameters  $\theta_C$  that maximize the testing accuracy.
Defense against DFME (Defender). Following [38, 22], we assume the defender does not know: (1) whether a query is malicious or benign; (2) the attack strategies adopted by an attacker; (3) the model architecture used by an attacker. The goals of the defender are three-fold: (1) effective: minimize the test accuracy that the attacker can achieve; (2) utility preserving: maintain the accuracy on benign inputs and minimize the perturbation magnitude of the output probability; (3) efficient: the defense procedure should be memory and computation efficient.
3.2 Preliminaries
Knowledge Distillation (KD). Existing DFME methods build on top of KD [5, 14, 44, 58, 24]. Assume we have a pre-trained teacher (target) model  $T$  with parameters  $\theta_{T}$ , and a student (clone) model  $C$  with parameters  $\theta_{C}$ . Suppose the output probabilities of teacher and student for the input  $\mathbf{x}$  are  $T(\mathbf{x};\theta_T)$  and  $C(\mathbf{x};\theta_C)$ , respectively. The training objective of KD is as the following:
$$
\mathcal {L} (\boldsymbol {x}, y) = \mathcal {L} _ {c} (\boldsymbol {x}, y) + \alpha \mathbb {K L} \left(T (\boldsymbol {x}; \boldsymbol {\theta} _ {T}), C (\boldsymbol {x}; \boldsymbol {\theta} _ {C})\right), \tag {1}
$$
where  $\mathcal{L}_c(x,y)$  is the cross-entropy loss,  $\mathbb{KL}$  is the KL divergence between two probability distributions, and  $\alpha$  is the weighting factor. In this work, we focus on defending against score-based and decision-based DFME methods. Due to space limitations, we give a brief description of score-based DFME while placing the details of decision-based DFME in Appendix 8.
Score-based DFME. The representative score-based DFME works are [51, 21]. We briefly describe how they work. The attacker has a pseudo data generator  $G$  parameterized by  $\theta_G$  with random vector  $z$  as input. It generates pseudo data by  $x = G(z; \theta_G)$ , where  $z \sim N(0, I)$ . Then, the attacker sends the query  $x$  to the target and clone model; they output class probabilities according to the generated pseudo data, i.e.,  $y_T = T(x; \theta_T)$  and  $y_C = C(x; \theta_C)$ . The attacker jointly optimizes the clone model  $\mathcal{C}$  and generator  $\mathcal{G}$  as below:
$$
\mathcal {L} _ {\mathcal {C}} = \mathbb {K L} (\boldsymbol {y} _ {T} | | \boldsymbol {y} _ {C}), \quad \mathcal {L} _ {\mathcal {G}} = - \mathbb {K L} (\boldsymbol {y} _ {T} | | \boldsymbol {y} _ {C}) \tag {2}
$$
Since the target network is black-box, to backpropagate into the generator network weights, they apply zeroth-order gradient [30] on the outputs of the generator, i.e.,
$$
\nabla_ {\boldsymbol {\theta} _ {G}} \mathcal {L} _ {G} = \frac {\partial \mathcal {L} _ {G}}{\partial \boldsymbol {\theta} _ {G}} = \frac {\partial \mathcal {L} _ {G}}{\partial \boldsymbol {x}} \frac {\partial \boldsymbol {x}}{\partial \boldsymbol {\theta} _ {G}}, \quad \text {w h e r e} \frac {\partial \mathcal {L} _ {G}}{\partial \boldsymbol {x}} = \frac {1}{m} \sum_ {i = 1} ^ {m} \frac {\mathcal {L} _ {G} (\boldsymbol {x} + \delta \boldsymbol {\mu} _ {i}) - \mathcal {L} _ {G} (\boldsymbol {x})}{\delta} \boldsymbol {\mu} _ {i} \tag {3}
$$
where  $\pmb{\mu}_i$  is a random direction;  $m$  is the number of queries;  $\delta$  is a small step size. The generator  $G$  and the clone model  $C$  alternatively update their parameters by minimizing the loss  $\mathcal{L}_{\mathcal{C}}$  and  $\mathcal{L}_{\mathcal{G}}$ . The attacker can obtain a clone model  $C(\pmb{x};\pmb{\theta}_C)$  after a certain number of training iterations.
To defend against DFME, we propose a DRO defensive training strategy. We present the defense method in Section 4.1. We then discuss the defensive training details and deployment algorithm in Section 4.2. We illustrate how the proposed defense can defend against DFME in Section 4.3.
4.1 Distributionally Robust Defensive Training
The core techniques of the score-based DFME are two-fold: (1) using KD loss to match the target and clone model outputs; (2) zeroth-order gradient estimation. On the other hand, decision-based DFME mainly relies on the label prediction of the target model. We propose a random perturbation technique to make the attacker estimate misleading information. However, adding random noise to the query input strongly restricts the flexibility of random perturbation since it is difficult to maintain the model utility while effectively defending against DFME simultaneously. Furthermore, the added randomness to the input should be input-dependent since different inputs have different sensitivities to the neural network decision boundary. We thus propose a flexible random perturbation generator to learn a data-dependent random perturbation that can adaptively generate different perturbations for different inputs while maintaining the performance on benign input. First, we denote the data-dependent random perturbation generator as  $h_{\omega}(\pmb{x}, \pmb{\epsilon})$  parameterized by parameters  $\omega$  with query  $\pmb{x}$  and random noise  $\pmb{\epsilon}$  as input. The perturbation generator is a two-block ResNet with filter 64. It only accounts for a tiny proportion of the target model. We explain our intuition in Figure 1. Our method builds
Figure 1: Illustration of our defense method. (a) The perturbation generator takes the input image and random Gaussian noise as input to generate data-dependent perturbation, which is added to the original data to disturb model extraction. (b) According to [54], the attack queries are closer to the decision boundary than the benign (positive and negative) queries. All queries with added perturbation become even closer to the decision boundary, causing some attack queries to invert labels while leaving the labels of benign queries unaffected. (c) The perturbation generator generates large perturbation on attack queries so that the output class probabilities are perturbed more significantly (top-1 label may change). In contrast, the outputs on benign queries are only perturbed slightly due to the DRO defensive training.
(a)
Figure 2: Illustration of our proposed defense mechanisms. (a) Using query inputs without any input perturbation results in favorable model utility but could not defend against model extraction attack. (b) The red region represents the distribution of benign query data. Applying input perturbation to every input yields excellent defense performance but compromises model utility. (c) The red region illustrates the distribution of benign query data. Distributionally robust optimization (DRO) ensures model utility by minimizing loss on the worst-case perturbed training data (simulation of test data), leading to significantly smaller perturbation magnitudes on benign inputs compared to training without DRO. Beyond the distribution of benign query data distribution (attack queries), DRO isn't employed, leading to arbitrary perturbation magnitudes on those inputs and strong defense against DFME.
on the intuition [54] that the attack query is closer to the decision boundary, while the benign query is farther away from the decision boundary in the DFME setting. We add random perturbation to the query input so that the attack query is closer to or crosses the decision boundary. Their output probabilities from the target model will be perturbed more significantly in the score-based setting. Their labels are more likely to be flipped in the decision-based setting. In contrast, benign queries are far from the decision boundary, thus not influencing benign queries much.
Simply adding the proposed data-dependent random perturbation to the query inputs would reduce the model utility. We propose a Distributionally Robust Optimization (DRO) framework to train the target model and perturbation generator to maintain the target model utility, i.e., (i) minimally sacrifice the classification accuracy on the test set, and (ii) minimize the perturbation magnitude of the output class probabilities. In addition, as depicted in Figure 2, we conducted a comparative analysis of three scenarios: one without random input perturbation, another with random input perturbation, and the third involving DRO defensive training. Our DRO defensive training exhibits a dual capability in effectively defending against DFME while maintaining the model utility.
Figure 3: The proposed DRO framework perturbs original training data distribution to simulate worst-case test data distributions considering the high uncertainty of test data distribution represented by the blue and orange color on the right figure.
Since the test data distribution is unknown during deployment, we propose a flexible framework with DRO to optimize under uncertainty [43]. DRO constructs an ambiguity set of probability distributions and optimizes the worst-case performance within the ambiguity set, thus guaranteeing
the performance. Our framework takes that the underlying probability distribution of test data  $\mu$  is unknown and lies in an ambiguity set of probability distributions around the training data distribution  $\mu_0$ . The proposed DRO framework perturbs the training data distribution to simulate the test data distribution, illustrated in Figure 3. It optimizes the worst-case performance in the ambiguity set to guarantee the model utility after adding random perturbation. We formulate the proposed DRO framework in the probability measure (distributions or densities) space:
$$
\min  _ {\boldsymbol {\theta} _ {T}, \boldsymbol {\omega} _ {\mu \in \mathcal {P}}} \sup  _ {\boldsymbol {\epsilon}} \mathbb {E} _ {\boldsymbol {x} \sim \mu} \mathcal {L} \left(\boldsymbol {\theta} _ {T}, \boldsymbol {x} + h _ {\boldsymbol {\omega}} (\boldsymbol {x}, \hat {\boldsymbol {\epsilon}}), y\right) \tag {4}
$$
$$
\mathrm {s . t .} \mathcal {P} = \left\{\mu : \mathbb {K L} \left(\mu \| \mu_ {0}\right) \leq \beta \right\} \tag {5}
$$
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mu} \| T (\boldsymbol {x} + h _ {\omega} (\boldsymbol {x}, \epsilon)) - T (\boldsymbol {x}) \| _ {1} \leq \tau , \tag {6}
$$