Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models
Year: 2023

Abstract:
Conventional knowledge distillation (KD) methods require access to the internal information of teachers, e.g., logits. However, such information may not always be accessible for large pre-trained language models (PLMs). In this work, we focus on decision-based KD for PLMs, where only teacher decisions (i.e., top-1 labels) are accessible. Considering the information gap between logits and decisions, we propose a novel method to estimate logits from the decision distributions. Specifically, decision distributions can be both derived as a function of logits theoretically and estimated with test-time data augmentation empirically. By combining the theoretical and empirical estimations of the decision distributions together, the estimation of logits can be successfully reduced to a simple root-finding problem. Extensive experiments show that our method significantly outperforms strong baselines on both natural language understanding and machine reading comprehension datasets.1

heading: 1 Introduction
text: Various natural language processing (NLP) tasks have witnessed promising performance from large pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020). However, PLMs are usually computationally expensive and memory intensive, hindering their deployment on resource-limited devices. Knowledge distillation (KD) (Hinton et al., 2015) is a popular technique to transfer knowledge from large PLMs to lightweight models. Previous KD works utilize various types of internal information from the teacher model, such as output logits (Sanh et al., 2019; Tang et al., 2019; Liu et al., 2020), hidden states (Sun et al., 2019b; Jiao et al., 2020),
† Peng Li (lipeng@air.tsinghua.edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn) are corresponding authors.
1Our code is available at https://github.com/THUNLP-MT/ DBKD-PLM
and attention maps (Li et al., 2020). In real-world applications, however, these types of information are sometimes not accessible due to commercial and privacy issues (Brown et al., 2020; Ouyang et al., 2022). Specifically, large-scale PLMs usually only provide decisions (i.e., top-1 labels) to users. Motivated by this scenario, we investigate the task of decision-based KD (Wang, 2021) for PLMs, in which only decisions of teacher predictions are available.
The information gap between teacher decisions and its internal states is the major challenge for the task. A straightforward approach for decisionbased KD is to treat teacher decisions as ground truth labels and use these labels to train a student model (Zhang et al., 2022; Sanyal et al., 2022). However, previous work reveals that logits contain rich knowledge (Hinton et al., 2015), relying only on decisions obviously suffers from information loss. To alleviate the problem, Wang (2021) proposes the DB3KD method to generate pseudo soft labels according to the sample’s robustness. However, DB3KD requires that the input of a model can be modified continuously (e.g., image), which hinders its application on PLMs as their inputs are discrete tokens. Therefore, how to fill the information gap under the discrete input setting remains a challenging problem.
Fortunately, the development of test-time data augmentation for discrete input (Liu, 2019; Shleifer, 2019; Xu et al., 2022) brings hope for resolving the challenge. The basic idea is to modify selected tokens in a piece of text under certain constraints to generate augmented samples and estimate or improve the desired properties of a model based on its behaviors on these samples. Test-time data augmentation has been shown to be effective for uncertainty estimation (Ayhan and Berens, 2018; Smith and Gal, 2018; Wang et al., 2019), adversary robustness (Xu et al., 2022), and so on. Is it possible to narrow down the information gap
13234
with test-time data argumentation in decision-based KD for PLMs?
In this work, we propose a novel decision-based KD method for PLMs. As illustrated in Figure 1, our method is capable of estimating the teacher logits for classes even without observed decisions, narrowing down the information gap between decision and logits. Specially, we estimate the logits by combining test-time data argumentation and noncentred orthant probability estimation. On the one hand, we can obtain an empirical estimation of the decision distribution around a sample by test-time data argumentation. On the other hand, we can also derive a theoretical formula for the decision distribution as a non-centred orthant probability, which is a function of logits. As a result, the problem of logits estimation can be reduced to finding the root of the equation that the function takes the value of the empirical estimation. Extensive experiments on various natural language understanding and machine reading comprehension datasets demonstrate the effectiveness of our proposed method, which outperforms strong baselines significantly. Moreover, quantitative analysis reveals that our method obtains better estimation of logits, narrowing down the information gap.

heading: 2 Related Work
text: Decision-based Knowledge Distillation. To advance conventional knowledge distillation (KD) to more challenging black-box model scenarios, Wang (2021) first propose the problem of decisionbased KD, where only teacher decisions (i.e., top-1 labels) are accessible to students. They address the problem by estimating the soft label (analogy to output probabilities) of a sample based on its distance to the decision boundary, which involves continuous modification to the original input fed to the black-box model. Instead, Zhang et al. (2022) and Sanyal et al. (2022) synthesize pseudo data in continuous space and leverage decisions of the teacher model on these data directly. In this work, we focus on decision-based KD for PLMs. Unfortunately, the original inputs of the PLMs are discrete tokens which can not be continuously modified. Therefore, these methods are not applicable to our scenario.
Decision-based KD is also related to black-box KD (Orekondy et al., 2019; Wang et al., 2020) and distillation-based black-box attacks (Zhou et al., 2020; Wang et al., 2021; Truong et al., 2021;
Kariyappa et al., 2021; Yu and Sun, 2022). Both of them involve distilling a student model from black-box models. However, these works generally assume that the score-based outputs are accessible. Decision-based KD focuses on a more challenging scenario where only top-1 labels are accessible.
Test-Time Data Augmentation. Test-time data augmentation is a common technique in computer vision (Krizhevsky et al., 2009; Simonyan and Zisserman, 2015; He et al., 2016; Wang et al., 2019; Lyzhov et al., 2020; Shanmugam et al., 2021) and is also feasible for natural language processing (NLP) (Liu, 2019; Shleifer, 2019; Xu et al., 2022). Although differing in final purpose, test-time and training-time data augmentation share a large portion of common techniques in NLP. Due to the discrete nature of language, one line of work conducts augmentation by modifying tokens based on rules (Şahin and Steedman, 2018; Wei and Zou, 2019; Chen et al., 2020a) or models (Sennrich et al., 2016; Yang et al., 2020; Quteineh et al., 2020; Anaby-Tavor et al., 2020), and another line of work operates in embedding or representation space (Chen et al., 2020b; Cheng et al., 2020; Chen et al., 2021; Wei et al., 2022). In this work, as we do not have access to the teacher model, we follow the first line of work to conduct test-time data augmentation.

heading: 3 Background
text: Knowledge Distillation (KD) is a technique that aims to transfer knowledge from the teacher model to the student model by aligning certain statistics, usually the logits, of the student to those of the teacher. Given input x, we denote the pre-softmax logits vector of the teacher and student as z and v, respectively. The process of KD involves minimizing the Kullback-Leibler (KL) divergence between the probabilities induced from z and v as follows:
LKD = KL (softmax(v/τ)||softmax(z/τ)) , (1) where τ is the temperature hyper-parameter. The student model is trained by minimizing the loss function
L = LCE + λLKD, (2)
where LCE is the cross entropy loss over the ground-truth label, and λ is the scaling factor used for balancing the importance of the two losses.

heading: 4 Methodology

heading: 4.1 Overview
text: In the decision-based scenario, the z item in Eq. 1 is not accessible. Instead, the PLM API returns the model decision d = argmax
1≤j≤L zj , where zj is the j-
th logit in z, and L denotes the dimension of output label space. Obviously, d only carries the information of “the j-th logit is the largest”. In constrast, z contains richer information. For example, comparing z1 = [0.6, 0.3] and z2 = [0.9, 0.1], although they correpond to the same decision, i.e., d = 1, they also imply that the second sample seems more likely to be of the first class. Therefore, there is a big information gap between logits and decsions, and our proposed method aims at narrowing down the gap by finding a better estimation of the logits.
Figure 1 shows the framework of our proposed method. The key idea is combining the empirical and theoretical estimations of the conditional decision distribution P (Y |x), where Y is a random variable denoting decision, to form an equation whose solution is the logits. Specially, first we leverage test-time data augmentation to generate N augmented samples for the given sample x and collect the teacher decisions for them. Then, we build an empirical estimation P̃ (Y |x) for P (Y |x) based on the decisions. Next, we derive a theoretical estimation Q(Y |x; ẑ) parameterized by the true logits ẑ for P (Y |x) and form the following equation
P̃ (Y |x)−Q(Y |x; ẑ) = 0. (3) Finally, by solving for ẑ, we get the estimated
logits and the student model can be trained by conventional KD (Eq. 2).
Compared with the existing decision-based KD method (Wang, 2021), our method leverages data augmentation instead of binary search and optimization on the input samples. Therefore, it is applicable to discrete inputs which can hardly be searched or optimized.

heading: 4.2 Empirical Estimation of the Conditional Decision Distribution
text: In this secion, we will introduce how to get P̃ (Y |x) in Eq. 3. Given a sample x and a teacher model Mθ parameterized by θ, we first generate N augmented samples X = {x̃i = F(x, i)}Ni=1 with a test-time data augmentation function F(·, ·). Then the teacher decisions D = {di = Mθ(x̃i)} are collected. Finally, P (Y |x) is approximated as
P̃ (Y |x) ≈ 1 N
N∑
i=1
1di , (4)
where 1di ∈ {0, 1}L is a L-dimensional one-hot vector whose di-th element is 1, and L is the number of categories. F(·, ·) plays a crucial role in the process and an ideal F(·, ·) should satisfy three requirements. First, it should conserve true labels (Wang et al., 2019). Second, it should have a low computational cost, since it will be computed repetitively. Third, the degree of noise introduced by F(·, ·) should be quantizable and controllable, crucial for the following steps. Thus, following Wei and Zou (2019), we define F(·, ·) as an operation randomly sam-
Algorithm 1: Teacher Logits Estimation Data: Input text x. Result: Teacher logits estimation ẑ Require: Teacher modelMθ, data augmenta-
tion transformation function F(·, ·), augmented data number N , maximal iteration number m, error bound ϵ, hyper-parameter σ, label number L. // Empirical estimation of the conditional decision distribution
1: {x̃n}Nn=1 ← {F(x, n)}Nn=1 2: {dn}Nn=1 ← {Mθ(x̃n)}Nn=1 3: P̃ (Y |x)← 1N ∑N i=1 1di
// Solving equation to obtain ẑ 4: k ← 0 5: ẑ ← 0 // ẑ = [ẑi], 1 ≤ i ≤ L 6: Initialize R ∈ R(L−1)×(L−1) whose diago-
nal elements are σ2 and the others are 2σ2
7: repeat 8: p← 0 // p = [pi], 1 ≤ i ≤ L 9: for i = 1, i ≤ L do
10: µ← [ẑi − ẑ1, . . . , ẑi − ẑi−1, ẑi − ẑi+1, . . . , ẑi − ẑL] 11: B ← CholeskyDecompose(R) 12: pi ←RecursiveIntegration(µ,B) 13: i← i+ 1 14: end for 15: k ← k + 1 16: ẑ ← P̃ (Y |x)− p+ ẑ 17: until |p− P̃ (Y |x)| ≤ ϵ or k = m 18: return ẑ
pled from synonym replacement, random insertion, random swap, and random deletion operations.

heading: 4.3 Theoretical Estimation of the Conditional Decision Distribution
text: In this section, we will introduce how to get the theoretical estimation Q(Y |x; ẑ) parameterized by ẑ in Eq. 3. The outline of the derivation is that we assume the logits are sampled from an L-dimensional distribution P (Z|x), where Z = [Zj ] is an Ldimensional random variable denoting logits. Then Q(Y = i|x; ẑ) is equal to the probability that the i-th dimension of Z takes the largest value, which can be calculated mathematically from P (Z|x).
Following the above outline, we have Q(Y = i|x; ẑ) = P ( Zi = max
1≤j≤L Zj
∣∣∣x ) . (5)
To derive the above probability, we reformulate it
in terms of orthant probability. First, we introduce an L − 1 dimensional auxiliary random variable U = [Uj ], which is defined as
Uj = { Zi − Zj (j < i) Zi − Zj+1 (j ≥ i) , 1 ≤ j ≤ L− 1.
(6) Note that the i-th dimension of Z is eliminated due to Zi − Zi = 0. Then Eq. 5 can be rewritten as a non-centred orthant probability distribution
Q(Y = i|x; ẑ) = P (Uj ≥ 0, 1 ≤ j ≤ L− 1). (7)
To simplify the calculation of the probability in Eq. 7, we assume Z follows a multivariate Gaussian distribution with mean ẑ and covariance matrix Σ, i.e., Z ∼ N (ẑ,Σ). Then we have U ∼ N (µ,R), where
µj = { ẑi − ẑj (j < i) ẑi − ẑj+1 (j ≥ i) , 1 ≤ j ≤ L− 1. (8)
And Eq. 7 can be calculated by the following multiple integrations ∫ +∞
0 · · ·
∫ +∞
0 ϕL−1(U ;µ,R)dU1 . . . dUL−1,
(9) where ϕL−1(U ;µ,R) is the probability density function.
We leverage the recursive algorithm proposed by Miwa et al. (2003) to solve the above integrations. Taking L = 4 as an example, the major steps of the algorithm are as follows. First, we decompose the covariance matrix R as R = BBT via Cholesky decomposition, where B is a lower triangular matrix. Then we have U = BM +µ, where M ∼ N (0, IL−1) and IL−1 is an identity matrix of dimension L− 1. Next, Q(Y = i|x; ẑ) can be further decomposed as
Q(Y = i|x; ẑ) = P (Uj ≥ 0, 1 ≤ j ≤ 3) = P (b11M1 + µ1 ≥ 0,
b21M1 + b22M2 + µ2 ≥ 0, b31M1 + b32M2 + b33M3 + µ3 ≥ 0),
(10) where bij denotes the elements in the B matrix, Mj denotes the j-th elements of the random variable M , and µj denotes j-th the elements of µ. Finally, the required probability is given when bij > 0
Q(Y = i|x; ẑ) = ∫ +∞
− µ1 b11
f1(t)ϕ(t)dt, (11)
where ϕ(t) is the standard normal probability density function, and f1 is defined as
f1(s) = ∫ +∞ −µ2−b21s
b22
f2(s, t)ϕ(t)dt, (12)
f2(s1, s2) = ∫ +∞ −µ3−b31s1−b32s2
b33
ϕ(t)dt. (13)
Algorithm 1 summarizes the entire procedure of our proposed framework, where line 12 refers to the integration steps in Eq. 11 to 13. We provide the proofs of the integration steps in Appendix A.4. In practice, we assume Σ is a diagonal matrix and Σii = σ
2 to simplify the calculation, where σ is a hyper-parameter of our algorithm.

heading: 5 Experiments

heading: 5.1 Experimental Settings
text: Datasets and Evaluation Metrics. We evaluate our method on machine reading comprehension (MRC) and natural language understanding (NLU) datasets. For MRC, two widely used multiple-choice datasets RACE (Lai et al., 2017) and DREAM (Sun et al., 2019a) are used. For NLU, we select sentiment analysis dataset SST2 (Socher et al., 2013), linguistic acceptability dataset CoLA (Warstadt et al., 2019), paraphrasing dataset MRPC (Dolan and Brockett, 2005) and QQP (Chen et al., 2017), and natural language inference (NLI) datasets RTE (Bentivogli et al., 2009), MNLI (Williams et al., 2018), and QNLI (Rajpurkar et al., 2016) as representative datasets. Following previous works (Lai et al., 2017; Sun et al., 2019a; Wang et al., 2018), we report Matthews correlation coefficient for CoLA, F1 and accuracy for MRPC and QQP, and accuracy for all the other datasets. For each experiment, the model is evaluated on the validation set once an epoch, and the checkpoint achieving the best validation results is evaluated on the test set. The results averaged over five random seeds are reported for the MRC datasets. Due to the submission quota, we only report results for one trial for the NLU tasks.
Baselines. We compare our method with the following four baselines:
• Hard: We regard the teacher decisions as the ground truth labels and train the student model solely with the cross entropy loss.
• Noisy Logits (Wang, 2021): The student model is trained via the KD objective (Eq. 2)
with the teacher logits replaced with randomly sampled soft labels.
• Smooth: We apply label smoothing (Szegedy et al., 2016) with a smoothing factor 0.1 on the teacher decision of the original sample, and use the smoothed decision as teacher prediction probability. This a straightforward approach to generate soft labels from teacher decisions.
To better investigate the upper bound of our method, we also leverage the following three baselines from Wang (2021):
• Student CE (Wang, 2021): The student model is trained using only the cross entropy loss calculated from the ground-truth labels.
• Standard KD: The student model is optimized with the standard KD objective (Eq. 2). Note that the teacher model is used as a white-box model in this baseline.
• Surrogate: Following Wang (2021), we train the student model via KD with a surrogate teacher, simulating training a lightweight, white-box teacher model for knowledge distillation.
Implementation Details. We implement the teacher model as the finetuned 12-layer BERT model (BERTBASE) or 24-layer BERT model (BERTLARGE) for each task. The student model is a 4-layer or 6-layer BERT-style model. Following previous KD works (Sun et al., 2019b; Li et al., 2021b), we initialize the student model from the raw 12-layer BERT model. We adopt EDA (Wei and Zou, 2019) as the tool for test-time data augmentation.

heading: 5.2 Results on MRC Datasets
text: Experimental results on the MRC datasets RACE and DREAM are shown in Table 1 and Table 2, respectively. In each table, we report three sets of results with different teacher and student architectures. For example, the string “12L→4L” in the tables means we leverage the finetuned 12-layer BERT model (BERTBASE) as the teacher, and the 4-layer BERT-style model as the student. Note that Teacher and Standard KD serve as the upper bounds of our method. Therefore, we do not directly compare our method with them. From these results, we can observe that:
(1) Our proposed method outperforms baselines consistently and significantly. The performance gap between our method and the second best baselines except Teacher and Standard KD are from 0.96 to 1.39 on the RACE datasets and from 0.51 to 0.72 on the DREAM dataset, indicating that narrowing down the information gap between logits and decisions are effective for decision-based KD.
(2) Surprisingly, our proposed method achieves comparable results with Standard KD under a few settings. Standard KD treats the teacher as a whitebox model and is an intuitive upper bound of our method. However, the smallest performance gap between our method and Standard KD is 0.06 (12L→4L on RACE-High) on the RACE datasets and is 0.25 (12L→4L) on the DREAM dataset. Moreover, among all the twelve pairs of results, there is a third of them with a gap of less than 0.50. These results further justify the effectiveness of our proposed method. And we argue that this is mainly due to our better estimation of the logits.
(3) None of the baselines besides Standard KD can consistently achieve better results than training
the student model without KD (Student CE), indicating that decision-based KD is a challenging task and the lost information from decisions compared with logits is essential. Hard performs slightly better than Student CE on the RACE datasets but significantly worse than Student CE on the DREAM dataset. We conjecture that this is because the teacher models have significantly better results on the RACE datasets than on the DREAM dataset, i.e., Hard can only work well with strong teacher models, whose decisions may be less noisy and the information gap between decisions and logits is smaller. Our method can be viewed as a special form of logits smoothing. However, both Noisy Logits and Smooth only achieves comparable or worse results than Student CE, indicating straightforward logits smoothing is not effective.
(4) Our method benefits from both better teacher models and larger student models. When the teacher grows larger (12L→4L v.s. 24L→4L), our method achieves a 1.10 performance gain on the DREAM dataset. Meanwhile, when the student models grow from 4 to 6 layers (12L→4L v.s. 12L→6L), the performance gains on all datasets are remarkably larger. The same trend is also observed for other baselines, suggesting that improving the capacity of the student model is a simple yet effective way to improve the performance of decision-based KD.

heading: 5.3 Results on NLU Datasets
text: Table 3 shows the results on the NLU datasets. First, our proposed method achieves the best results among all decision-based baselines, justifying that our method is generalizable to a large range of NLU
.
tasks. Although our method does not outperform Surrogate on the RTE and MNLI-mm datasets, the gap is only 0.2. Second, the performance gap between ours and Standard KD is also small, providing extra evidence that our method estimates the teacher logits well. Third, all the baselines excluding Teacher and Standard KD have comparable performance, suggesting that decision-based KD is also challenging for NLU tasks. Above all, in conjunction with the results on the MRC datasets, we can conclude that our method is effective for diverse tasks and model architectures.

heading: 5.4 Analysis on Logits Estimation
text: We have conjectured that the good performance of our method comes from better logits estimation. To justify this assumption, we conduct a quantitative analysis in this section. We compute the mean squared errors (MSEs) between the soft labels generated from each method after softmax and teacher predictions on the training set of RACE-High 2. As shown in Figure 2, the soft labels generated by our method are the closest to the teacher predictions among all methods. However, it should be noted that the probabilities (or logits) of the teacher are not perfect, as it does not achieve perfect final performance on the dataset. Therefore, the MSEs have a positive correlation with the final performance but are not oracle indicators.

heading: 5.5 Ablation Study
text: This section consists of a series of experiments aimed at validating the contributions of different components in our method. First, we compare our method with its two variants in Figure 3: (1) w/o
2As Hard produces zero probabilities, Kullback–Leibler divergence is not applicable. A temperature (τ in Eq. 1) of 10 is leveraged when needed.
Empirical Estimations. In this variant, we replace the P̃ (Y |x) in Eq. 4 with the teacher decision on original data to skip the empirical estimation step. (2) w/o Theoretical Estimation. We replace the Q(Y |x; ẑ) term in Eq. 3 with softmax(z) to skip the theoretical estimation step. For each dataset, we also count the percentage of empirical estimations P̃ (Y |x) being one-hot vectors, which means that teacher decisions are consistent on augmented inputs. According to the results, we find that the performance drops for both variants on all datasets, indicating the necessity of empirical estimation and theoretical estimation. Interestingly, we also find a positive correlation between the percentage of one-hot P̃ (Y |x) and the performance degradation from w/o Theoretical Estimation variant. This phenomenon highlights the capability of the theoretical estimation step to estimate teacher logits and narrow the information gap between decisions and logits even without observed decisions.
Second, we further analyze the effect of empirical estimation by changing the sampling times N in Eq. 4. As shown in Figure 4, when N increases, the performance of our method first increases and then stabilizes. Considering a larger N leads to
more queries to the teacher model, N should be as low as possible without compromising the model performance. Therefore, N = 10 is the optimal choice according to the results.
Finally, we investigate the contribution of Eq. 3, which combines the empirical estimation and the theoretical estimation together. In our framework, the root ẑ of the equation is found by fixed-point iteration, and its precision is controlled by the error bound ϵ. Results in Figure 5 show a negative correlation between ϵ and KD performance. As ϵ increases from 10−4 to 10−1, the performance of our method slightly drops. When ϵ increases to 1, which means the logits estimation becomes extremely inaccurate, the performance drops dramatically and is close to the performance of Student CE method.

heading: 5.6 Computational Cost Analysis
text: The additional computational cost of our method compared to Standard KD consists of two parts. The first part is test-time data augmentation, which necessitates multiple queries to the teacher model for each training sample. In this paper, we set the default number of augmented samples N per training case to 10. The second part is solving Eq. 3 using the empirical estimation of decision distribution P̃ (Y |x), which is made negligible by pre-building a lookup table from P̃ (Y |x) to logits estimation ẑ before KD. In total, the additional cost of our method mainly comes from 10 queries made to the teacher model per training sample. By
contrast, the existing soft label generation method DB3KD (Wang, 2021) requires 1,000 to 20,000 queries to the teacher model per training sample.

heading: 6 Conclusion
text: We introduce a novel decision-based KD method, which bridges the information gap between teacher decisions and logits by estimating teacher logits. In contrast to existing solutions for decision-based KD, our method is applicable to NLP tasks with discrete inputs. Extensive experiments over various tasks and model architectures demonstrate the effectiveness of our proposed method.
One future direction for the decision-based KD is the exploration of other NLP tasks, such as neural machine translation, text generation, and question answering. The other direction is KD from non-NN models to NN models, which benefits the training of NN models with additional information from a wider range of models. Unlike conventional KD, decision-based KD does not require internal information from NN models and is promising for solving this problem.
Limitations
This study has two main limitations. The first limitation is its reliance on the assumption that teacher logits on augmented data follow a Gaussian distribution. This assumption is used in the derivation of teacher logits in Section 4.3. However, in practice, teacher logits may not strictly follow a Gaussian distribution. It is challenging to estimate teacher logits under more realistic assumptions, which requires thorough investigations on the distribution of teacher logits and more complex computations for logits estimation.
The second limitation is that our method still requires access to the training dataset of the downstream tasks. In this paper, we focus on KD when teacher PLMs only return decisions. However, our method is not capable of KD without publicly available training data, which is a more challenging scenario for decision-based KD. We believe training a data generation model (Wang, 2021; Zhang et al., 2022; Sanyal et al., 2022) might be useful for such cases.
Ethics Statement
In ethical considerations, our method risks being used as a means of model stealing. Therefore, defensive techniques against the proposed method are required. However, it also has significant positive implications. On the one hand, it can serve as a powerful tool for research on model extraction attacks, thereby promoting the advancement of related studies. On the other hand, it has practical applications in real-world scenarios. For instance, a company may prefer to use a smaller model due to cost considerations, and our method allows for the easy distillation of smaller models without requiring white-box access to larger models. Additionally, our method can be used to distill non-NN models into NN models, reducing the number of model types that need to be maintained and simplifying operation and maintenance.

heading: Acknowledgement
text: This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152, 62236011). We thank all anonymous reviewers for their valuable comments and suggestions on this work. We also thank Shuo Wang and Xiaoyue Mi for their suggestions on the writing.

heading: A Appendix
text: A.1 Experiment Details
Dataset Details In this paper, we use seven different datasets, and all of them are in the English language. We downloaded these datasets from the Datasets (Lhoest et al., 2021) library of version 2.4.0, and our use is consistent with their intended use. The other details of the datasets we used are summarized in Table 4.
Model Details We used BERT-like models (Devlin et al., 2019) in our experiments, including BERTBASE (110M parameters), BERTLARGE (340M parameters), 4-layer BERT-like models (53M parameters), and 6-layer BERT-like models (67M parameters). For BERTBASE and BERTLARGE, the raw model checkpoints are obtained from Huggingface Transformers (Wolf et al., 2020) platform. Following Li et al. (2021b), we initialize the 4-layer and 6-layer BERT-like models from the first 4 and 6 layers of the raw BERTBASE model, respectively.
Other Details We finetune the BERTBASE and BERTLARGE models for 4 epochs. Following Li et al. (2021a), we train small 4-layer or 6-layer models for 10 epochs. We use a learning rate of 5×10−5 for MRC tasks and 2×10−5 for NLU tasks. σ in Algorithm 1 is tuned from {0.5, 1, 2, 4}, λ in Eq. 2 is tuned from {0.2, 0.5, 0.7}, and τ in Eq. 1 is tuned from {5, 10, 20} expect for our method, which performs better in the range of {1, 2, 4}. The α parameters of all operations in EDA are sampled from a half-normal distribution, and we adjust the scale of the distribution to align its expectation with the default α = 0.1 in EDA. For MRC tasks, following Sun et al. (2019b), we concatenate the input passage and the question with a [SEP] token and append each answer at the end of the question. The random seeds we used for experiments on MRC datasets and ablation studies on NLU datasets are from 1 to 5. The random seed for teacher training and other NLU experiments is 1. The training of a 4-layer student model on one RTX 3090 Ti GPU costs approximately 6.5 hours for our method.
A.2 Experimental Results on Generative Language Models
Theoretically, our method can be applied to generative LMs. In this paper, we evaluate the effectiveness of our method on the RACE dataset. We finetune a 12-layer GPT-2 (Radford et al., 2019) teacher to predict the class label (A/B/C/D) given the context, question, and options as prompt. Then we distill the teacher model to a 4-layer GPT-2 student. For the student model, the output vocabulary at the answer position is restricted to class label tokens. Table 5 shows the performance of our method and baseline methods on the dev set of RACE-High. Our method significantly outperforms decision-based baseline methods and the student CE method.
Different from classification tasks, our method will require much more queries to the teacher model on generation tasks because of the following reasons:
1. The label space dimension L in generative tasks is equal to the vocabulary size, which is quite large. The computational cost of building the look-up table will increase.
2. For each position i in a sequence, our method estimates the logits in the i-th position given all the tokens before the i-th position. Therefore, to estimate the logits in the entire sequence, we need to sample teacher decisions in each position. As a result, the computational cost will multiply by the sequence length.
Therefore, how to improve the efficiency of applying our method to generation tasks is an interesting future research direction.
A.3 Detailed Analysis of the Data Augmentation Techniques
In this paper, we use the EDA augmentation tool (Wei and Zou, 2019) for each sample, including its α parameter and the following four default augmentation techniques. Given a input sentence, a α parameter, and the sentence length lsent, the four techniques can be describe as following:
1. Synonym Replacement: First, select αlsent words that are not stop words randomly. Second, replace each word with a random WordNet (Miller, 1995) synonym of itself.
2. Random Insertion: First, select a word in the sentence that is not a stop word randomly. Second, find a random synonym of the selected word. Third, insert the synonym into a random position in the sentence. Finally, do the above steps αlsent times.
3. Random Swap: First, choose two words in the sentence randomly. Second, swap the positions of the chosen words. Finally, do the above steps αlsent times.
4. Random Deletion: Remove each word in the sentence with probability α randomly.
In Table 6, we provide further ablation studies on each technique of EDA. We also include the performance of Surrogate method which is the best decision-based baseline. According to the results, our method is robust to different data augmentation techniques.
A.4 Proofs
In this section, we provide the detailed proofs of Eq. 11 to 13.
In Section 4.3, we assume Σ is a diagonal matrix and Σii = σ2. Therefore, the diagonal elements of B are positive, and Eq. 10 can be rewritten as:
Q(Y = i|x; ẑ) = P (M1 ≥ −µ1 b11 ,
M2 ≥ −µ2 − b21M1
b22 ,
M3 ≥ −µ3 − b31M1 − b32M2
b33 ).
(14) Then Q(Y = i|x; ẑ) can be calculated recursively. Eq. 11 is an integration on M1 ≥ −µ1b11 , while Eq. 12 and Eq. 13 integrate on M2 ≥ −µ2−b21M1
b22 and M3 ≥ −µ3−b31M1−b32M2b33 , respec-
tively.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation Section
3 A2. Did you discuss any potential risks of your work? Ethics Section
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract Section and Introduction Section
3 A4. Have you used AI writing assistants when working on this paper? Grammarly, to check grammar of the whole paper
B 3 Did you use or create scientific artifacts? Section 5
3 B1. Did you cite the creators of artifacts you used? Section 5
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In Appendix
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Appendix
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Appendix
C 3 Did you run computational experiments? In Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? The experimental setup and hyperparameter search methods are in Section 5 and Appendix. We do not include the best-found hyperparameter values
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In section 5 and appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In section 5 and appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: One future direction for the decision-based KD is the exploration of other NLP tasks, such as neural machine translation, text generation, and question answering. The other direction is KD from non-NN models to NN models, which benefits the training of NN models with additional information from a wider range of models. Unlike conventional KD, decision-based KD does not require internal information from NN models and is promising for solving this problem. Limitations
This study has two main limitations. The first limitation is its reliance on the assumption that teacher logits on augmented data follow a Gaussian distribution. This assumption is used in the derivation of teacher logits in Section 4. 3. However, in practice, teacher logits may not strictly follow a Gaussian distribution. It is challenging to estimate teacher logits under more realistic assumptions, which requires thorough investigations on the distribution of teacher logits and more complex computations for logits estimation. The second limitation is that our method still requires access to the training dataset of the downstream tasks. In this paper, we focus on KD when teacher PLMs only return decisions. However, our method is not capable of KD without publicly available training data, which is a more challenging scenario for decision-based KD. We believe training a data generation model (Wang, 2021; Zhang et al. , 2022; Sanyal et al. , 2022) might be useful for such cases